Optimizing Random Forest Hyperparameters in Python

Random Forest hyperparameter tuning is about refining model parameters to boost performance and accuracy. By tweaking settings like the number of trees, tree depth, and feature selection, one can create a more effective and generalizable machine learning model.

Random Forest Hyperparameters

When tuning Random Forest models, several hyperparameters can be adjusted:

1. n_estimators: This defines the number of trees in the forest. Generally, more trees improve performance but also increase computational costs. For example, using 100 trees is common.

   • Default: n_estimators=100

2. max_features: This limits the number of features considered when splitting a node, helping to control overfitting.

   • Default: max_features="sqrt" (Options: "sqrt", "log2", None)
   • sqrt: Uses the square root of total features, commonly used to reduce overfitting and speed up the model.
   • log2: Uses the base-2 logarithm of the total number of features, providing more randomness and reducing overfitting more than the square root.
   • None: Uses all features for node splitting, increasing model complexity and potential overfitting, especially with many features.

3. max_depth: Controls the maximum depth of each tree. A shallow tree may underfit, while a deep tree might overfit, so selecting the right value is crucial.

   • Default: max_depth=None

4. max_leaf_nodes: Limits the number of leaf nodes, controlling tree size and complexity. If set to None, the number is unlimited.

   • Default: max_leaf_nodes=None

5. max_samples: Determines how much of the dataset is used for each tree. None implies using the entire dataset.

   • Default: max_samples=None

6. min_samples_split: Specifies the minimum number of samples required to split an internal node. By default, each node has two sub-nodes.

   • Default: min_samples_split=2

Hyperparameter Tuning with Scikit-learn

Scikit-learn provides tools for hyperparameter tuning, crucial for enhancing machine learning model performance. This involves selecting the ideal parameter set to maximize model efficiency and accuracy. Two popular techniques are GridSearchCV and RandomizedSearchCV, both essential for automating model fine-tuning.

Below is an example using Random Forest for heart disease prediction.

from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

data = pd.read_csv("heart.csv")

X = data.drop("target", axis=1)
y = data['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

model = RandomForestClassifier(
    n_estimators=100, 
    max_features="sqrt", 
    max_depth=6, 
    max_leaf_nodes=6
)

model.fit(X_train, y_train)

y_pred = model.predict(X_test)

print(classification_report(y_pred, y_test))

Hyperparameter Tuning using GridSearchCV

GridSearchCV is a method that exhaustively searches through all possible parameter combinations specified in a parameter grid to find the best model configuration.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    'n_estimators': [100, 200],
    'max_depth': [None, 10, 20],
    'min_samples_split': [2, 5],
    'min_samples_leaf': [1, 2],
    'bootstrap': [True, False]
}

grid_search = GridSearchCV(RandomForestClassifier(), param_grid=param_grid, cv=5)
grid_search.fit(X_train, y_train)

print("Best Parameters:", grid_search.best_params_)
print("Best Estimator:", grid_search.best_estimator_)

Hyperparameter Tuning using RandomizedSearchCV

RandomizedSearchCV performs a random search over a specified parameter grid, randomly selecting combinations to evaluate the model, often achieving faster results, especially with many hyperparameters.

random_search = RandomizedSearchCV(RandomForestClassifier(),
                                   param_grid)
random_search.fit(X_train, y_train)
print(random_search.best_estimator_)

Both GridSearchCV and RandomizedSearchCV help in identifying the best hyperparameter combinations, leading to models with improved accuracy and more balanced performance metrics across different classes.