Understanding Random Forest Classifier with Scikit-Learn
Random Forest is an ensemble learning algorithm that creates multiple decision trees and merges their outputs to enhance prediction accuracy and minimize overfitting. In Scikit...
Random Forest is an ensemble learning algorithm that creates multiple decision trees and merges their outputs to enhance prediction accuracy and minimize overfitting. In Scikit-learn, the Random Forest Classifier is popular for classification tasks due to its ability to manage large datasets and complex nonlinear relationships effectively.
How Random Forest Classifier Works
- Bootstrap Sampling: Random samples of data are taken with replacement to train each decision tree.
- Random Feature Selection: Each tree is trained using a random subset of features.
- Build Decision Trees: Trees split the data based on the best feature from their subset until a stopping condition, like maximum depth, is reached.
- Make Predictions: Each tree provides its own prediction result.
- Majority Voting: The final prediction is determined by the majority vote from all trees.
Advantages of Random Forest Classification
- It efficiently handles large and high-dimensional datasets.
- By combining multiple decision trees, it reduces the likelihood of overfitting compared to using a single tree.
- It is robust against noisy data and performs well with categorical variables.
Implementing Random Forest Classification in Python
Before constructing a Random Forest model in Python, understanding its parameters is essential:
- n_estimators: Number of trees in the forest.
- max_depth: Maximum depth of each tree.
- max_features: Number of features to consider for splitting at each node.
- criterion: Function to measure split quality, such as 'gini' or 'entropy'.
- min_samples_split: Minimum number of samples required to split an internal node.
- min_samples_leaf: Minimum number of samples required to be at a leaf node.
- bootstrap: Whether bootstrap samples are used when building trees (True or False).
1. Import Required Libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
2. Import Dataset
The Iris Dataset, available in Scikit-learn, will be used. It includes data on three Iris flower species, characterized by features like sepal length, width, etc.
iris = load_iris()
df = pd.DataFrame(data=iris.data, columns=iris.feature_names)
df['target'] = iris.target
3. Data Preparation
Separate the features (X) from the target variable (y).
X = df.iloc[:, :-1].values
y = df.iloc[:, -1].values
4. Splitting the Dataset
Divide the dataset into training and testing sets to train the model and evaluate its performance.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
5. Feature Scaling
While Random Forest isn't highly sensitive to feature scaling, it's generally good practice when combining models.
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
6. Building the Random Forest Classifier
Create and train a Random Forest Classifier model, then make predictions.
classifier = RandomForestClassifier(n_estimators=100, random_state=42)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
7. Evaluation of the Model
Use accuracy scores and confusion matrices to evaluate the model's performance.
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy*100:.2f}%')
conf_matrix = confusion_matrix(y_test, y_pred)
plt.figure(figsize=(8, 6))
sns.heatmap(conf_matrix, annot=True, fmt='g', cmap='Blues', cbar=False,
xticklabels=iris.target_names, yticklabels=iris.target_names)
plt.title('Confusion Matrix Heatmap')
plt.xlabel('Predicted Labels')
plt.ylabel('True Labels')
plt.show()
8. Feature Importance
Random Forest Classifiers can indicate which features are most crucial for predictions.
feature_importances = classifier.feature_importances_
plt.barh(iris.feature_names, feature_importances)
plt.xlabel('Feature Importance')
plt.title('Feature Importance in Random Forest Classifier')
plt.show()
From the graph, petal width (cm) emerges as the most significant feature, closely followed by petal length (cm). Sepal measurements are less important, indicating the classifier relies more on petal dimensions for predicting the flower species.