Understanding Gaussian Naive Bayes

Gaussian Naive Bayes is a classification algorithm that operates efficiently on datasets with continuous features, assuming that these features follow a Gaussian distribution. This method's "naive" assumption simplifies computations, resulting in a model that is both fast and effective.

• Advantages:
  • Performs well with small datasets and is simple to implement and interpret.
  • Suitable for continuous numerical features.
  • Efficient and effective for classification problems.
  • Commonly applied in spam detection, medical diagnosis, and pattern recognition.

Mathematical Foundation

Gaussian Naive Bayes assumes that the likelihood of each feature given a class follows a Gaussian distribution. This is expressed mathematically as:

[ P(x_i \mid y) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{(x - \mu)^2}{2\sigma^2}} ]

Where:

• ( x_i ): Feature value.
• ( \mu ): Mean of the feature values for a given class.
• ( \sigma ): Standard deviation of the feature values for that class.
• ( \pi ): Mathematical constant (approximately 3.14159).
• ( e ): Base of the natural logarithm.

The algorithm classifies new data points by finding the class with the highest posterior probability.

Why Gaussian Naive Bayes is Effective for Continuous Data

This algorithm is particularly effective for continuous data because it assumes a normal distribution for each feature. When this assumption holds, the algorithm can make accurate predictions, such as in medical diagnostics or real estate price forecasting.

Practical Example

Consider a binary classification task using a single feature: petal length. Here's a simplified dataset:

| Petal Length (cm) | Class Label | |-------------------|--------------------| | 0 | Iris-setosa | | 1 | Iris-versicolor |

To classify a new sample with a petal length of 1.6 cm, the algorithm follows these steps:

1. Separate by Class:

   • Class 0: [1.4, 1.3, 1.5]
   • Class 1: [4.5, 4.7, 4.6]

2. Calculate Mean and Variance:

   • For Class 0: ( \mu_0 = 1.4 ), ( \sigma_0^2 = 0.0067 )
   • For Class 1: ( \mu_1 = 4.6 ), ( \sigma_1^2 = 0.0067 )

3. Gaussian Likelihood:

   • For ( x = 1.6 ):
     • Class 0: ( P(1.6 \mid C=0) \approx 0.247 )
     • Class 1: ( P(1.6 \mid C=1) \approx 0 )

4. Multiply by Class Priors:

   • Assuming equal priors: ( P(C=0) = P(C=1) = 0.5 )
   • Then: ( P(C=0 \mid x) \propto 0.1235 ), ( P(C=1 \mid x) = 0 )

5. Prediction:

   • Since ( P(C=0 \mid x) > P(C=1 \mid x) ), the predicted class is Iris-setosa.

Implementation

In this section, Gaussian Naive Bayes is applied to the Iris Dataset, which includes features like Sepal Length, Sepal Width, Petal Length, and Petal Width. The goal is to identify the species of Iris flower based on these features.

1. Importing Libraries

import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import LabelEncoder

2. Loading and Preparing the Dataset

iris = load_iris()
data = pd.DataFrame(iris.data, columns=iris.feature_names)
data['Species'] = iris.target

X = data.drop("Species", axis=1)
y = data['Species']

3. Encoding and Splitting the Dataset

le = LabelEncoder()
y = le.fit_transform(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

4. Creating and Training the Model

gnb = GaussianNB()
gnb.fit(X_train, y_train)

5. Plotting Gaussian Distributions

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

feature_names = iris.feature_names
num_features = len(feature_names)
num_classes = len(np.unique(y))

X_np = X.to_numpy() 

for feature_index in range(num_features):
    feature_name = feature_names[feature_index]
    x_vals = np.linspace(X_np[:, feature_index].min(), X_np[:, feature_index].max(), 200)

    plt.figure(figsize=(8, 4))

    for cls in range(num_classes):
        mean = gnb.theta_[cls, feature_index]
        std = np.sqrt(gnb.var_[cls, feature_index])
        y_vals = norm.pdf(x_vals, mean, std)
        plt.plot(x_vals, y_vals, label=f"Class {cls} ({iris.target_names[cls]})")

    plt.title(f"Gaussian Distribution - {feature_name}")
    plt.xlabel(feature_name)
    plt.ylabel("Probability Density")
    plt.legend()
    plt.grid(True)

    plt.show()

6. Making Predictions

y_pred = gnb.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"The Accuracy of Prediction on Iris Flower is: {accuracy}")

High accuracy indicates the model's ability to effectively differentiate between the three species of Iris based on the given features.

Related Topics

• Naive Bayes Classifiers
• Bernoulli Naive Bayes
• Gaussian Naive Bayes using Python Libraries
• Multinomial Naive Bayes
• Plotting Decision Boundaries for Gaussian Naive Bayes