Understanding Bernoulli Naive Bayes for Binary Classification

Bernoulli Naive Bayes is a specific type of the Naive Bayes algorithm, primarily used in scenarios where data is binary. It models the occurrence of features using the Bernoulli distribution. This method is particularly effective for classifying binary features such as 'Yes' or 'No', '1' or '0', and 'True' or 'False'. An important assumption in this model is that features are independent of each other.

Mathematics

In the Bernoulli Naive Bayes model, each feature is assumed to be conditionally independent given the class ( y ). The likelihood of each feature occurring is calculated as follows:

[ p(x_i|y) = p(i|y)x_i + (1 - p(i|y))(1 - x_i) ]

• ( p(x_i | y) ) is the conditional probability of ( x_i ) occurring given ( y ).
• ( i ) is the feature index.
• ( x_i ) holds a binary value, either 0 or 1.

The Bernoulli Naive Bayes relies on the Bernoulli distribution.

Bernoulli Distribution

The Bernoulli distribution is employed for discrete probability calculation, determining either success or failure. A random variable here can be 1 or 0, with probabilities denoted by ( p ) or ( (1-p) ), respectively. The mathematical expression is:

[ f(x) = \begin{cases} p^x \cdot (1-p)^{1-x} & \text{if } x=0,1 \ 0 & \text{otherwise} \end{cases} ]

For ( x=1 ), ( f(x) ) equals ( p ); for ( x=0 ), it equals ( 1-p ), with ( p ) indicating event success.

Example

Consider a binary classification problem to illustrate Bernoulli Naive Bayes. Here are some example messages:

| Message ID | Message Text | Class | |------------|---------------------|---------| | 1 | "buy cheap now" | Spam | | 2 | "limited offer buy" | Spam | | 3 | "meet me now" | Not Spam| | 4 | "let's catch up" | Not Spam|

Step-by-Step Process

1. Vocabulary Extraction: Identify unique words from the dataset: {buy, cheap, now, limited, offer, meet, me, let's, catch, up}. The vocabulary size is 10.

2. Binary Feature Matrix: Represent each message using binary features (presence = 1, absence = 0).

3. Laplace Smoothing: Apply Laplace smoothing to avoid zero probabilities:

   [ P(w_i = 1 | C) = \frac{\text{count}(w_i, C) + 1}{N_C + 2} ]

   where ( N_C = 2 ) for both classes, making the denominator 4.

4. Calculate Word Probabilities: Compute probabilities for each word in the 'Spam' and 'Not Spam' classes.

5. Classify a Message: For a message like "buy now", calculate the likelihood for each class:

   • For Spam: ( P(\text{Spam} | d) \propto 0.5 \cdot 0.75 \cdot 0.5 = 0.1875 )
   • For Not Spam: ( P(\text{Not Spam} | d) \propto 0.5 \cdot 0.25 \cdot 0.5 = 0.0625 )

6. Final Classification: Since ( P(\text{Spam} | d) > P(\text{Not Spam} | d) ), the message is classified as Spam.

Implementing Bernoulli Naive Bayes

To classify emails, consider a dataset with columns for labels ('ham' or 'spam'), their numeric equivalents, and the email text. The dataset contains 5171 entries.

1. Import Libraries

import numpy as np
import pandas as pd
from sklearn.naive_bayes import BernoulliNB
from sklearn.feature_extraction.text import CountVectorizer

2. Data Analysis

Load and preprocess the dataset:

df = pd.read_csv("spam_ham_dataset.csv")
print(df.shape)
df = df.drop(['Unnamed: 0'], axis=1)

3. Count Vectorizer

Convert text data into numerical format using Count Vectorizer:

x = df["text"].values
y = df["label_num"].values
cv = CountVectorizer()
x = cv.fit_transform(x)

4. Data Splitting, Model Training, and Prediction

Split the dataset and train the model:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=0)

bnb = BernoulliNB(binarize=0.0)
model = bnb.fit(X_train, y_train)
y_pred = bnb.predict(X_test)

from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))

The classification report indicates an overall accuracy of 86%.

Difference Between Naive Bayes Models

| Aspect | Gaussian Naive Bayes | Multinomial Naive Bayes | Bernoulli Naive Bayes | |--------|----------------------|-------------------------|-----------------------| | Feature Type | Continuous | Discrete | Binary | | Assumption | Gaussian distribution | Multinomial distribution | Bernoulli distribution | | Common Use Case | Continuous features | Text classification | Binary classification | | Data Representation | Continuous variables | Discrete counts | Binary values | | Mathematical Model | Gaussian distribution | Multinomial distribution | Bernoulli distribution | | Example | Predict using numeric features | Predict using word counts | Classify based on word presence |

Bernoulli Naive Bayes is frequently used for spam detection, text classification, sentiment analysis, and determining word presence in documents.