Understanding Multinomial Naive Bayes for Text Classification

Multinomial Naive Bayes is a variant of the Naive Bayes algorithm tailored for handling discrete data, particularly effective in text classification tasks. This approach models word frequencies, assuming a multinomial distribution for features like words. It is widely employed in applications such as spam detection, document classification, and sentiment analysis.

How Multinomial Naive Bayes Works

Multinomial Naive Bayes uses word frequencies to classify text. While Naive Bayes assumes word independence, the multinomial aspect focuses on the frequency of words within a document. The model learns from training data by analyzing word occurrences in different classes, such as spam or non-spam.

Example: If the word 'free' frequently appears in spam emails, the model leverages this to predict the likelihood of new emails being spam. The probability of a document belonging to a class is computed using a class-conditional multinomial distribution:

[ P(X|C) = \frac{n!}{n_1! n_2! \dots n_m!} , p_1^{n_1} p_2^{n_2} \dots p_m^{n_m} ]

Where:

( n ) is the total number of trials.
( n_i ) is the count of occurrences for outcome ( i ).
( p_i ) is the probability of outcome ( i ).

To estimate how likely each word is in a class, we use Maximum Likelihood Estimation (MLE), which calculates probabilities based on word counts from the data:

[ \theta_{c,i} = \frac{\text{count}(w_i, c) + 1}{N + v} ]

Where:

(\text{count}(w_i, c)) is the number of times word ( w_i ) appears in documents of class ( c ).
( N ) is the total number of words in documents of class ( c ).
( v ) is the vocabulary size.

Multinomial Naive Bayes in Spam Detection

To illustrate how Multinomial Naive Bayes works, consider a scenario where messages are classified as Spam or Not Spam.

1. Vocabulary

First, identify all unique words in the dataset.

Vocabulary Size: ( V = 10 )
Vocabulary: {buy, cheap, now, limited, offer, meet, me, let’s, catch, up}

2. Word Frequencies by Class

Spam Class (M1, M2):

buy: 2
cheap: 1
now: 1
limited: 1
offer: 1
Total words: 6

Not Spam Class (M3, M4):

meet: 1
me: 1
now: 1
let's: 1
catch: 1
up: 1
Total words: 6

3. Test Message

Test Message: "buy now"

4. Applying Multinomial Naive Bayes

Probability Formula:
[ P(C|d) \propto P(C) \cdot \prod_i P(w_i|C)^{f_i} ]

Prior Probabilities:
[ P(\text{Spam}) = 0.5, \quad P(\text{Not Spam}) = 0.5 ]

Apply Laplace Smoothing:
To prevent zero probabilities, Laplace smoothing is applied:

[ P(w|C) = \frac{\text{count}(w, C) + 1}{\text{total words in } C + V} ]

Spam Class:

( P(\text{buy} \mid \text{Spam}) = \frac{3}{16} )
( P(\text{now} \mid \text{Spam}) = \frac{2}{16} )
( P(\text{Spam} \mid d) \propto 0.5 \cdot \frac{3}{16} \cdot \frac{2}{16} = \frac{3}{256} )

Not Spam Class:

( P(\text{buy} \mid \text{Not Spam}) = \frac{1}{16} )
( P(\text{now} \mid \text{Not Spam}) = \frac{2}{16} )
( P(\text{Not Spam} \mid d) \propto 0.5 \cdot \frac{1}{16} \cdot \frac{2}{16} = \frac{1}{256} )

5. Final Classification

Since ( P(\text{Spam} \mid d) = \frac{3}{256} > \frac{1}{256} = P(\text{Not Spam} \mid d) ), the message is classified as Spam.

Implementation Example

1. Importing Libraries

Import necessary libraries for data handling, model training, and evaluation.

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score

2. Creating the Dataset

Create a dataset of text messages labeled as spam or not spam with a pandas DataFrame.

data = {
    'text': [
        'Free money now',
        'Call now to claim your prize',
        'Meet me at the park',
        'Let’s catch up later',
        'Win a new car today!',
        'Lunch plans?',
        'Congratulations! You won a lottery',
        'Can you send me the report?',
        'Exclusive offer for you',
        'Are you coming to the meeting?'
    ],
    'label': ['spam', 'spam', 'not spam', 'not spam', 'spam', 'not spam', 'spam', 'not spam', 'spam', 'not spam']
}

![Illustration for: ```python
data = &#123;
    'text':...](https://storage.googleapis.com/xfinit-blogs-scraper-assets-664708921442/blog-assets/images/f8ad12fa-ede5-4c89-937b-5c0e535e8c93.jpg)

df = pd.DataFrame(data)

3. Mapping Labels to Numerical Values

Convert text labels to numerical values for model compatibility.

df['label'] = df['label'].map(&#123;'spam': 1, 'not spam': 0&#125;)

4. Splitting the Data

Divide the dataset into training and testing sets.

X = df['text']
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

5. Vectorizing the Text Data

Convert text data into numerical form using CountVectorizer.

vectorizer = CountVectorizer()
X_train_vectors = vectorizer.fit_transform(X_train)
X_test_vectors = vectorizer.transform(X_test)

6. Training the Naive Bayes Model

Train a Multinomial Naive Bayes classifier with the vectorized data.

model = MultinomialNB()
model.fit(X_train_vectors, y_train)

7. Making Predictions and Evaluating Accuracy

Predict and evaluate the model's accuracy on test data.

y_pred = model.predict(X_test_vectors)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: &#123;accuracy*100:.2f&#125;%")

Output:
Accuracy: 66.67%

8. Predicting for a Custom Message

Test the model with a new message to see its classification.

custom_message = ["Congratulations, you've won a free vacation"]
custom_vector = vectorizer.transform(custom_message)
prediction = model.predict(custom_vector)
print("Prediction for custom message:", "Spam" if prediction[0] == 1 else "Not Spam")

Output:
Congratulations, you've won a free vacation
Prediction for custom message: Spam

Multinomial Naive Bayes vs Gaussian Naive Bayes

While both Multinomial and Gaussian Naive Bayes are variants of the same algorithm, they differ in their suitable data types and applications:

Multinomial Naive Bayes: Designed for discrete data, particularly in text classification tasks. It assumes feature counts like word counts and is efficient for datasets with high feature numbers.
Gaussian Naive Bayes: Suitable for continuous data where features follow a Gaussian distribution. Commonly used in tasks involving continuous data like medical diagnosis and fraud detection.