Exploring Logistic Regression

Logistic Regression is a widely used supervised machine learning technique for solving classification problems. Unlike linear regression, which predicts continuous values, logistic regression estimates the probability of an input belonging to a particular class.

Key Features of Logistic Regression

• Binary Classification: It is primarily used for binary classification tasks, where the output can be one of two categories, such as Yes/No, True/False, or 0/1.
• Sigmoid Function: The algorithm employs a sigmoid function to transform inputs into a probability value between 0 and 1.

Types of Logistic Regression

Logistic regression can be categorized based on the nature of the dependent variable:

1. Binomial Logistic Regression: Applied when the dependent variable has two possible categories like Yes/No or 0/1. This is the most common type of logistic regression.

2. Multinomial Logistic Regression: Used when the dependent variable has three or more categories without a natural order, such as classifying animals into "cat," "dog," or "sheep."

3. Ordinal Logistic Regression: Utilized when the dependent variable has three or more ordered categories, such as ratings like "low," "medium," and "high."

Assumptions of Logistic Regression

For logistic regression to be effective, certain assumptions must be met:

1. Independent Observations: Data points should be independent.
2. Binary Dependent Variables: Assumes the dependent variable is binary. Functions like SoftMax are used for more than two categories.
3. Linearity of Independent Variables and Log Odds: Assumes a linear relationship between independent variables and the log odds of the dependent variable.
4. Absence of Outliers: Extreme outliers should be minimized as they can skew the results.
5. Sufficiently Large Sample Size: A large dataset is needed for reliable and stable results.

Understanding the Sigmoid Function

1. The sigmoid function is crucial in logistic regression, converting model outputs into a probability range between 0 and 1.
2. This function maps real numbers into a range between 0 and 1, forming an "S" shaped curve.
3. A threshold, usually 0.5, is used to decide class labels: values equal to or above the threshold are classified as Class 1, otherwise as Class 0.

Working of Logistic Regression

Logistic regression transforms the linear regression continuous value output into categorical output using a sigmoid function. This process maps independent variables into a value between 0 and 1, known as the logistic function.

Consider input features as a matrix:

[X = \begin{bmatrix} x_{11} & ... & x_{1m}\ x_{21} & ... & x_{2m} \ \vdots & \ddots & \vdots \ x_{n1} & ... & x_{nm} \end{bmatrix}]

With a binary dependent variable (Y) having values 0 or 1:

[Y = \begin{cases} 0 & \text{ if } \text{Class 1} \ 1 & \text{ if } \text{Class 2} \end{cases}]

A multi-linear function is applied to input variables (X):

[z = \left(\sum_{i=1}^{n} w_{i}x_{i}\right) + b]

Here, (x_i) is the ith observation, (w_i) are weights, and (b) is the bias term. The dot product of weight and input is represented as:

[z = w \cdot X + b]

The continuous value (z) is converted into a probability using the sigmoid function:

[\sigma(z) = \frac{1}{1+e^{-z}}]

Logistic Regression Equation and Odds

The odds of the dependent event occurring are modeled as:

[\frac{p(x)}{1-p(x)} = e^z]

The natural logarithm of odds provides the log-odds or logit:

[\log \left[ \frac{p(x)}{1-p(x)} \right] = w \cdot X + b]

The logistic regression equation is then:

[p(X; b, w) = \frac{1}{1+e^{-w \cdot X + b}}]

Likelihood Function for Logistic Regression

The objective is to find weights and bias that maximize the data observation likelihood. For each data point:

• For (y=1), predicted probabilities are (p(X; b, w)).
• For (y=0), predicted probabilities are (1-p(X; b, w)).

The likelihood function is:

[L(b,w) = \prod_{i=1}^{n} p(x_i)^{y_i}(1-p(x_i))^{1-y_i}]

The log-likelihood function is derived by taking natural logs.

Gradient of the Log-likelihood Function

To optimize weights and bias, gradient ascent is used on the log-likelihood function. The gradient with respect to each weight (w_j) is:

[\frac{\partial J(l(b,w))}{\partial w_j} = \sum_{i=n}^{n}(y_i - p(x_i; b, w))x_{ij}]

Terminologies in Logistic Regression

• Independent Variables: Input features used to predict the dependent variable.
• Dependent Variable: The target categorical variable.
• Logistic Function: Converts independent variables into a probability between 0 and 1.
• Odds: Ratio of event probability to its non-occurrence probability.
• Log-Odds (Logit): Natural logarithm of the odds.
• Coefficient: Parameters estimated by the model indicating effects of independent variables.
• Intercept: Constant term representing log-odds when independent variables are zero.
• Maximum Likelihood Estimation (MLE): Method to estimate model coefficients by maximizing data likelihood.

Implementing Logistic Regression in Python

1. Binomial Logistic Regression

In binomial logistic regression, the target variable has two values like "0" or "1". The sigmoid function is used for prediction. Below is an example using a breast cancer dataset:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_breast_cancer(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=23)

clf = LogisticRegression(max_iter=10000, random_state=0)
clf.fit(X_train, y_train)

acc = accuracy_score(y_test, clf.predict(X_test)) * 100
print(f"Logistic Regression model accuracy: {acc:.2f}%")

2. Multinomial Logistic Regression

For multinomial logistic regression, the target variable can have three or more unordered types. The softmax function is used instead of the sigmoid function. Here's an example using the Digits dataset:

from sklearn.model_selection import train_test_split
from sklearn import datasets, linear_model, metrics

digits = datasets.load_digits()

X = digits.data
y = digits.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1)

reg = linear_model.LogisticRegression(max_iter=10000, random_state=0)
reg.fit(X_train, y_train)

y_pred = reg.predict(X_test)

print(f"Logistic Regression model accuracy: {metrics.accuracy_score(y_test,y_pred)*100:.2f}%")

Evaluating Logistic Regression Models

Evaluating a logistic regression model is crucial to ensure its performance on unseen data. Common evaluation metrics include:

1. Accuracy: Proportion of correctly classified instances.

2. Precision: Accuracy of positive predictions.

3. Recall (Sensitivity or True Positive Rate): Proportion of correctly predicted positive instances.

4. F1 Score: Harmonic mean of precision and recall.

5. Area Under the Receiver Operating Characteristic Curve (AUC-ROC): Measures model performance across classification thresholds.

6. Area Under the Precision-Recall Curve (AUC-PR): Summary of model performance across precision-recall trade-offs.

Differences Between Linear and Logistic Regression

Logistic regression and linear regression differ in application and output. Here's a comparison:

• Definition: Linear regression predicts continuous values, while logistic regression predicts categorical values.
• Problem Type: Linear regression is for regression problems; logistic regression is for classification problems.
• Output Type: Linear regression outputs continuous values; logistic regression outputs categorical values.
• Curve/Model Fitting: Linear regression finds the best fit line; logistic regression finds an S-Curve.
• Estimation Method: Linear regression uses least square estimation; logistic regression uses maximum likelihood estimation.
• Output Example: Linear regression outputs continuous variables like price; logistic regression outputs categorical variables like 0 or 1.
• Relationship Requirement: Linear regression requires a linear relationship between variables; logistic regression does not.