Scaling Techniques in Feature Engineering: Normalization and Standardization

Feature engineering is a critical step in machine learning that involves creating, transforming, or selecting key features from raw data to enhance model performance. By identifying the most useful patterns and relationships, feature engineering significantly improves how models learn and interpret data.

Benefits of Feature Engineering

• Enhanced Pattern Recognition: Well-crafted features enable models to identify complex patterns more effectively.
• Increased Prediction Accuracy: Filtering out noise and irrelevant details improves the precision of model predictions.
• Reduced Overfitting: Concentrating on significant features helps models generalize better across different datasets.
• Improved Model Interpretability: Features that are clear and informative aid in understanding the model's decisions.

Scaling Techniques in Feature Engineering

1. Absolute Maximum Scaling

Absolute Maximum Scaling involves dividing each feature value by the maximum absolute value of that feature, which rescales data to the range of -1 to 1.

• Sensitive to Outliers: Extreme values can distort the scaling.
• Best for Clean Data: More effective when data lacks strong outliers.

Scaling Formula: [ X_{\text{scaled}} = \frac{X_i}{\text{max}(|X|)} ]

Implementation:

import pandas as pd
import numpy as np

df = pd.read_csv('Housing.csv')
df = df.select_dtypes(include=np.number)

max_abs = np.max(np.abs(df), axis=0)
scaled_df = df / max_abs
scaled_df.head()

2. Min-Max Scaling

Min-Max Scaling rescales features by subtracting the minimum value and dividing by the range (max - min), mapping values to 0-1 while maintaining the original distribution.

Scaling Formula: [ X_{\text{scaled}} = \frac{X_i - X_{\text{min}}}{X_{\text{max}} - X_{\text{min}}} ]

Implementation:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
scaled_data = scaler.fit_transform(df)
scaled_df = pd.DataFrame(scaled_data, columns=df.columns)
scaled_df.head()

3. Normalization (Vector Normalization)

Normalization adjusts each data sample so its Euclidean norm becomes 1, focusing on the direction rather than magnitude.

Scaling Formula: [ X_{\text{scaled}} = \frac{X_i}{| X |} ]

Implementation:

from sklearn.preprocessing import Normalizer

scaler = Normalizer()
scaled_data = scaler.fit_transform(df)
scaled_df = pd.DataFrame(scaled_data, columns=df.columns)
scaled_df.head()

4. Standardization

Standardization scales features by subtracting the mean and dividing by the standard deviation, resulting in zero mean and unit variance.

Scaling Formula: [ X_{\text{scaled}} = \frac{X_i - \mu}{\sigma} ]

Implementation:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaled_data = scaler.fit_transform(df)
scaled_df = pd.DataFrame(scaled_data, columns=df.columns)
print(scaled_df.head())

5. Robust Scaling

Robust Scaling uses the median and interquartile range (IQR) for scaling, making it resistant to outliers.

Scaling Formula: [ X_{\text{scaled}} = \frac{X_i - X_{\text{median}}}{\text{IQR}} ]

Implementation:

from sklearn.preprocessing import RobustScaler

scaler = RobustScaler()
scaled_data = scaler.fit_transform(df)
scaled_df = pd.DataFrame(scaled_data, columns=df.columns)
print(scaled_df.head())

Comparing Feature Scaling Techniques

| Method | Description | Sensitivity to Outliers | Typical Use Cases | |--------|-------------|-------------------------|-------------------| | Absolute Maximum Scaling | Divides values by max absolute value | Yes | Sparse data, simple scaling | | Min-Max Scaling | Rescales via min-max normalization | Yes | Neural networks, bounded features | | Normalization | Scales each sample to unit length | No | Direction-based tasks, text classification | | Standardization | Centers to mean 0, scales to unit variance | Moderate | Most ML algorithms, normal data | | Robust Scaling | Centers on median, scales using IQR | Low | Data with outliers, skewed data |

Advantages

• Boosts Model Performance: Enhances accuracy by presenting features in comparable scales.
• Speeds Up Convergence: Facilitates faster training for gradient-based algorithms.
• Prevents Feature Bias: Ensures all features contribute fairly without dominance.
• Increases Numerical Stability: Minimizes risks of computational errors.
• Facilitates Compatibility: Prepares data for models reliant on distances and gradients.