Logistic Regression

Despite its name containing "regression," logistic regression is fundamentally a classification algorithm — and arguably the most important one to understand deeply. It serves as the gateway between linear models and neural networks, forms the output layer of deep learning classifiers, and remains the go-to algorithm when you need an interpretable, probabilistic classifier. If you understand logistic regression thoroughly, you have laid the groundwork for understanding everything from simple classifiers to complex deep learning architectures.

Why Not Use Linear Regression for Classification?

This is the question that motivates logistic regression beautifully. Suppose you want to predict whether an email is spam (1) or not spam (0). You might think: "I will fit a line to the data and predict 1 if the output is above 0.5, else predict 0."

The problem? Linear regression outputs any number from negative infinity to positive infinity. It can predict 2.7 or -1.3, which makes no sense as a probability. We need a function that squashes any input into the range [0, 1] — enter the sigmoid function.

The Sigmoid Function: The Heart of Logistic Regression

The sigmoid (or logistic) function transforms any real number into a probability between 0 and 1:

Here is the key insight: logistic regression first computes a linear combination of features (just like linear regression), then passes the result through the sigmoid to get a probability:

The sigmoid function is differentiable everywhere, which is crucial for optimization using gradient descent. Its derivative has an elegant form: σ'(z) = σ(z) × (1 - σ(z)).

The Loss Function: Binary Cross-Entropy

We cannot use Mean Squared Error (MSE) as the loss function because the sigmoid makes it non-convex — gradient descent would get stuck in local minima. Instead, logistic regression uses binary cross-entropy (also called log loss):

The intuition behind this loss function is elegant: when the true label is 1 and the model predicts probability 0.99, the loss is tiny (-log(0.99) ≈ 0.01). When the true label is 1 but the model predicts 0.01, the loss is enormous (-log(0.01) ≈ 4.6). The function heavily penalizes confident wrong predictions.

Gradient Descent Optimization

Training logistic regression means finding weights w and bias b that minimize the cross-entropy loss. We use gradient descent to iteratively update parameters:

The gradient has a remarkably clean form — it is simply the prediction error multiplied by the feature value, averaged over all training examples. This is why logistic regression is so popular: the math is clean, the optimization is convex (guaranteed to find the global minimum), and training is fast.

Implementation From Scratch and with Sklearn

Regularization: Preventing Overfitting

When features are highly correlated or the model has too many parameters relative to training samples, logistic regression can overfit. Regularization adds a penalty term to the loss function that discourages large weights:

• L2 (Ridge): Adds λ × Σ wⱼ² — shrinks all weights toward zero but never exactly to zero. This is the default in sklearn (controlled by C = 1/λ).
• L1 (Lasso): Adds λ × Σ |wⱼ| — can drive some weights exactly to zero, effectively performing feature selection.
• Elastic Net: Combines both L1 and L2 penalties.

The C parameter in sklearn is the inverse of regularization strength. Smaller C means stronger regularization (simpler model). Use cross-validation to find the optimal C value.

Multi-Class Extensions

Logistic regression naturally handles binary classification. For multi-class problems, two strategies exist:

One-vs-Rest (OvR): Train K separate binary classifiers, each distinguishing one class from all others. Predict the class whose classifier gives the highest probability.

Multinomial (Softmax): Generalize the sigmoid to multiple classes using the softmax function. This trains a single model that outputs probabilities for all classes simultaneously.

Decision Boundary Interpretation

The decision boundary of logistic regression is always linear (a straight line in 2D, a hyperplane in higher dimensions). The boundary is where the predicted probability equals exactly 0.5, which occurs when w₁x₁ + w₂x₂ + ... + b = 0. Points on one side get predicted as class 0, points on the other as class 1. The magnitude of each weight tells you how much that feature influences the prediction — making logistic regression highly interpretable.

When to Use Logistic Regression

Use logistic regression as your first classifier for any new problem. It trains fast, produces probabilistic outputs, is highly interpretable, works well with linearly separable data, and serves as a strong baseline. Move to more complex models only when logistic regression is clearly insufficient — and even then, compare against it to understand what the complex model adds.

Key Takeaways

Logistic regression applies the sigmoid function to a linear combination of features to produce class probabilities. It uses cross-entropy loss and gradient descent for training, with regularization to prevent overfitting. Despite its simplicity, it remains one of the most important algorithms in machine learning — both as a standalone classifier and as the foundation for neural networks. Master logistic regression and you understand the core mechanics of how machines learn to classify.