Lab 6.5
Linear & Logistic Regression
Overview of the lab
- Linear regression — model, error term, cost function, gradient descent and the closed form, multiple regression, assumptions, evaluation
- Logistic regression — regression for classification, logit and sigmoid, likelihood and cross-entropy, gradient descent
- Multiple classes — one-vs-rest and softmax regression
Notation: the slides write the coefficients as \(w_0, w_1, \dots, w_n\); the linear regression notes use \(\beta_0, \beta_1, \dots, \beta_n\). They are the same parameters.
Linear regression — predicting continuous values
Linear regression is a statistical and machine learning method used to model the relationship between a dependent variable and one or more independent variables. Its goal is to find a line (or hyperplane) that best fits the data.
- \(y\) — dependent variable (the output value we predict)
- \(x_1, x_2, \dots, x_n\) — independent variables (input values for the features)
- \(\beta_0, \beta_1, \dots, \beta_n\) — the coefficients we learn (\(w_0, w_1, \dots, w_n\) on the slides)
- residual: \(y_{actual} - y_{pred}\)
1. Simple linear regression
In simple linear regression, we model the relationship between an input variable \(x\) and an output variable \(y\) with the equation \(y = \beta_0 + \beta_1 x + \varepsilon\), where:
- \(\beta_0\) is the intercept
- \(\beta_1\) is the slope
- \(\varepsilon\) is the error term
The predicted value is written as \(\hat{y} = \beta_0 + \beta_1 x\).
| Goal function | Approximation with the model |
|---|---|
| \(y = \beta_0 + \beta_1 x + \varepsilon\) | \(\hat{y} = \beta_0 + \beta_1 x\) |
2. Interpretation
- The intercept \(\beta_0\) is the predicted value of \(y\) when \(x = 0\).
- The slope \(\beta_1\) tells us how much \(y\) changes when \(x\) increases by 1 unit.
For example, if \(\hat{y} = 2 + 3x\), then when \(x = 0\), \(\hat{y} = 2\), and for every increase of 1 in \(x\), \(\hat{y}\) increases by 3.
Meaning of the error term
\(\varepsilon\) (the error term) is the difference between the actual value and the value predicted by the linear model: \(\varepsilon = y - \hat{y}\). It exists because real-world data is never perfectly linear. Even if \(x\) strongly influences \(y\), there are always other factors affecting the outcome. The error term captures:
- Missing variables: variables that affect \(y\) but are not included in the model. Example: predicting salary from education, \(salary = \beta_0 + \beta_1 \cdot education + \varepsilon\). But salary also depends on experience, skills, location, industry — these influences appear inside \(\varepsilon\).
- Random noise: natural randomness in measurements or behaviour. Example: two people with the same study hours may get different exam scores due to stress, luck, or question difficulty.
- Model imperfection: the true relationship might not be perfectly linear. Example: the real relationship might be \(y = 2 + 3x + 0.5x^2\), but if we fit a linear model, the difference is absorbed into \(\varepsilon\).
Intuition
Actual value = predictable part + unpredictable part: \(y = (\beta_0 + \beta_1 x) + \varepsilon\). The model tries to estimate \(\beta_0\) and \(\beta_1\) so that the errors \(\varepsilon\) are as small as possible.
In practice, we do not directly observe \(\varepsilon\) — we estimate it using the residual \(= y - \hat{y}\). Residuals are our observed approximations of the true error terms. Without the error term, the model would assume perfect predictions, which almost never happens in real data. The error term makes the model statistically realistic and allows us to analyse uncertainty.
Try it: residuals
Sales vs. TV advertising budget, like the figure on the slides (simulated data of the same shape). Every grey line is one residual \(y - \hat{y}\). Move the line and watch the error metrics from section 7.
3. Cost function
To find the best-fitting line, linear regression minimizes the sum of squared errors. For \(n\) data points, the cost function is:
Since \(\hat{y}_i = \beta_0 + \beta_1 x_i\), this becomes:
This is also called the mean squared error (MSE), up to a constant factor depending on convention.
4. Finding the best line
The optimal values of \(\beta_0\) and \(\beta_1\) are chosen so that the cost function is as small as possible. To find them, we minimize the cost function using partial derivatives — commonly with gradient descent.
Partial derivative with respect to \(\beta_0\)
Apply the chain rule to each term \(i = 1 \dots n\):
\[ \frac{\partial}{\partial \beta_0}(\beta_0 + \beta_1 x_i - y_i)^2 = 2(\beta_0 + \beta_1 x_i - y_i) \cdot \frac{\partial}{\partial \beta_0}(\beta_0 + \beta_1 x_i - y_i) \]Differentiate the inner expression:
\[ \frac{\partial}{\partial \beta_0}(\beta_0 + \beta_1 x_i - y_i) = 1 \]Apply this to the entire sum and factor out constants:
\[ \frac{\partial J}{\partial \beta_0} = \frac{1}{n}\sum_{i=1}^{n}2(\beta_0 + \beta_1 x_i - y_i) = \frac{2}{n}\sum_{i=1}^{n}(\beta_0 + \beta_1 x_i - y_i) \]Partial derivative with respect to \(\beta_1\)
Differentiate the inner expression:
\[ \frac{\partial}{\partial \beta_1}(\beta_0 + \beta_1 x_i - y_i) = x_i \]Combine the results and apply to the full cost function:
\[ \frac{\partial J}{\partial \beta_1} = \frac{1}{n}\sum_{i=1}^{n}2(\beta_0 + \beta_1 x_i - y_i)\,x_i = \frac{2}{n}\sum_{i=1}^{n}(\beta_0 + \beta_1 x_i - y_i)\,x_i \]Gradient descent update rules
After computing the gradients, we update the parameters in the opposite direction of the gradient, where \(\alpha\) is the learning rate:
Substituting the derivatives:
Intuition
The partial derivatives measure how much the cost function changes when we slightly change a parameter.
- If the derivative is positive, increasing the parameter increases the cost.
- If the derivative is negative, increasing the parameter decreases the cost.
Gradient descent repeatedly updates the parameters to move downhill on the cost surface, gradually finding the best-fitting line.
Try it: gradient descent on the cost surface
The ellipses are contour lines of \(J(\beta_0, \beta_1)\) for the TV data above (with TV measured in hundreds — a simple feature scaling that makes gradient descent behave). The ★ is the closed-form minimum. Press play and watch each update step downhill; then raise the learning rate \(\alpha\) until the steps overshoot and diverge.
Closed-form solution
For simple linear regression, the closed-form solution is obtained directly by solving \(\frac{\partial J}{\partial \beta_0} = 0\) and \(\frac{\partial J}{\partial \beta_1} = 0\):
where \(\bar{x}\) is the mean of the input values and \(\bar{y}\) is the mean of the output values.
Comparing closed form with gradient descent
| Method | Idea | Parameters | Rule |
|---|---|---|---|
| Gradient descent | Move gradually downhill | Updated repeatedly using \(\alpha\) | \(\beta \leftarrow \beta - \alpha \nabla J(\beta)\) |
| Closed form | Solve derivative = 0 directly | Computed once | \(\nabla J(\beta) = 0\) |
The closed form works well for small linear regression problems, but it becomes expensive when there are many features because it requires matrix inversion. Gradient descent:
- works for large datasets
- works for neural networks
- works for models without closed-form solutions
That is why machine learning libraries usually rely on gradient-based optimization.
5. Multiple linear regression
When there are multiple input variables, the model becomes:
In matrix form, this is often written as \(\hat{y} = X\beta\), where \(X\) is the design matrix and \(\beta\) is the vector of coefficients. The normal equation gives the solution:
6. Assumptions of linear regression
Linear regression works best when several assumptions are approximately true:
- Linearity: an approximately linear relationship between the input data and the output.
- Independence: observations should be independent; the error for one data point should not depend on another.
- Homoscedasticity: the variance of the errors must be constant for all predictor values.
- Normality of errors: errors are approximately normally distributed.
- No strong multicollinearity: predictors are not highly correlated with each other.
7. Evaluating the model
A common metric is the coefficient of determination \(R^2\):
- \(R^2 = 1\) means a perfect fit.
- \(R^2 = 0\) means the model does no better than predicting the mean. (Try a flat line through the mean in the residual playground above.)
Another common metric is the root mean squared error:
8. Example
Suppose we want to predict exam score \(y\) from study hours \(x\). A fitted model might be \(\hat{y} = 40 + 5x\). This means:
- a student who studies 0 hours is predicted to score 40;
- each additional study hour increases the predicted score by 5 points.
9. Advantages and limitations
| Advantages | Limitations |
|---|---|
| Simple to understand | Cannot model complex nonlinear relationships well |
| Fast to train & easy to interpret | Sensitive to outliers |
| Works well when relationships are approximately linear | Assumes a linear relationship between variables; performance can degrade if assumptions are violated |
Logistic regression — classification
Using regression for classification
Although regression is typically used to predict continuous values, any regression technique can also be adapted for classification. Instead of predicting a numeric quantity, we train a separate regression model for each class.
Training phase — for each class, perform a regression model specific to that class, with the training outputs:
- 1 for training instances that belong to the class;
- 0 for instances that do not belong to the class.
Each model learns how strongly an instance belongs to its corresponding class.
Prediction phase — to classify a new instance:
- Evaluate all regression models.
- Each model produces a membership value (a score).
- The predicted class is the one whose model produces the largest output value: \(\hat{y} = \arg\max_k f_k(x)\), where \(f_k(x)\) is the regression output for class \(k\).
When linear regression is used in this framework, the method is called multi-response linear regression: multiple outputs are predicted simultaneously — one for each class.
Problem with regression-based classification
A key limitation is that the membership values are not guaranteed to lie in the range \([0, 1]\). As a result, they cannot be interpreted as valid probabilities — outputs may even be negative or greater than 1. In practice, the outputs are often adjusted using simple heuristics:
- clip values into the range \([0, 1]\);
- normalize them so that the values sum to 1.
This produces values that resemble probability estimates, although they are not theoretically well-founded probabilities.
Moving to logistic regression
Linear regression can be adapted for classification, but its predictions are not restricted to the range \([0, 1]\). To address this issue, we use logistic regression, a model specifically designed for classification.
Key idea: logistic regression builds a linear model for a transformed target variable instead of modeling the probability directly. With two classes (class 0 and class 1), the model predicts the probability that an instance belongs to class 1, \(P(1 \mid a_1, a_2, \dots, a_k)\), where \(a_1, \dots, a_k\) are the input features.
Odds, log-odds and the logit transformation
- The odds of class 1 are the ratio between the probability of class 1 and the probability of class 0: \(\dfrac{P(1 \mid a_1,\dots,a_k)}{1 - P(1 \mid a_1,\dots,a_k)}\).
- Taking the logarithm of the odds gives the log-odds, or logit: \(\text{logit}(p) = \log\left(\dfrac{p}{1-p}\right)\).
Why use the logit transformation? Probabilities are restricted to \(p \in [0, 1]\), but a linear model can produce values anywhere in \((-\infty, +\infty)\). The logit solves this mismatch: \(\text{logit}: [0,1] \rightarrow (-\infty, +\infty)\). After the transformation, the target is no longer restricted and can be modeled with a linear function.
Logistic regression assumes that the log-odds are a linear function of the features:
\[ \log\left(\frac{p}{1-p}\right) = w_0 + w_1 x_1 + \dots + w_n x_n \]The model is still linear in the parameters, but it operates on the transformed probability.
Recovering the probability: the sigmoid
To convert the linear output back to a probability, we apply the inverse logit — the logistic (sigmoid) function. With the linear score \(z = w_0 + w_1 x_1 + \dots + w_n x_n\):
The model outputs a probability \(p\); to convert it to a class prediction we threshold it:
Parameters are found from training data using maximum likelihood.
The model
Given input features \(x\), logistic regression computes a linear score \(z = w^T x + b\), converted into a probability using the sigmoid function:
Probability of one training example
The probability the model assigns to the observed label is \(\hat{p}\) if the true label is 1, and \(1 - \hat{p}\) if the true label is 0. This simply means the model should assign high probability to the correct label.
Probability of the whole dataset — the likelihood
With \(n\) training examples that are independent given the model parameters, the probability of observing the entire dataset is the product of the probabilities of each example — the likelihood:
Log-likelihood
Using the compact form \(P(y_i \mid x_i) = \hat{p}_i^{\,y_i} (1-\hat{p}_i)^{1-y_i}\) with \(y_i \in \{0, 1\}\), the likelihood is \(L(w) = \prod_{i=1}^{n} \hat{p}_i^{\,y_i}(1-\hat{p}_i)^{1-y_i}\), and taking the logarithm converts the product into a sum:
Loss function: binary cross-entropy
Machine learning models are trained by minimizing a loss function, so we minimize the negative log-likelihood:
This loss is called binary cross-entropy. Intuition:
- If the true label is 1: loss \(= -\log(\hat{p})\) — a prediction close to 1 gives a small loss, close to 0 a very large loss.
- If the true label is 0: loss \(= -\log(1 - \hat{p})\) — a prediction close to 0 gives a small loss, close to 1 a very large loss.
Therefore the loss strongly penalizes confident but incorrect predictions.
Gradient descent
Gradient descent updates parameters in the direction that reduces the loss. General update rule, where \(\theta\) is a parameter and \(\eta\) is the learning rate:
For logistic regression, the derivative simplifies nicely. For one example, \(\dfrac{\partial L}{\partial z} = \hat{y} - y\) — the gradient depends on the prediction error. Using the chain rule:
Parameter update rules:
Intuition: the term \((\hat{y} - y)\) is the prediction error. If the model predicts too high → weights decrease; if it predicts too low → weights increase. Each feature \(x\) scales how much the weight changes.
Try it: training logistic regression
Two features, two classes. The shading is \(\hat{p} = P(y = 1 \mid x)\) and the line is the decision boundary \(\hat{p} = 0.5\). Watch gradient descent lower the cross-entropy loss.
Loss over iterations
What happens with perfectly separable data?
With \(p(y_i = 1 \mid x_i) = \sigma(w^T x_i + b)\), logistic regression maximizes the log-likelihood
If the data is perfectly separable, logistic regression can always increase the likelihood further by making the weights larger, because larger weights push the probabilities closer to \(p = 1\) for positives and \(p = 0\) for negatives. It keeps thinking: "If I increase the weights, I can make my predictions more confident." Switch the playground above to Perfectly separable and watch \(\lVert w \rVert\) keep growing.
| Weight | Probability |
|---|---|
| small | 0.8 |
| larger | 0.95 |
| larger | 0.999 |
| larger | 0.999999 |
Logistic regression with multiple classes
One-vs-Rest (OvR)
If there are \(K\) classes, we train \(K\) separate logistic regression models. Each model predicts the probability \(P(y = k \mid x)\): is input example \(x\) from class \(k\) or not?
- Compute the probability from each model.
- Choose the class with the highest probability — the class where \(x\) belongs: \(\hat{y} = \arg\max_k P(y = k \mid x)\).
Multinomial logistic regression (softmax)
Instead of training separate models, we use the softmax function to compute probabilities for all classes at once. This extension is called multiclass logistic regression or softmax regression: the model predicts a probability distribution over all classes.
Problem setup: input features \(x \in \mathbb{R}^d\), \(K\) classes, labels \(y \in \{1, 2, \dots, K\}\). For each class \(k\), the model learns a weight vector \(w_k\) and a bias \(b_k\).
Linear scores (logits): for every class we compute \(z_k = w_k^T x + b_k\) (on the slides: \(z_k = w_{k0} + w_{k1} x_1 + \dots + w_{kn} x_n\)). Collecting all scores gives the vector \(z = [z_1, z_2, \dots, z_K]\) — the logits.
This ensures the probabilities are positive and sum to 1. The prediction is the class with the largest probability: \(\hat{y} = \arg\max_k P(y = k \mid x)\).
Likelihood and loss: assuming independent examples, \(L(W) = \prod_{i=1}^{n} P(y_i \mid x_i)\), and the log-likelihood is
We minimize the negative log-likelihood — the multiclass cross-entropy:
Example: classifying fruit
We classify fruit into \(K = 3\) classes — apple, banana, orange — using the features weight and sweetness. For one fruit \(x = [2, 1]\), the model computes the logits \(z = [1.2, 0.3, 2.1]\). The calculator starts from these values; drag the logits or change the true label.
What happens if the model is wrong? If the model predicted \(P(apple) = 0.7,\ P(banana) = 0.2,\ P(orange) = 0.1\) but the true class is orange, the loss becomes \(-\log(0.1) \approx 2.30\). This large loss strongly penalizes confident incorrect predictions.
Assumptions of logistic regression
- The training examples should be independent of each other.
- Linear relationship with the log-odds: \(\log\left(\frac{p}{1-p}\right) = w_0 + w_1 x_1 + \dots + w_n x_n\).
- Predictor variables should not be highly correlated with each other.
- Large sample size.
- No extreme outliers in predictors — extreme values in the features can strongly affect the estimated coefficients.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| Simple and easy to implement | Cannot capture complex nonlinear relationships unless features are transformed |
| Interpretable model | Sensitive to multicollinearity: highly correlated predictors can make coefficients unstable and difficult to interpret |
| Produces probabilities | Requires feature engineering for complex problems |
| Works well for linearly separable data | Sensitive to outliers |
| Efficient for large datasets |
To be explored: L1 and L2 regularization of logistic regression.
15 auto-graded tasks, 100 points, no derivations: regression vs. classification, intercept and slope, residuals and R², the sigmoid, probabilities and thresholds, decision boundaries and multiclass.