Data 311: Machine Learning

Lecture 3: Assessing Regression Models

Dr. Irene Vrbik

University of British Columbia Okanagan

Testing vs. Training MSE

When MSE is calculated using $X_{T r}$ we call it the training MSE

$M S E_{T r} = \frac{1}{n} \sum_{i = 1}^{n} (y_{i} - \hat{f} (x_{i}))^{2}$ When MSE is calculated using $X_{T e}$ we call it the test MSE:

$M S E_{T e} = \frac{1}{m} \sum_{i = 1}^{m} (y_{i} - \hat{f} (x_{i}))^{2}$

Goal

Plainly speaking, we do not really care how well the method works on the training data.

We are interested in the accuracy of predictions to data (e.g. new patients, future stock prices) it’s never seen before.
${MSE}_{T e}$ is a metric of how well $\hat{f}$ generalizes to unseen data, thus, we want this (rather than ${MSE}_{T r}$ ) to be as low as possible.

Error types

In general, $\hat{f}$ will not match $f$ perfectly and the discrepancy can be broken down into two types of errors…

reducible error which we can reduce by picking a better statistical learning technique; and
irreducible error which we can never improve even if we estimate $f$ perfectly

Math

Suppose, for notational convenience, we have a training set $X_{T r} = S$ that was used to fit our model ${\hat{f}}_{S}$ . Given a test set $X_{T e} = (X, Y)$ , we can write:

\begin{aligned} = E [(y - \hat{y})^{2}] \\ = E [(y - {\hat{f}}_{S} (x))^{2}] \\ = E [(f (x) + ϵ - {\hat{f}}_{S} (x))^{2}] \\ = E [{((f (x) - {\hat{f}}_{S} (x)) + ϵ)}^{2}] \\ = E [(f (x) - \hat{f} (x))^{2} + 2 (f (x) - \hat{f} (x)) ϵ + ϵ^{2}] \end{aligned}

$x$ and ${\hat{f}}_{S} (x)$ assumed fixed, $f$ fixed (but unknown)

\begin{aligned} = {(f (x) - \hat{f} (x))}^{2} + 2 (f (x) - {\hat{f}}_{S} (x)) E [ϵ] + E [ϵ^{2}] \\ = (f (x) - {\hat{f}}_{S} (x))^{2} + E [ϵ^{2}] since E [ϵ] = 0 \end{aligned}

Recall $Var (X) = E [X^{2}] - (E [X])^{2}$ .
Hence $Var (ϵ) = E [ϵ^{2}] - (E [ϵ])^{2} = E [ϵ^{2}] (since E [ϵ] = 0)$

\begin{aligned} = \underset{reducible error}{\underset{⏟}{(f (x) - {\hat{f}}_{S} (x))^{2}}} + \underset{irreducible error}{\underset{⏟}{Var [ϵ]}} \end{aligned}

Math (cont’d)

Reducible error can be further decomposed into error due to squared bias and variance.

\begin{aligned} = E [reducible error] \\ = E [(f (x) - {\hat{f}}_{S} (x))^{2}] \\ = \underset{{Bias}^{2} (\hat{f} (x))}{\underset{⏟}{(f (x) - E [{\hat{f}}_{S} (x)])^{2}}} + \underset{Var (\hat{f} (x))}{\underset{⏟}{E [({\hat{f}}_{S} (x) - E [{\hat{f}}_{S} (x)]^{2})]}} \end{aligned}

Decomposition of MSE

All together we have the expected test MSE at a new point $x$ .
$\begin{aligned} = E_{Y | x, S} [(y - \hat{f} (x))^{2}] \\ = \underset{reducible error}{\underset{⏟}{Var (\hat{f} (x)) + {Bias}^{2} (\hat{f} (x))}} + \underset{irreducible error}{\underset{⏟}{Var (ϵ)}} \end{aligned}$ It’s the average $M S E_{T e}$ we’d get if we repeatedly estimated $f$ using a large number of training sets, and tested at each at $x$

Definitions

Bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model.

High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting)

Variance refers to the amount by which $\hat{f}$ would change if we estimated it using a different training data set.

High variance may result from an algorithm modeling the random noise in the training data (overfitting).

Target Visualizations

Inspired by Essays by Scott Fortmann-Roe

We can create a graphical visualization of bias and variance using a bulls-eye diagram. DESCRIBE IMAGE - Imagine that the center of the target is a model that perfectly estimates $f$ . - Each hit represents an individual realization of our model, given the chance variability in the training data we gather. - Sometimes we get a “good” training data so we predict very well (close to bullseye) - sometimes we get “bad” training (eg. full of outliers or non-standard values) resulting in poorer predictions.

Low Bias (first row) Models that approximates the real-life problem well will have low bias (hits will be centered around the bullseye)

High Bias (second row) will systematically be “off the mark”

Low variance (first column) indicates that $\hat{f}$ would not change much even if we estimated it using a different training data set. (so the hits will all be close together)

High variance (second column) indicates that $\hat{f}$ will be very sensitivity to small fluctuations in the training (so the hits will be less close together)

Target Explained

Low Bias (first row) Models that approximates the real-life problem well will have low bias (hits will be centered around the bullseye)
High Bias (second row) will systematically be “off the mark”
Low variance (first column) indicates that $\hat{f}$ would not change much even if we estimated it using a different training data set. (so the hits will all be close together)
High variance (second column) indicates that $\hat{f}$ will be very sensitivity to small fluctuations in the training (so the hits will be less close together)

Section 2.2.2

The following simulations are taken from your ISLR2 textbook.
Looking at a variety of different data we can gain some general insights on ${MSE}_{T e}$ and ${MSE}_{T r}$
Again, the particulars about the specific models (linear regression and loess smoothing splines) are not important.
What is important is it understand how the level of flexibility impact the bias and variance of the models (and therefore the MSE).

Figure 2.9 (left plot)

ISLR Figure 2.9 (left plot) Data simulated from $f$ , shown in black. Three estimates of $f$ are shown: the linear regression line (orang ecurve), and two smoothing spline fits (blue and green curves). jump to right-hand panel

Green curve

Low bias, high variance
As the green curve is the most flexible, it matches the training data very closely
However, it is much more wiggly than $f$ (ie the “true” generating black curve)

Blue curve

Low bias, low variance
The blue curve strikes the balance between low variance and low bias
As one may expect, the average fitted curve is quite similar to $f$ (ie the “true” generating black curve)

Estimated test MSE

To estimate the expected ${MSE}_{T r}$ and ${MSE}_{T e}$ their values over a very large collection of data sets.
The average training MSE and testing MSE is plotted in gray and red, respectively, as a function of flexibility.
The flexibility of these models are mapped to numeric values on the $x$ -axis (lower values indicate less flexibility).
Squares (🟧, 🟦, 🟩) indicate the MSEs associated with the corresponding orange, blue, and green models.

Test MSE

The orange and green models have high ${MSE}_{T e}$ but for different reasons

orange is underfitting
green is overfitting

Blue is close to optimal

Minimizing test MSE

The horizontal dashed line indicates $Var (ϵ)$ , the irreducible error
This line corresponds to the lowest achievable test MSE among all possible methods.
Hence, the “blue model” is close to optimal.

Training MSE

The green curve has the lowest training MSE of all three methods, since it corresponds to the most flexible of the three curves fit in the left-hand panel

Data 311: Machine Learning Lecture 3: Assessing Regression Models Dr. Irene Vrbik University of British Columbia Okanagan

Data 311: Machine Learning
Recap
Motivation
metrics for the classification...
Introduction
Outline
Statistical Learning Model
Workflow
Regression Problem
MSE
MSE Visualized
Fit a model \(\hat...
For each $x_{i}$ ...
… and predicted value...
We average the squared...
Properties of MSE
Motivating Question
Splitting
Slide 19
Notation
Model Fitting
Slide 22
Testing vs. Training MSE
MSE train = average...
MSE test = average...
Goal
Error types
Math
Math (cont’d)
Reducible error
Decomposition of MSE
Irreducible Error
Bias-Variance
Definitions
Target Visualizations
Target Explained
Simulations
Training set 1
Training set 2
Generative Model
Low Variance High Bias
Training set 1 Let’s...
Training set 2 Fitting...
10 training sets
Low Variance
High Bias
Low Bias High Variance
Green Model
Training set 2 Fitting...
Training set 2 Fitting...
10 training sets
High Variance
Low Bias
Low bias and Low variance
Blue Model
Training set 1 Fitting...
Training set 2 Fitting...
10 training sets
Low Variance
Low Bias
Average model across simulations
Goldilocks Principle
Bias Variance Tradeoff
Looking Forward
ISLR Simulations
Section 2.2.2
Figure 2.9 (left plot)
Figure 2.9 (left...
Figure 2.9 (left...
Figure 2.9 (left...
Estimated test MSE
Figure 2.9 (right)
Test MSE The orange...
Minimizing test MSE...
Training MSE The...
Training MSE The...
Training MSE The...
General Trends