Data 311: Machine Learning

Lecture 3: Assessing Regression Models

Dr. Irene Vrbik

University of British Columbia Okanagan

Recap

• Last lecture we introduced some notation and terminology that we will be using throughout the course.

• We discussed how different tasks, namely inference vs prediction may lead us to favour certain models over others.

• Today we discuss the topic of model assessment to address the challenging task of model selection

• For inference we would gravitate to perhaps a more simple parametric model since we value interpretibility over prediction power whereas a flexible non-parameteric approach may result in better predictions and would therefore be better suited for prediction.
• Other considerations: time it takes to run, how easy they are to explain, but today we focus on how well they can predict

Motivation

• There is no one-size-fits-all best model.

• This course aims to introduce you a small subset of the machine learning approaches available each with their own set of limitations.

• Selecting the best approach for a particular problem can be one of the most challenging tasks for a data scientist.

• Today’s lecture will add to this list of considerations when choosing a model in the supervised regression setting …

• we are searching for a model that is “good enough” (remember all models are wrong but some are useful)

• all models will have some predictive error and with many candidate models to choose from, there are a number of considerations that may go into narrowing it down.

• today: look at key theoretical concepts that arise in estimating \(f\)

metrics for the classification problem (i.e. when the response variable is categorical) will be covered later.

Introduction

• If our data is labelled, we may want to investigate how our predictions for some response/output \(y\) (denoted \(\hat y\) and based on the fitted model \(\hat f\)) compare to the observed values \(y\).

• Naturally, the closer our predicted values are to the true response value, the better.

• Our response variable will generally be one of two forms: categorical or numeric. Today we focus on assessing models with a numeric response.

Outline

• MSE and Testing vs. Training MSE

• Decomposition of MSE

• Reducible error and Irreducible Error

• Bias-Variance

• Simulations

Statistical Learning Model

Recall our general model for statistical learning \[Y = f(X) + \epsilon\]

• where \(X\) are our inputs,
• \(Y\) is the numeric output (at least in this setting)
• \(\epsilon\) is the error term (independent of \(X\) and with mean 0),
• \(f\) is the systematic information \(X\) provides about \(Y\).

Goal: find an \(\hat f(X)\) that approximates the true function \(f(x)\) as well as possible.

• find an \(\hat f(X)\) “as close as possible” to \(f\)

\(\epsilon\) attributed to factors that are beyond the model’s control.

• may be from unmeasured variables that are useful in predicting \(Y\):sinc ewe don’t measure them, \(f\) cannot use them for its prediction.
• measurement errors
• stochastic or random distribution. This randomness makes it impossible to make perfectly accurate predictions, as there will always be some level of uncertainty associated with the predictions.
• The quantity may also contain unmeasurable variation.

Workflow

Regression Problem

• When the response variable is numeric, we fall into the category of regression (the topic for the next three lectures)

• In the context of regression, the most commonly-used metric for assessing performance is the mean squared error (or MSE)

• In words, the MSE represents the average squared difference of a prediction \(\hat y = \hat f(x)\) from its true value \(y\).

Our measuring stick for closeness in this scenario going to be the average squared difference of a prediction \(\hat y = \hat f(x)\) from its true value \(y\).

MSE

\[\begin{align} \text{MSE} &= \frac{1}{n} \sum_{i=1}^n (y_i-\hat y_i)^2 \end{align}\]

where \(\hat f(x_i) = \hat y_i\) is the prediction for the \(i\)th input \(x_i\), and \(y_i\) is the response actually observed (ie “truth”).

In the simple linear regression context, this was known as the Residual Sum of Squares (or RSS).

• bias error (error from wrong model assumptions),
• variance (error from sensitivity to small fluctuations in training data) and
• irreducible error (inherent noise in the problem itself). it is not affected by model choice.

MSE Visualized

Given some data: \((X, Y)\)

Fit a model \(\hat f\) (plotted in orange)

For each \(x_i\) we have a true value \(y_i\), …

… and predicted value \(\hat f(x_i) = \hat y_i\)

We average the squared differences to get MSE

Properties of MSE

\[MSE = \frac{1}{n} \sum_{i=1}^n (y_i-\hat f(x_i))^2 \]

Notice that this has some desirable properties:

• MSE is small when predicted values , \(\hat y_i - \hat f(x_i)\), are close to the true response, \(y\)

• MSE is large when predicted values, \(\hat y_i - \hat f(x_i)\), are far from the true response, \(y\)

Question: Does anyone see a problem with this definition?

1. This is calculated over the training set, but what about the test set?

2. We could fit a model with 0 MSE (connect-the-dots), but is that desirable?

Hint: think back to the connect the dots example from last lecture 1. we could fit a model with 0 MSE (think about our connect-the-dots example) that would be horrible at prediction. 1. Furthermore, We are testing using observed (past) data when we are interested in predicting unseen (future) data. 1. (advanced answer) many statistical methods specifically estimate coefficients so as to minimize the training set MSE. For these methods, the training set MSE can be quite small, but the test MSE is often much larger. - we want MSE to to be minimal, \(x_{1},\dots ,x_{n}\) and for points outside of our sample

Motivating Question

Question: Would this be a good model?

No!

• even though it has an MSE it wouldn’t be generalizable
• this is OVERFITTING to the data

Splitting

Notation

We denote the training set, i.e. the collection of observations we use to fit our model by:

\[\textbf{X}_{Tr} = \{(x_1, y_1),(x_2, y_2),...,(x_n, y_n)\} = \{x_i, y_i\}_{1}^n\]

We will denote the testing set, i.e. the collection of observations that we keep separate from the fitting process and reserve for assessing the model by:

\[\textbf{X}_{Te} = \{(x_{n+1}, y_{n+1}),...,(x_{n+m}, y_{n+m})\} = \{x_i, y_i\}_{1}^m\]

Model Fitting

• We might find our \(\hat f\) based on training data \(\textbf{X}_{Tr}\) and see how close \(\hat f(x_{n+1},...,x_{n+m})\) predicts \((y_{n+1},...y_{n+m})\)

• Recall that \(\textbf{X}_{Te}\) is not used to train the statistical learning method and has never been “seen” by the algorithm.

Testing vs. Training MSE

When MSE is calculated using \(\textbf{X}_{Tr}\) we call it the training MSE

\[MSE_{Tr} = \frac{1}{n} \sum_{i=1}^n (y_i-\hat f(x_i))^2\] When MSE is calculated using \(\textbf{X}_{Te}\) we call it the test MSE:

\[MSE_{Te} = \frac{1}{m} \sum_{i=1}^m (y_i-\hat f(x_i))^2\]

MSE definitions exist for both sets of data - It is not very useful to have a model that can predict perfectly to data it’s already seen. - Rather, we are interested in the accuracy of the predictions that we obtain when we apply our method to previously unseen test data. - Test MSE mimics just that. It quantifies how well it generalizes to unseen data - eg stocks - don’t care about how it predicts last weeks stock price, only tomorrows - eg clinical measures (eg weight, blood pressure, height, age, family history of disease) + y/n for disease, train model, predict for future patients based on clinical measuresments. not current ones for which we already have the diagnosis for.

MSE train = average squared differences using training data

MSE test = average squared differences using the test data

• for MES test we need some new data
• it is assumed that training and testing data come from the same population
• Green points were generated from the same model
• It is important to emphasize that the test data was not used in the fitting of the orange curve (if it was our fitted line \(\hat f\) would have looked quite different)

Goal

• Plainly speaking, we do not really care how well the method works on the training data.

• eg stocks - don’t care about how it predicts last weeks stock price, only tomorrows
• eg clinical measures for patients that do/do not have a disease. Train data:
  • \(X\): clinical measures: eg weight, blood pressure, height, age, family history of disease)
  • \(Y\): y/n for disease, train model,
• Goal: predict for future patients based on clinical measurements not current ones for which we already have the diagnosis for.

• We are interested in the accuracy of predictions to data (e.g. new patients, future stock prices) it’s never seen before.

• \(\text{MSE}_{Te}\) is a metric of how well \(\hat f\) generalizes to unseen data, thus, we want this (rather than \(\text{MSE}_{Tr}\)) to be as low as possible.

• If \((x_0, y_0)\) is a new[^1] data point we want \(\hat y_0 = \hat f(x_0)\) to be as close to \(y_0\) as possible.
• If \(\hat f\) is chosen well it should generalizes to this “unseen” data
• Test MSE mimics just that. It quantifies how well it generalizes to unseen data

To put our goal another way we want the method that gives the lowest test MSE - Test MSE allows us to quantify how well it generalizes to unseen data

Error types

In general, \(\hat f\) will not match \(f\) perfectly and the discrepancy can be broken down into two types of errors…

1. reducible error which we can reduce by picking a better statistical learning technique; and

2. irreducible error which we can never improve even if we estimate \(f\) perfectly

• This error is reducible because we can potentially improve the accuracy of \(\hat f\) by using the most appropriate statistical learning technique to estimate \(f\)
• irreducible error exists because there will always be some variability in our data which can never be captured introduced by epsilon (e.g., inaccuracies in collection or reporting of data, non-deterministic behaviour, or just any type of noise that cannot be accounted for.
• to put another way, even if we know f we will never be able to predict y with absolute certainty because of the random epsilon term

Math

Suppose, for notational convenience, we have a training set \(X_{Tr} = S\) that was used to fit our model \(\hat f_S\). Given a test set \(X_{Te} = (X, Y)\), we can write:

\[\begin{align*} &= E[(y - \hat y)^2]\\ &= E[(y - \hat f_S(x))^2]\\ &= E[(f(x) + \epsilon - \hat f_S(x))^2]\\ &= E[\left((f(x) - \hat f_S(x)) + \epsilon \right)^2]\\ &= E[(f(x) - \hat f(x))^2 + 2(f(x) - \hat f(x))\epsilon + \epsilon^2]\\ \end{align*}\]

\(x\) and \(\hat f_S(x)\) assumed fixed, \(f\) fixed (but unknown)

\[\begin{align*} &= \left(f(x) - \hat f(x)\right)^2 + 2\left(f(x) - \hat f_S(x)\right)E[\epsilon] + E[\epsilon^2]\\ &= (f(x) - \hat f_S(x))^2 + E[\epsilon^2] \quad \quad {\scriptsize \text{since $E[\epsilon] = 0$ }}\\ \end{align*}\]

Recall \(\text{Var}(X) = E[X^2] - (E[X])^2\).
Hence \(\text{Var}(\epsilon) = E[\epsilon^2] - (E[\epsilon])^2 = E[\epsilon^2] \text{ (since } E[\epsilon] = 0)\)

\[\begin{align*} &= \underbrace{(f(x) - \hat f_S(x))^2}_{\text{reducible error}} + \underbrace{\text{Var}[\epsilon]}_{\text{irreducible error}}\\ \end{align*}\]

• represents the average, or expected value, of the squared expected value difference between the predicted and actual value of\(Y\),

• The expected prediction error is for a random \(Y\) given a fixed \(x\) and a random \(\hat f_S\). As such, the expectation is over \(Y∣X=x\) and \(S\). Our estimated function \(\hat f\) is random depending on the training data \(S\), which is used to perform the estimation.

Math (cont’d)

Reducible error can be further decomposed into error due to squared bias and variance.

\[\begin{align*} &= \mathbb{E}\left[\text{reducible error}\right]\\ &=\mathbb{E}\left[ (f(x) - \hat f_S(x))^2 \right]\\ &= \underbrace{ (f(x) - \mathbb{E}[\hat f_S(x)])^2 }_{\text{Bias}^2(\hat f(x))} + \underbrace{ \mathbb{E}\left[ (\hat f_S(x) - \mathbb{E}[\hat f_S(x)]^2) \right]}_{\text{Var}(\hat f(x))} \end{align*}\]

• Bias occurs when an algorithm produces results that are systemically prejudiced due to erroneous assumptions in the machine learning process.
• High bias usually stems from oversimplifying model assumptions, while high variance from overly complex assumptions.
• When our model suffers from high bias, the average response of the model is far from the true value and we call this underfitting. When our model suffers from high variance, this is usually a result of its inability to generalize well beyond the training data and we call this overfitting. Goal: strike a balance between bias-variance tradeoff.

Reducible error

• The reducible error as its name suggest is the portion of the error in a model that can be reduced or eliminated through improvements in the modeling process.

• Our goal can now be rephrased as minimizing the reducible error (also known as model error) as much as possible.

  • Using a more sophisticated or appropriate model.
  • Collecting more relevant data or improving data quality.
  • Tuning model hyperparameters for better performance.

• It is typically associated with the model’s limitations, assumptions, or simplifications.

other examples: - Feature engineering to select or transform input variables effectively.

Decomposition of MSE

All together we have the expected test MSE at a new point \(x\).
\[\begin{align*} &=\text{E}_{Y|x, S}\left[(y - \hat f (x))^2\right] \\ &=\underbrace{\text{Var}(\hat f(x)) + \text{Bias}^2(\hat f(x))}_{\text{reducible error}} + \underbrace{\text{Var}(\epsilon)}_{\text{irreducible error}} \end{align*}\] It’s the average \(MSE_{Te}\) we’d get if we repeatedly estimated \(f\) using a large number of training sets, and tested at each at \(x\)

The expected test MSE, for a given value \(x\), is decomposed into the sum of three fundamental quantities:

• the variance of \(\hat f(x)\)
• the (squared) bias of \(\hat f(x_0)\) and
• the variance of error term \(\epsilon\).

For a more meticulous look at the math see: R for statistical learning, or tds

Irreducible Error

• The variance of error term \(\epsilon\) is our irreducible error

• The irreducible error is a measure of the amount of noise in our data.

• Even if we estimate \(f\) perfectly, our data will always have some noise that can not be reduced.

Important no matter how good we make our model, our data will have certain amount of noise or irreducible error that can not be removed.

Bias-Variance

Definitions

Bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model.

• High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting)

• erroneous assumptions in the learning algorithm. [1]

• a ruler can never approximate a squiggle

Variance refers to the amount by which \(\hat f\) would change if we estimated it using a different training data set.

The variance is an error from sensitivity to small fluctuations in the training set.Wiki - fitting too close to the data on hand - Ideally the estimate for \(f\) should not vary too much between training sets

• High variance may result from an algorithm modeling the random noise in the training data (overfitting).

Bias vs Overfitting: Generally speaking, - Usually high bias stems from oversimplifying model assumptions. - more flexible models have higher variance. “overfitting refers specifically to the case in which a less flexible model woul have yielded a smaller test MSE”

Target Visualizations

Inspired by Essays by Scott Fortmann-Roe

We can create a graphical visualization of bias and variance using a bulls-eye diagram. DESCRIBE IMAGE - Imagine that the center of the target is a model that perfectly estimates \(f\). - Each hit represents an individual realization of our model, given the chance variability in the training data we gather. - Sometimes we get a “good” training data so we predict very well (close to bullseye) - sometimes we get “bad” training (eg. full of outliers or non-standard values) resulting in poorer predictions.

Low Bias (first row) Models that approximates the real-life problem well will have low bias (hits will be centered around the bullseye)

High Bias (second row) will systematically be “off the mark”

Low variance (first column) indicates that \(\hat f\) would not change much even if we estimated it using a different training data set. (so the hits will all be close together)

High variance (second column) indicates that \(\hat f\) will be very sensitivity to small fluctuations in the training (so the hits will be less close together)

Target Explained

• Low Bias (first row) Models that approximates the real-life problem well will have low bias (hits will be centered around the bullseye)

• High Bias (second row) will systematically be “off the mark”

• Low variance (first column) indicates that \(\hat f\) would not change much even if we estimated it using a different training data set. (so the hits will all be close together)

• High variance (second column) indicates that \(\hat f\) will be very sensitivity to small fluctuations in the training (so the hits will be less close together)

Simulations

Let’s simulate a training set of size \(n = 100\) by uniformly generating \(X\) values between 0 and \(2\pi\), and generate \(Y\) values according to this formula (\(f(x) = \sin(x)\)): \[\begin{align*} Y &= \text{sin}(X) + \epsilon \end{align*}\] where \(\epsilon \sim \text{Normal}(\mu =0, \sigma = 1)\).

• I don’t want to get bogged down in the code in lecture
• lecture focus = concepts
• You will go through the code for this in lab of next week

Training set 1

Training set #1: This is one example of a potential training set

Training set 2

Training set #2: Here’s another…

Generative Model

Since we know \(f\) we can plot as well

IRL we won’t know \(f\)

Low Variance High Bias

• Investing a Low Variance High Bias Model (SLR)
• Spoiler alert: these models will typically underfit

Training set 1

Let’s start by fitting a simple linear regression (SLR) model to training set #1. The resulting fitted line \(\hat f\) is plotted below

Training set 2

Fitting a SLR to training set #2 will produce this (slightly different) fitted line \(\hat f\)

10 training sets

The fitted line for 10 different fits using 10 different training sets.

Low Variance

• If we do this repeatedly on different training sets simulated from the same model described on this slide, you will notice that we don’t get very much variation in our fitted model.

• This model is therefore said to have low variance.

• That is to say, it is not sensitivity to small fluctuations in the training set.

High Bias

High bias refers to a situation where a model makes strong simplifying assumptions about the data, resulting in systematic errors in its predictions.

• this straight line is not flexible enough to capture the curved relationship between these two variables

Models with high bias and underfitting are closely related concepts, and they often go hand in hand.

• Models with high bias are typically too simplistic or not complex enough to capture the underlying patterns in the data.
• High bias is a characteristic of the model itself, indicating that it’s not flexible or expressive enough to fit the data well.
• Models with high bias tend to have poor training performance (i.e., they do not fit the training data well) and poor generalization to new, unseen data.

Underfitting is a consequence of high bias. It refers to the poor performance of a model on both the training and test data due to its inability to capture the data’s underlying patterns.

Underfitting is a common issue in machine learning whereby a model is too simple to capture the underlying patterns in the data.

Generally speaking models that underfit tend to have high bias and low variance

Low Bias High Variance

lets explore a model with low bias but high variance

Green Model

• To demonstrate this concept we’ll use the local polynomial model (using the loess function in R).

• We will call this the “green model”

• Don’t worry if you don’t know what this model is. We are unlikely to cover these in this course — just consider it a relatively flexible model.

• The span argument controls level of flexibility. The “green model” will fit a local polynomial with a lot of flexibility.

• Locally Weighted Least Squares Regression
• span controls the degree of smoothing

Training set 2

Fitting a highly flexible loess model to training set #1 will produce this fitted curve \(\hat f\)

describe the image

Training set 2

Fitting a highly flexible loess model to training set #2 will produce this (very different) fitted curve \(\hat f\)

• flip back and forth and compare the dip/peak at around 2
• explain how this is fitting too closely on the training data

10 training sets

The fitted model for 10 different fits using 10 different training sets.

High Variance

High Variance models tend to to capture noise and random fluctuations in the training data rather than the underlying patterns.

• This “green” model has high variance since it is very sensitivity to small fluctuations in the training set and highly variable from training set to training set.

Models like these are said to be overfitting to the training data. Overfitting tends to result in very low training error and comparitively high testing error (poor generalization)

The terms “overfitting” and “high variance models” are often used interchangeably, but they can represent slightly different perspectives on the same issue.

Overfitting

• Overfitting is primarily a concept that focuses on the relationship between a model and the training data. It occurs when a model fits the training data too closely, capturing not only the underlying patterns but also the noise or random fluctuations in the data.

• overfitting results in very low training error, essentially memorize the training data.

• poor generalization

High Variance - defined more on complexity rather than their relationship with the training data. - will tend to have low training error and low bias

In practice, the two concepts are closely related, and high variance often leads to overfitting.

Low Bias

\[\text{Bias}^2(\hat f(x)) = (f(x) - \mathbb{E}[\hat f_S(x)])^2 \]

• While small changes in training set causes the estimate \(\hat f\) to change considerably, on average, it is capturing the general sine wave nature of this data.

• Thus the green model has low bias.

• Even though a single fitted model corresponds too closely to the training data, on average this model is close the truth.

Low bias and Low variance

lets explore a model with low bias and low variance

Blue Model

• As before we will use the local polynomial model (using the loess function in R) to fit this model.

• We will call this the “blue model”

• We will adjust the span argument to decrease the level of flexibility.

Training set 1

Fitting a loess model with medium flexibility to training set #1 will produce this fitted curve \(\hat f\) plotted below

DESCRIBE IMAGE - comparing this to the truth we can see this is doing a great job of estimating that sine wave

Training set 2

Fitting a loess model with medium flexibility to training set #2 will produce this (different) fitted curve \(\hat f\)

10 training sets

The fitted model for 10 different fits using 10 different training sets.

Low Variance

• If we do this repeatedly on different training sets simulated from the same model described on this slide, you will notice that we don’t get very much variation in our fitted model.

• This model is therefore said to have low variance

• That is to say, it is not sensitivity to small fluctuations in the training set.

Low Bias

• The blue model has low bias.

• That is, on average the estimate is close the truth.

• Unlike the green model (which also has low bias), this model is not overfitting to the data (low bias).

• The blue model strikes a nice balance between low variance and low bias.

Average model across simulations

Let’s explore what the average model looks like for each of these scenarios …

• While the average blue and green model both are generalizable and estimate the true black curve well, remember this is the average of many (50 training sets) and classical (at least in this section of the course), we are just doing this once.
• Later in the course we’ll talk about ensemble methods that build on this idea that the average of a lot of models might be better than one

Goldilocks Principle

Image adapted from MollyMooCrafts

• Bias refers to the error introduced by approximating a real-world problem, which may be complex, by a simplified model.

• Variance refers to the error introduced due to the model’s sensitivity to the random noise or fluctuations in the training data.

• we want to strike a balance between two types of errors that affect the performance of predictive models: variance and bias.

• Achieving the right balance between these two factors is essential for building models that generalize well to new, unseen data.

Bias Variance Tradeoff

The bias–variance tradeoff is the conflict in trying to simultaneously minimize these two sources of error that prevent supervised learning algorithms from generalizing beyond their training set:[1][2][3]

• Since the test MSE is comprised of both bias and variance we want to try to reduce both of them.

• However, as we decrease bias, we tend to increase variance and vice versa.

• We don’t want our models to be too simple (bias) which could lead to underfitting
• at the same time We don’t want our models to be too flexible (variable) which could lead to overfitting
• We want our algorithms to be generalizable
• dont just do well on classifying the pictures of cats and dogs we’ve learnt with
• we want classify new images into cat or dog with good accuracy

Looking Forward

• To manage the variance-bias tradeoff effectively, you can employ techniques such as cross-validation, regularization, feature selection, and ensemble methods (e.g., bagging and boosting), all of which will be discussed throughout the course.

• These techniques help you find the optimal model complexity and reduce overfitting or underfitting, ultimately improving a model’s generalization performance.

ISLR Simulations

if we have time go through these (otherwise skip)

Section 2.2.2

• The following simulations are taken from your ISLR2 textbook.

• Looking at a variety of different data we can gain some general insights on \(\text{MSE}_{Te}\) and \(\text{MSE}_{Tr}\)

• Again, the particulars about the specific models (linear regression and loess smoothing splines) are not important.

• What is important is it understand how the level of flexibility impact the bias and variance of the models (and therefore the MSE).

Figure 2.9 (left plot)

ISLR Figure 2.9 (left plot) Data simulated from \(f\), shown in black. Three estimates of \(f\) are shown: the linear regression line (orang ecurve), and two smoothing spline fits (blue and green curves). jump to right-hand panel

Here is a simulation from the texbook - we have the truth in black - simple model in orange - medium flexible model in blue - very flexible model in green

The methods have varying level of flexibility: 1. orange least flexible 1. medium flexibility 1. - .content-box-green[most flexible]

The more flexible the model, the closer the curve fits to the observed data.

Figure 2.9 (left plot).

Orange curve

• High bias, low variance

• Low Variance the oranage fit would not have much variability from training set to training set

• High Bias it systematically underestimate between 40–80 and overestimate towards the boundaries, for example.

Variance better to see in repeated fits (visuals on this to come)

Figure 2.9 (left plot).

Green curve

• Low bias, high variance

• As the green curve is the most flexible, it matches the training data very closely

• However, it is much more wiggly than \(f\) (ie the “true” generating black curve)

High Variance You can imagine if we simulated more data from this the green curve could look quite different

Overfitting much more complicated than it needs to be

Low Bias you cant visualize this here, but again thinking of averages this model would do OK

Figure 2.9 (left plot).

Blue curve

• Low bias, low variance

• The blue curve strikes the balance between low variance and low bias

• As one may expect, the average fitted curve is quite similar to \(f\) (ie the “true” generating black curve)

• draw similar conclusions as our last simulation…
• You can imagine we can move the span dial up and down to get slightly more wiggly lines than the blue, but less wiggly than the gree

Estimated test MSE

• To estimate the expected \(\text{MSE}_{Tr}\) and \(\text{MSE}_{Te}\) their values over a very large collection of data sets.

• The average training MSE and testing MSE is plotted in gray and red, respectively, as a function of flexibility.

• The flexibility of these models are mapped to numeric values on the \(x\)-axis (lower values indicate less flexibility).

• Squares (🟧, 🟦, 🟩) indicate the MSEs associated with the corresponding orange, blue, and green models.

Figure 2.9 (right)

ILSR Figure 2.9 (right plot) Training MSE (grey curve), test MSE (red curve), and minimum possible test MSE over all methods (dashed line). Squares represent the training and test MSEs for the three fits shown in the left-hand panel.

The irreducible error serves as a limit to the model’s predictive performance because it represents the natural variability and unpredictability in real-world data.

Test MSE

The orange and green models have high \(\text{MSE}_{Te}\) but for different reasons

• orange is underfitting
• green is overfitting

Blue is close to optimal

1. The orange is not flexible enough to estimate \(f\)
2. Green fits too flexible close to the training set and is not generalizable to test data

• Blue (unsurprisingly) has the lowest \(\text{MSE}_{Te}\)

U-shape - Initially \(MSE_{TEST} \downarrow \text{ as flexibility } \uparrow\) - sweet spot (best “blue” model) - \(MSE_{TEST} \uparrow \text{ as flexibility } \uparrow\)

The orange and green models both have high test MSE but for different reasons - orange too simple (not flexible enough) - green overly flexible (fit to close to training)

The blue curve minimizes the test MSE, which should not be surprising given that visually it appears to estimate f the best

ISLR Figure 2.9 (right plot) Test MSE is displayed using the U-shaped (red curve) and the minimum possible test MSE over all methods (dashed line)

Minimizing test MSE

• The horizontal dashed line indicates \(\text{Var}(\epsilon)\), the irreducible error

• This line corresponds to the lowest achievable test MSE among all possible methods.

• Hence, the “blue model” is close to optimal.

The blue curve minimizes the test MSE, which should not be surprising given that visually it appears to estimate f the best

ISLR Figure 2.9 (right plot) Test MSE is displayed using the U-shaped (red curve) and the minimum possible test MSE over all methods (dashed line)

Training MSE

The green curve has the lowest training MSE of all three methods, since it corresponds to the most flexible of the three curves fit in the left-hand panel

\(MSE_{TRAIN} \downarrow \text{ as flexibility } \uparrow\)

This is a fundamental property of statistical learning that holds regardless of the particular data set at hand and regardless of the SL method.

• As we increase flexibility the TRAINING MSE tends to go down
• makes sense since we will tend to fit closer and closer to the training data

Green

• The most flexible green model fits very closely to the training data, therefore MSE training is small.
• This curve much more wiggly than the true f so for new data, the curve does not predict well on average, hence MSE test is large

ISLR Figure 2.9 (right plot) Training MSE is depicted using the gray curve and the minimum possible test MSE over all methods (dashed line)

Training MSE

The orange curve has the highest training MSE of all three methods, since even on the training set, it is not flexible enough to approximate the underlying relationship

ISLR Figure 2.9 (right plot) Training MSE is depicted using the gray curve and the minimum possible test MSE over all methods (dashed line)

Training MSE

The blue curve obtains a similar training and testing MSE.

General Trends

• The U-shaped test MSE is a fundamental property of statistical learning that holds regardless of the particular dataset and statistical method being used.

• As model flexibility increases, the training MSE will decrease, but the test MSE may not.

• When a method yields a small training MSE but a large test MSE, is is said to be overfitting the data.

• We almost always expect the training MSE to be smaller than the test MSE.

• The training MSE declines monotonically as flexibility increases.

• When we overfit the training data, the test MSE will be very large because the supposed patterns that the method found in the training data simply don’t exist in the test data.

• We almost always expect the training MSE to be smaller than the test MSE. because most statistical learning methods either directly or indirectly seek to minimize the training MSE.