Data 311: Machine Learning

Lecture 3: Assessing Models

Dr. Irene Vrbik

University of British Columbia Okanagan

Recap

• Supervised vs Unsupervised: categorized by the presence or absence of output (response) \(y\)

• Regression vs Classification: dictated by if \(y\) is continuous (regression) vs. categorical (classification)

• Goals of Inference vs Prediction

• Training vs Testing

• Last lecture we introduced some notation and terminology that we will be using throughout the course.

• We discussed how different tasks, namely inference vs prediction may lead us to favour certain models over others.

• For inference we would gravitate to perhaps a more simple parametric model since we value interpretibility over prediction power whereas a flexible non-parameteric approach may result in better predictions and would therefore be better suited for prediction.

• Other considerations: time it takes to run, how easy they are to explain, but today we focus on how well they can predict

• Today we discuss the topic of model assessment to address the challenging task of model selection

• Note that regression and classification models will have different metrics.

Outline

In this lecture we will be covering:

• Reducible Error vs Irreducible Error
• Performance Metrics for Regression and Metrics for Classification
• Why Splitting data into a training and test set is important
  • MSE testing vs MSE training
• Decomposition of MSE
• Bias-Variance tradeoff

Statistical Learning Model

\[Y = f(X) + \epsilon\]

• where \(X\) are our inputs,
• \(Y\) is the numeric output (at least in this setting)
• \(\epsilon\) is the error term (independent of \(X\) and with mean 0),
• \(f\) is the systematic information \(X\) provides about \(Y\).

Goal

The primary goal in statistical learning is to estimate \(f(X)\) as accurately as possible so that the model can make reliable predictions of \(Y\) based on \(X\).

Recall our general model for statistical learning

Goal: find an \(\hat f\) that approximates the true function \(f\) as well as possible.

Goal: Develop models to predict an outcome (Y) using input variables (X).

• find an \(\hat f(X)\) “as close as possible” to \(f\)

\(\epsilon\) attributed to factors that are beyond the model’s control.

• may be from unmeasured variables that are useful in predicting \(Y\):sinc ewe don’t measure them, \(f\) cannot use them for its prediction.
• measurement errors
• stochastic or random distribution. This randomness makes it impossible to make perfectly accurate predictions, as there will always be some level of uncertainty associated with the predictions.
• The quantity may also contain unmeasurable variation.

Income (Simulated) Data

(a) ISLR Fig 2.2 (left) The red dots are the observed values of income (in tens of thousands of dollars) and years of education for 30 individuals.

(b) ISLR Fig 2.2 (right) The blue curve represents the true underlying relationship between income and years of education, which is generally unknown

Figure 1

\(f\) is known in this case because the data were simulated)

Visual breakdown

• The blue curve represents \(f\) the true¹ underlying relationship between income (\(y\)) and years of education (\(x\)).

• The black lines represent the error (\(\epsilon\)) associated with each observation.

Note that some errors are positive (if an observation lies above the blue curve) and some are negative (if an observation lies below the curve). Overall, these errors have approximately mean zero.

• shows income versus years of education for 30 individuals.

• The plot suggests income can be predicted using years of education

• \(f\) needs to be estimated from the observed data points.

• In the simulated Income data set, the true function \(f\) is shown as a blue curve in the right panel, but the true function \(f\) is unknown.

• Vertical lines indicate error terms \(\epsilon\)

• Observations vary around the blue curve, with errors averaging to approximately zero.

1. which is generally unknown (but is known in this case because the data were simulated)

Considering more predictors

In general, \(f\) may involve more than one input variable.

ISLR Figure 2.3 Income plotted as a function of years of education and seniority. Here \(f\) is a two-dimensional surface that must be estimated based on the observed data.

notation \(X = (X_1, X_2, \dots, X_p)\)

Estimates

• In essence, statistical learning refers to a set of approaches for estimating \(f\).

• Once we find \(\hat f\),our estimate for \(f\), we may obtain \(\hat Y\)s, estimates for our output \(Y\) based on

  \[ \hat Y = \hat f(X) \]

• The accuracy of \(\hat Y\) depends on two things: the reducible error and the irreducible error.

• There is no one-size-fits-all best model.

• This course aims to introduce you a small subset of the machine learning approaches available each with their own set of limitations.

• Selecting the best approach for a particular problem can be one of the most challenging tasks for a data scientist.

• Today’s lecture will add to this list of considerations when choosing a model in the supervised regression setting …

• we are searching for a model that is “good enough” (remember all models are wrong but some are useful)

• all models will have some predictive error and with many candidate models to choose from, there are a number of considerations that may go into narrowing it down.

• today: look at key theoretical concepts that arise in estimating \(f\)

Reducible Error

Definition: Errors that can be minimized by improving the model (either by choosing a more appropriate model or improving the model’s training process)

Sources: Include model bias (incorrect assumptions, overly simplistic models) and variance (model sensitivity to data fluctuations).

Reduction Methods: Can be reduced by using better algorithms, adjusting model complexity, gathering more data, or refining features.

Goal: Improve model fit to better capture the true relationship between input and output variables.

Reducible error refers to the part of the prediction error that can be reduced by improving the model.

• It arises from inaccuracies in estimating the function \(f(X)\) with \(\hat f(X)\)

• It can be reduced by using better algorithms, adjusting model complexity, gathering more data, or refining features.

• A common source of reducible includes an incorrect model assumption

Irreducible Error

Irreducible error refers to the part of the prediction error that cannot be reduced or eliminated, regardless of the model or method used.

Definition: Errors that cannot be reduced, no matter how the model is improved.

Sources: Inherent noise in the data, measurement errors, or unobserved factors that affect the outcome.

Nature: Represents the variability in the response variable that cannot be explained by the model or predictors.

Impact: Sets a lower bound on the best possible model accuracy

• It is inherent in the variability of the response variable \(Y\) that cannot be explained by the predictors \(X\).

• It represents the inherent noise in the data, measurement errors, or unobserved factors that affect the outcome.

• It is our lower bound in terms of possible model accuracy

Even if we knew \(f\) perfectly, we would still away that this irreducible error

We can improve on the reducible error by picking a better model

Example of Reducible error

iClicker

Which model do you think has more reducible error? A. The purple line B. The green curve

the purple and green lines represent two different statistical learning models (we’ll learn about how to fit them next lecture)

Question: Which model do you think has more reducible error.

Reducible error is the portion of the prediction error in a statistical model that can be decreased by improving the model’s fit to the data.

rephrase Question: Which model do you think is more inaccurate in modeling the true relationship between the input variables (predictors) and the output variable (response).

Comments

• The purple model assumes a straight-line relationship between the predictor (Years of Education) and the response (Income)

• However, the true relationship, as indicated by blue curve, is non-linear and follows a curved pattern.

• Since the linear model (purple) does not capture the curvature it systematically¹ deviates from the true relationship.

• This discrepancy contributes to the reducible error since this error could be reduced by fitting a more appropriate model that accounts for the curve, such as a polynomial model (green curve).

PURPLE systematically deviates: we’d tend to overestimate the income of those with 12 years of education and underestimate the income of those with 18 years of education.

We can improve the accuracy of the PURPLE model by using a more appropriate statistical model (one that allows for a curves)

1. e.g. tend to overestimate the income of those with 12 years of education and underestimate the income of those with 18 years of education

Motivation

• In practice, we won’t know \(f\), so we can’t simply measure how different \(\hat f\) is from \(f\).

In simulation land, it was really easy to compare curves but in the real world we dont know what that \(f\) curve looks like (if we did there’d be no need for ML)

• If our data is labelled, we investigate how \(\hat y\) (our predictions for some output \(y\)) compare to the observed values \(y\).

There is a number of ways this comparison can be made and we’ll be looking at several metrics today

• Naturally, the closer our predicted values are to the true response value, the better.

There is some considerations we will need to make here that deal with training vs. testing but we will get to that for now lets look at our…

• Our Performance Metrics will depend on whether we’re doing regression (where the output \(y\) is numeric) or classification (where the output \(y\) is categorical).

Performance Metrics

Regression

\[\begin{align} RSS &= \sum_i^n (y_i - \hat y)^2 \\ MSE &= \frac{1}{n}\sum_i^n (y_i - \hat y)^2 \\ RMSE &= \sqrt{\frac{1}{n}\sum_i^n (y_i - \hat y)^2} \end{align}\]

Classification

• Misclassification Rate
• Accuracy
• Precision
• Recall¹
• F1-score
• and others …

• Residual Sum of Squares (RSS): Measures the total squared difference between predicted and true values.
• Mean Squared Error (MSE): The average of squared differences between predictions and true values.
• Root Mean Squared Error (RMSE): The square root of MSE, easier to interpret as it’s in the same units as the target variable.

Classification - Accuracy: Proportion of correct predictions. - Precision: Proportion of true positive predictions among all positive predictions. - Recall: Proportion of true positive predictions among all actual positives. - F1 Score: Harmonic mean of precision and recall, balancing both metrics.

Evaluation of Unsupervised Models:

Metrics like homogeneity, silhouette score, and completeness measure how well clusters or patterns are formed in unsupervised learning.

1. AKA “sensitivity” or “True Positive Rate”

Advertising Dataset

The Advertising dataset from the ISLR textbook which consists of the sales of a particular product across 200 different markets, along with advertising budgets for the product in different media: TV, radio, and newspaper.

library(dplyr)
Advertising <- read.csv("https://www.statlearning.com/s/Advertising.csv")
Advertising <- rename(Advertising, market = X) 
head(Advertising)

  market    TV radio newspaper sales
1      1 230.1  37.8      69.2  22.1
2      2  44.5  39.3      45.1  10.4
3      3  17.2  45.9      69.3   9.3
4      4 151.5  41.3      58.5  18.5
5      5 180.8  10.8      58.4  12.9
6      6   8.7  48.9      75.0   7.2

Suppose we wish to investigate the association between advertising and sales of a particular product …

# reduce some white space in the top & right margin
par(mar=c(5.1, 4.1, 0.1, 0.1))

# Set up the plotting area for 3 plots in one row
par(mfrow = c(1, 3))  # 1 row, 3 columns of plots

plot(Advertising$TV, Advertising$sales, 
     xlab = "TV", ylab = "Sales")
     
#  makes the columns available as variables
attach(Advertising)

plot(radio, sales)
plot(newspaper, sales)

The plot displays sales, in thousands of units, as a function of TV, radio, and newspaper budgets, in thousands of dollars, for 200 different markets.

Why This Matters

1. Real-World Impact: The ability to accurately estimate relationships from data, like predicting sales from advertising spend, directly affects decision-making in business, healthcare, and beyond.

2. Challenges in Model Selection: With many models available, choosing the right one is crucial but complex. There is no perfect model—just ones that are “good enough” for the task.

3. Understanding Errors and Assumptions: All models have errors. Learning to understand and manage these errors is key to building useful predictive models.

4. Empowering Data-Driven Decisions: By grasping the theoretical foundations and practical tools of statistical learning, you’ll be better equipped to tackle diverse problems and make informed, data-driven decisions.

Why Estimate \(f\)?

While our performance metrics will help us choose between models, we should also keep in mind the two main reasons that we may wish to estimating \(f\):

• Prediction: Forecasting future outcomes based on inputs.

• Inference: Understand the relationship between variables.

Goal Prediction: In the Advertising dataset, we might want to focus on predicting the sales of product based on a set ad budget.

Goal Inference: Alternatively, we might want to understand how advertising expenditure in each medium influences sales to better allocate our spending.

even if we can predict the sales of a product with high accuracy i still may want a non-black box model that’s better at inference so i can strategically decide on HOW to spend my ad money

Model Training Process

• Data Preparation: Split data into training and test sets.

• Model Training: Use training data to fit the model.

• Hyperparameter Tuning: Adjust model parameters using validation data to minimize errors.

• Model Evaluation: Evaluate the final model using test data to assess how well it generalizes to new, unseen data.

Metrics for Regression

Regression Problem

• When the response variable is numeric, we fall into the category of regression (the topic for the next three lectures)

• The most commonly-used metric for assessing performance is the mean squared error (or MSE)

• In words, the MSE represents the average squared difference of a prediction \(\hat y = \hat f(x)\) from its true value \(y\).

• Note that the MSE is quite sensitive to outliers.¹

Our measuring stick for closeness in this scenario going to be the average squared difference of a prediction \(\hat y = \hat f(x)\) from its true value \(y\).

There are other metrics like the MAE (Mean absolute error) that are less sensitive to outliers but theyre not as common

1. Metrics like the MAE (mean absolute error) are less sensitive to outliers but less commonly used.

MSE vs RSS

\[ \text{MSE} = \frac{1}{n} \sum_{i=1}^n (y_i-\hat y_i)^2 \]

where \(\hat f(x_i) = \hat y_i\) is the prediction for the \(i\)th input \(x_i\), and \(y_i\) is the response actually observed (ie “truth”).

\[ RSS = \sum_{i=1}^n (y_i - \hat y_i)^2 \]

The MSE is closely related to the Residual Sum of Squares (or RSS) commonly in linear regression.

• Residual Sum of Squares (RSS): Measures the total squared difference between the predicted and true values. Lower RSS indicates a better fit.

• Mean Squared Error (MSE): The average of the squared differences between the predicted and true values, making it similar to RSS but averaged. Lower MSE indicates a better fit.

• bias error (error from wrong model assumptions),
• variance (error from sensitivity to small fluctuations in training data) and
• irreducible error (inherent noise in the problem itself). it is not affected by model choice.

MSE vs RMSE

\[ \text{RMSE} = \sqrt{\text{MSE}} = \sqrt{\frac{1}{n} \sum_{i=1}^n (y_i-\hat y_i)^2} \]

• RMSE (Root Mean Squared Error) is the square root of MSE; it expresses the error in the same units as the target variable.
• This makes RMSE easier to interpret and understand compared to MSE, which is in squared units.

Root Mean Squared Error (RMSE): The square root of MSE, which puts the error in the same units as the response variable, making it easier to interpret. Lower RMSE indicates a better fit.

MSE is used when large errors need greater penalization,

There are other ones like MAE

For example, if RMSE is 5, it suggests that, on average, the predictions deviate from the actual values by about 5 units.

Metrics for Classification

Confusion matrix

• When \(y\) is categorical we fall in the classification setting.

• When summarizing our predictions we often look at a confusion matrix.

	Predicted Yes (+)	Predicted No (-)
Actual Yes (+)	\(a\) = True Positive	\(b\) = False Negative
Actual No (-)	\(c\) = False Positive	\(d\) = True Negative

Misclassification Rate

	Predicted Yes (+)	Predicted No(-)
Actual Yes (+)	\(a\) = True Positive	\(b\) = False Negative
Actual No (-)	\(c\) = False Positive	\(d\) = True Negative

\[\begin{align} \text{Misclassification Rate} &= \frac{b+c}{a + b + c+d} = \frac{b+c}{n} \\[0.5cm] &= \frac{\text{incorrect predictions}}{\text{total predictions}} \end{align}\]

Accuracy

	Predicted Yes (+)	Predicted No(-)
Actual Yes (+)	\(a\) = True Positive	\(b\) = False Negative
Actual No (-)	\(c\) = False Positive	\(d\) = True Negative

\[\begin{align} \text{Accuracy} &= \frac{a+d}{a + b + c+d} = \frac{a+d}{n} \\[0.5cm] &= \frac{\text{correct predictions}}{\text{total predictions}} \end{align}\]

AKA error rate: proporotion of observations we misclassified

There are two ways to be wrong:

false negative: Predicting Y=0 when Y = 1

false positive: Predicting Y=1 when Y = 0

Depending on the context we may care more about making mistakes in one of the categories more than the other:

e.g. false negative is bad in medicine since a person who has a disease will not get treatment

e.g. false positive in a spam detectro is bad because we would miss an important email

Precision

	Predicted Yes (+)	Predicted No(-)
Actual Yes (+)	\(a\) = True Positive	\(b\) = False Negative
Actual No (-)	\(c\) = False Positive	\(d\) = True Negative

\[\begin{align} \text{Precision} &= \frac{a}{a + c} = \frac{\text{true positives}}{\text{predicted positives}} \end{align}\]

In other words, precision is the proportion of predicted “Yes”s, that are actually “Yes”

Precision is more suitable when the goal is to ensure that positive predictions are correct

Use precision when avoiding false positives in positive predictions is crucial.

good for spam: High precision ensures that legitimate emails (ham) are not mistakenly marked as spam.

Weakness: Does not account for false negatives, so it may miss a large number of actual positives.

Higher precision suggests that the model is good at avoiding false positives in positive predictions.

Recall (TP Rate/sensitivity)

	Predicted Yes (+)	Predicted No(-)
Actual Yes (+)	\(a\) = True Positive	\(b\) = False Negative
Actual No (-)	\(c\) = False Positive	\(d\) = True Negative

\[\begin{align} \text{Recall} &= \frac{a}{a + b} = \frac{\text{true positives}}{\text{actual positives}} \end{align}\]

In other words, recall (AKA sensitivity or true positive rate) is the proportion of actual “Yes”s that were predicted “Yes”.

• Foocus: Precision measures the accuracy of positive predictions, specifically minimizing false positives.

• When to Use: Precision is important when the cost of false positives is high, and you want to ensure that all predicted positives are correct. For example, in spam detection, a high precision means fewer legitimate emails are marked as spam.

• Strength: Good at avoiding false positives among predicted positives.

• Weakness: Does not account for false negatives, so it may miss a large number of actual positives.

Higher recall suggests that the model is good at avoiding false negatives.

Specificity (TN rate)

	Predicted Yes (+)	Predicted No(-)
Actual Yes (+)	\(a\) = True Positive	\(b\) = False Negative
Actual No (-)	\(c\) = False Positive	\(d\) = True Negative

\[\begin{align} \text{Precision} &= \frac{d}{c + d} = \frac{\text{true negatives}}{\text{actual negatives}} \end{align}\]

In other words, specificity (AKA true negative rate) is the proportion of actual “No”s that were predicted “No”

Tackles the weakness of precision. Use specificity when avoiding false positives among actual negatives is the priority.

• e.g. quality control or screenings for rare conditions where false positives could lead to unnecessary further actions.

• Focus: Specificity measures the ability to correctly identify all actual negatives, minimizing false positives.

• When to Use: Specificity is important when it is crucial to correctly identify negatives, avoiding false positives, such as in quality control or screenings for rare conditions where false positives could lead to unnecessary further actions.

• Strength: Good at avoiding false positives among actual negatives.

• Weakness: Does not account for false negatives, so it may fail to identify many positives if it focuses too much on being specific to negatives.

Key Distinction:

• Precision minimizes false positives among the predicted positives.

• Specificity minimizes false positives among the actual negatives.

Summary:

• Use precision when avoiding false positives in positive predictions is crucial.

• Use recall when avoiding false negatives and capturing all positives is critical.

• Use specificity when avoiding false positives among actual negatives is the priority.

Higher specificity suggests that the model is good at avoiding false positives among actual negatives.

F1

A popular secondary measure is the F1 score. It is the harmonic mean¹ between Precision and Recall.

\[ \text{F1} = \dfrac{2 \times \text{Precision} \times \text{Recall}} {\text{Precision} + \text{Recall}} \]

Lots of work has shown that F1 is more reliable than mis/classification rates for summarizing performance on unbalanced data sets.

We’ll return to these metrics when we start talking about classification

• Recall and precision are often in tension with each other; increasing one metric may lead to a decrease in the other. Specificity is not typically in direct tension with recall or precision.

• Why? I don’t want to make big mistakes in either direction

• Harmonic Mean since it penalizes the extreme values.

• click the link to wiki if you want to see harmonic mean more generally defined, but for our purpose we only need it for 2 values (precision and recall).

1. Harmonic mean between \(x\) and \(y\): \(\dfrac{2xy}{x+y}\)

MSE Visualized

Given some data: \((X, Y)\)

Fit a model \(\hat f\) (plotted in orange)

For each \(x_i\) we have a true value \(y_i\), …

… and predicted value \(\hat f(x_i) = \hat y_i\)

We average the squared differences to get MSE

Properties of MSE

\[MSE = \frac{1}{n} \sum_{i=1}^n (y_i-\hat f(x_i))^2 \]

Notice that this has some desirable properties:

• MSE is small when predicted values , \(\hat y_i - \hat f(x_i)\), are close to the true response, \(y\)

• MSE is large when predicted values, \(\hat y_i - \hat f(x_i)\), are far from the true response, \(y\)

Question: Does anyone see a problem with this definition?

1. This is calculated over the training set, but what about the test set?

2. We could fit a model with 0 MSE (connect-the-dots), but is that desirable?

Hint: think back to the connect the dots example from last lecture 1. we could fit a model with 0 MSE (think about our connect-the-dots example) that would be horrible at prediction. 1. Furthermore, We are testing using observed (past) data when we are interested in predicting unseen (future) data. 1. (advanced answer) many statistical methods specifically estimate coefficients so as to minimize the training set MSE. For these methods, the training set MSE can be quite small, but the test MSE is often much larger. - we want MSE to to be minimal, \(x_{1},\dots ,x_{n}\) and for points outside of our sample

Discussion Question

Question: Would this be a good model?

Give students 5 minutes to discuss with their neighbour

No!

• even though it has an MSE it wouldn’t be generalizable
• this is OVERFITTING to the data

Splitting

Notation

We denote the training set, i.e. the collection of observations we use to fit our model by:

\[\textbf{X}_{Tr} = \{(x_1, y_1),(x_2, y_2),...,(x_n, y_n)\} = \{x_i, y_i\}_{1}^n\]

We will denote the testing set, i.e. the collection of observations that we keep separate from the fitting process and reserve for assessing the model by:

\[\textbf{X}_{Te} = \{(x_{n+1}, y_{n+1}),...,(x_{n+m}, y_{n+m})\} = \{x_i, y_i\}_{1}^m\]

Model Fitting

• We might find our \(\hat f\) based on training data \(\textbf{X}_{Tr}\) and see how close \(\hat f(x_{n+1},...,x_{n+m})\) predicts \((y_{n+1},...y_{n+m})\)

• Recall that \(\textbf{X}_{Te}\) is not used to train the statistical learning method and has never been “seen” by the algorithm.

Testing vs. Training MSE

When MSE is calculated using \(\textbf{X}_{Tr}\) we call it the training MSE

\[MSE_{Tr} = \frac{1}{n} \sum_{i=1}^n (y_i-\hat f(x_i))^2\] When MSE is calculated using \(\textbf{X}_{Te}\) we call it the test MSE:

\[MSE_{Te} = \frac{1}{m} \sum_{i=1}^m (y_i-\hat f(x_i))^2\]

MSE definitions exist for both sets of data

• It is not very useful to have a model that can predict perfectly to data it’s already seen.

• Rather, we are interested in the accuracy of the predictions that we obtain when we apply our method to previously unseen test data.

• Test MSE mimics just that. It quantifies how well it generalizes to unseen data

  • eg stocks - don’t care about how it predicts last weeks stock price, only tomorrows

  • eg clinical measures (eg weight, blood pressure, height, age, family history of disease) + y/n for disease, train model, predict for future patients based on clinical measuresments. not current ones for which we already have the diagnosis for.

MSE train = average squared differences using training data

MSE test = average squared differences using the test data

• for MES test we need some new data
• it is assumed that training and testing data come from the same population
• Green points were generated from the same model
• It is important to emphasize that the test data was not used in the fitting of the orange curve (if it was our fitted line \(\hat f\) would have looked quite different)

iClicker

iClicker

Which of the following statements is generally true about MSE (Mean Squared Error) on the training and testing sets?

1. MSE on the training set tends to be smaller than on the testing set.

2. MSE on the testing set tends to be smaller than on the training set.

3. MSE on both sets should be equal for a good model.

4. MSE on the training set tends to be larger than on the testing set.

Correct answer: A. MSE on the testing set is typically larger than MSE on the training set because the model is optimized and fitted to the training data, often leading to better performance on that data compared to unseen testing data.

Goal

• Plainly speaking, we do not really care how well the method works on the training data.

• eg stocks - don’t care about how it predicts last weeks stock price, only tomorrows
• eg clinical measures for patients that do/do not have a disease. Train data:
  • \(X\): clinical measures: eg weight, blood pressure, height, age, family history of disease)
  • \(Y\): y/n for disease, train model,
• Goal: predict for future patients based on clinical measurements not current ones for which we already have the diagnosis for.

• We are interested in the accuracy of predictions to data it’s never seen before, e.g. new patients, future stock prices, …

MSE testing vs MSE training

\(\text{MSE}_{Te}\) is a metric of how well \(\hat f\) generalizes to unseen data, thus, we want this (rather than \(\text{MSE}_{Tr}\)) to be as low as possible.

• If \((x_0, y_0)\) is a new[^1] data point we want \(\hat y_0 = \hat f(x_0)\) to be as close to \(y_0\) as possible.
• If \(\hat f\) is chosen well it should generalizes to this “unseen” data
• Test MSE mimics just that. It quantifies how well it generalizes to unseen data

To put our goal another way we want the method that gives the lowest test MSE - Test MSE allows us to quantify how well it generalizes to unseen data

Math

Suppose, for notational convenience, we have a training set \(X_{Tr} = S\) that was used to fit our model \(\hat f_S\). Given a test set \(X_{Te} = (X, Y)\), we can write our expected prediction error as:

\[\begin{align*} &= E[(y - \hat y)^2]\\ &= E[(y - \hat f_S(x))^2]\\ &= E[(f(x) + \epsilon - \hat f_S(x))^2]\\ &= E[\left((f(x) - \hat f_S(x)) + \epsilon \right)^2]\\ &= E[(f(x) - \hat f(x))^2 + 2(f(x) - \hat f(x))\epsilon + \epsilon^2]\\ &= E[(f(x) - \hat f(x))^2] + 2E[(f(x) - \hat f(x))\epsilon] + E[\epsilon^2]\\ \end{align*}\]

Since \(\epsilon\) and \(\hat f(x)\) are independent; recall \(E[XY] = E[X]\cdot E[Y]\) when \(X\perp Y\)

\[\begin{align*} &= E[(f(x) - \hat f(x))^2] + 2E[(f(x) - \hat f(x))]E[\epsilon] + E[\epsilon^2]\\ \end{align*}\]

Recall an assumption of Statistical learning models: \(E[\epsilon] =0\)

\[\begin{align*} &= E[(f(x) - \hat f(x))^2] + E[\epsilon^2]\\ \end{align*}\]

Since \(\text{Var}(\epsilon) = E[\epsilon^2] - (E[\epsilon])^2 = E[\epsilon^2] \text{ (since } E[\epsilon] = 0)\)

\[\begin{align*} &= E[\underbrace{(f(x) - \hat f_S(x))^2}_{\text{reducible error}}] + \underbrace{\text{Var}[\epsilon]}_{\text{irreducible error}}\\ \end{align*}\]

This is the expected prediction error (see here)

where the expectation taken over all sources of randomness involved in the modeling process, including the randomness in the data and the randomness in the model’s training procedure.

This shows us mathematically that errors in our predictions boil down to two sources: reducible error (which we can control by our fitted model) and our irreducible error (which we have no control over).

• represents the average, or expected value, of the squared expected value difference between the predicted and actual value of\(Y\),

• The expected prediction error is for a random \(Y\) given a fixed \(x\) and a random \(\hat f_S\). As such, the expectation is over \(Y∣X=x\) and \(S\). Our estimated function \(\hat f\) is random depending on the training data \(S\), which is used to perform the estimation.

For more details I suggest: https://daviddalpiaz.github.io/r4sl/biasvariance-tradeoff.html#biasvariance-tradeoff

Math (cont’d)

Reducible error can be further decomposed into error due to squared bias and variance.

\[\begin{align*} &= \mathbb{E}\left[\text{reducible error}\right]\\ &=\mathbb{E}\left[ (f(x) - \hat f_S(x))^2 \right]\\ &= \underbrace{ (f(x) - \mathbb{E}[\hat f_S(x)])^2 }_{\text{Bias}^2(\hat f(x))} + \underbrace{ \mathbb{E}\left[ (\hat f_S(x) - \mathbb{E}[\hat f_S(x)]^2) \right]}_{\text{Var}(\hat f(x))} \end{align*}\]

• Bias occurs when an algorithm produces results that are systemically prejudiced due to erroneous assumptions in the machine learning process.
• High bias usually stems from oversimplifying model assumptions, while high variance from overly complex assumptions.
• When our model suffers from high bias, the average response of the model is far from the true value and we call this underfitting. When our model suffers from high variance, this is usually a result of its inability to generalize well beyond the training data and we call this overfitting. Goal: strike a balance between bias-variance tradeoff.

Again since \(f\) is assumed fixed.

Recall \(\text{Bias}(\hat\theta) = E[\hat \theta] - \theta\) and \(\text{Var}(X) = E[(X - E[X]^2)]\)

Decomposition of MSE

All together we have the expected test MSE at a new point \(x\).
\[\begin{align*} &=\text{E}_{Y|x, S}\left[(y - \hat f (x))^2\right] \\ &=\underbrace{\text{Bias}^2(\hat f(x)) + \text{Var}(\hat f(x))}_{\text{reducible error}} + \underbrace{\text{Var}(\epsilon)}_{\text{irreducible error}} \end{align*}\] It’s the average \(MSE_{Te}\) we’d get if we repeatedly estimated \(f\) using a large number of training sets, and tested at each at \(x\)

The expected test MSE, for a given value \(x\), is decomposed into the sum of three fundamental quantities:

• the variance of \(\hat f(x)\)
• the (squared) bias of \(\hat f(x_0)\) and
• the variance of error term \(\epsilon\).

For a more meticulous look at the math see: R for statistical learning, or tds

Types of Error revisted

• The variance of error term \(\epsilon\) is our irreducible error.

  • a measure of the amount of noise in our data.
  • even if we estimate \(f\) perfectly, our data will always have some noise that can not be reduced.

• This shows us that the reducible error (as defined earlier) is composed of bias and variance components.

• Our goal can be rephrased as minimizing the reducible error by decreasing the bias and variance as much as possible

• The reducible error as its name suggest is the portion of the error in a model that can be reduced or eliminated through improvements in the modeling process.

  • Using a more sophisticated or appropriate model.
  • Collecting more relevant data or improving data quality.
  • Tuning model hyperparameters for better performance.

• It is typically associated with the model’s limitations, assumptions, or simplifications.

other examples: - Feature engineering to select or transform input variables effectively.

No matter how good we make our model, our data will have certain amount of noise or irreducible error that can not be removed.

Problem

• Problem: as we tend to decrease the bias, we tend to increase the variance and vice versa.

• There is an inherent trade-off between bias and variance is known as the Bias Variance Tradeoff

• Understanding these components (bias/variance a model) will help us in selecting and tuning models to improve their performance on new, unseen data.

Bias-Variance

Bias

Bias (of a model) refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model.

• High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting)

Figure 2: high bias

Figure 3: low bias

• Imagine that the center of the target is a model that perfectly estimates \(f\).

• Each hit represents an individual realization of our model, given the chance variability in the training data we gather.

Mathematically, the bias of an estimator \(\hat{\theta}\) for a parameter \(\theta\) is defined as:

\[ \text{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - {\theta} \]

• Interpretation: An estimator is considered unbiased if the expected value of the estimator equals the true parameter value. Otherwise, it is biased, meaning it systematically overestimates or underestimates the parameter.

• Example: If you’re estimating the mean of a population using the sample mean, and the expected value of the sample mean equals the true population mean, then the sample mean is an unbiased estimator.

• Sometimes we get a “good” training data so we predict very well (close to bulls eye)

• sometimes we get “bad” training (eg. full of outliers or non-standard values) resulting in poorer predictions.

• erroneous assumptions in the learning algorithm. [1]

• a ruler can never approximate a squiggle

• Example: A linear regression model might have high bias when used on data with a nonlinear relationship because it assumes a linear relationship between variables, thus leading to significant errors.

Variance

Variance refers to the amount by which \(\hat f\) would change if we estimated it using a different training data set.

The variance is an error from sensitivity to small fluctuations in the training set.Wiki - fitting too close to the data on hand - Ideally the estimate for \(f\) should not vary too much between training sets

• High variance may result from an algorithm modeling the random noise in the training data (overfitting).

Figure 4: low variance

Figure 5: high variance

Most of us are probably familiar with the bias of a parameter \(\text{Bias}(\hat\theta) = E[\hat \theta] - \theta\) ; the systematic error in the estimation of a model’s parameter(s)

Similarly we can define the bias associated with a model

Bias vs Overfitting: Generally speaking, - Usually high bias stems from oversimplifying model assumptions. - more flexible models have higher variance. “overfitting refers specifically to the case in which a less flexible model woul have yielded a smaller test MSE”

Target Visualizations

Inspired by Essays by Scott Fortmann-Roe

• Imagine that the center of the target is a model that perfectly estimates \(f\).

• Each hit represents an individual realization of our model, given the chance variability in the training data we gather.

• Sometimes we get a “good” training data so we predict very well (close to bullseye)

• sometimes we get “bad” training (eg. full of outliers or non-standard values) resulting in poorer predictions.

Low Bias (first row) Models that approximates the real-life problem well will have low bias (hits will be centered around the bullseye)

High Bias (second row) will systematically be “off the mark”

Low variance (first column) indicates that \(\hat f\) would not change much even if we estimated it using a different training data set. (so the hits will all be close together)

High variance (second column) indicates that \(\hat f\) will be very sensitivity to small fluctuations in the training (so the hits will be less close together)

Target Explained (Bias)

• Low Bias (first row) models approximate the real-life problem well; that is \(\hat f\) will be centered around \(f\) (🎯)
• High Bias (second row) will systematically be “off the mark”; these tend to be models that are too simple

Impact High bias leads to underfitting, where the model performs poorly on both the training set and new, unseen data. It fails to learn the true relationships in the data adequately.

Example: A linear regression model used to fit data with a clear quadratic pattern will have high bias because the model’s simplicity cannot capture the true relationship.

Target Explained (Variance)

• Low variance (first column) indicates that \(\hat f\) would not change much even if we estimated it using a different training data set. (so the hits will all be close together)

• High variance (second column) indicates that \(\hat f\) will be very sensitivity to small fluctuations in the training; tend to be models that are overly complex

Impact High variance leads to overfitting, where the model performs well on the training data but poorly on new, unseen data because it is too closely fitted to the quirks of the training set.

Example: A decision tree model that is allowed to grow without constraint may fit the training data perfectly, capturing all small variations, but will struggle to make accurate predictions on new data.

Simulations

Let’s simulate a training set of size \(n = 100\) by uniformly generating \(X\) values between 0 and \(2\pi\), and generate \(Y\) values according to this formula (\(f(x) = \sin(x)\)): \[\begin{align*} Y &= \text{sin}(X) + \epsilon \end{align*}\] where \(\epsilon \sim \text{Normal}(\mu =0, \sigma = 1)\).

• I don’t want to get bogged down in the code in lecture
• lecture focus = concepts
• You will go through the code for this in lab of next week

Training set 1

Training set #1: This is one example of a potential training set

Training set 2

Training set #2: Here’s another…

Generative Model

Since we know \(f\) we can plot as well

IRL we won’t know \(f\)

Low Variance High Bias

• Investing a Low Variance High Bias Model (SLR)
• Spoiler alert: these models will typically underfit

Training set 1

Let’s start by fitting a simple linear regression (SLR) model to training set #1. The resulting fitted line \(\hat f\) is plotted below

Training set 2

Fitting a SLR to training set #2 will produce this (slightly different) fitted line \(\hat f\)

10 training sets

The fitted line for 10 different fits using 10 different training sets.

Low Variance

• If we do this repeatedly on different training sets simulated from the same model described on this slide, you will notice that we don’t get very much variation in our fitted model.

• This model is therefore said to have low variance.

• That is to say, it is not sensitivity to small fluctuations in the training set.

High Bias

High bias refers to a situation where a model makes strong simplifying assumptions about the data, resulting in systematic errors in its predictions.

• this straight line is not flexible enough to capture the curved relationship between these two variables

Models with high bias and underfitting are closely related concepts, and they often go hand in hand.

• Models with high bias are typically too simplistic or not complex enough to capture the underlying patterns in the data.
• High bias is a characteristic of the model itself, indicating that it’s not flexible or expressive enough to fit the data well.
• Models with high bias tend to have poor training performance (i.e., they do not fit the training data well) and poor generalization to new, unseen data.

Underfitting is a consequence of high bias. It refers to the poor performance of a model on both the training and test data due to its inability to capture the data’s underlying patterns.

Underfitting is a common issue in machine learning whereby a model is too simple to capture the underlying patterns in the data.

Generally speaking models that underfit tend to have high bias and low variance

Low Bias High Variance

lets explore a model with low bias but high variance

Green Model

• To demonstrate this concept we’ll use the local polynomial model (using the loess function in R).

• We will call this the “green model”

• Don’t worry if you don’t know what this model is. We are unlikely to cover these in this course — just consider it a relatively flexible model.

• Locally Weighted Least Squares Regression
• span controls the degree of smoothing

The span argument controls level of flexibility. The “green model” will fit a local polynomial with a lot of flexibility.

Training set 1

Fitting a highly flexible loess model to training set #1 will produce this fitted curve \(\hat f\)

describe the image

Training set 2

Fitting a highly flexible loess model to training set #2 will produce this (very different) fitted curve \(\hat f\)

• flip back and forth and compare the dip/peak at around 2
• explain how this is fitting too closely on the training data

10 training sets

The fitted model for 10 different fits using 10 different training sets.

High Variance

High Variance models tend to to capture noise and random fluctuations in the training data rather than the underlying patterns.

• This “green” model has high variance since it is very sensitivity to small fluctuations in the training set and highly variable from training set to training set.

Models like these are said to be overfitting to the training data. Overfitting tends to result in very low training error and comparitively high testing error (poor generalization)

The terms “overfitting” and “high variance models” are often used interchangeably, but they can represent slightly different perspectives on the same issue.

Overfitting

• Overfitting is primarily a concept that focuses on the relationship between a model and the training data. It occurs when a model fits the training data too closely, capturing not only the underlying patterns but also the noise or random fluctuations in the data.

• overfitting results in very low training error, essentially memorize the training data.

• poor generalization

High Variance - defined more on complexity rather than their relationship with the training data. - will tend to have low training error and low bias

In practice, the two concepts are closely related, and high variance often leads to overfitting.

Low Bias

\[\text{Bias}^2(\hat f(x)) = (f(x) - \mathbb{E}[\hat f_S(x)])^2 \]

• While small changes in training set causes the estimate \(\hat f\) to change considerably, on average, it is capturing the general sine wave nature of this data.

• Thus the green model has low bias.

• Even though a single fitted model corresponds too closely to the training data, on average this model is close the truth.

Low bias and Low variance

lets explore a model with low bias and low variance

Blue Model

• As before we will use the local polynomial model (using the loess function in R) to fit this model.

• We will call this the “blue model”

• We will adjust the span argument to decrease the level of flexibility.

Training set 1

Fitting a loess model with medium flexibility to training set #1 will produce this fitted curve \(\hat f\) plotted below

DESCRIBE IMAGE - comparing this to the truth we can see this is doing a great job of estimating that sine wave

Training set 2

Fitting a loess model with medium flexibility to training set #2 will produce this (different) fitted curve \(\hat f\)

10 training sets

The fitted model for 10 different fits using 10 different training sets.

Low Variance

• If we do this repeatedly on different training sets simulated from the same model described on this slide, you will notice that we don’t get very much variation in our fitted model.

• This model is therefore said to have low variance

• That is to say, it is not sensitivity to small fluctuations in the training set.

Low Bias

• The blue model has low bias.

• That is, on average the estimate is close the truth.

• Unlike the green model (which also has low bias), this model is not overfitting to the data (low bias).

• The blue model strikes a nice balance between low variance and low bias.

Average model across simulations

Let’s explore what the average model looks like for each of these scenarios …

• While the average blue and green model both are generalizable and estimate the true black curve well, remember this is the average of many (50 training sets) and classical (at least in this section of the course), we are just doing this once.
• Later in the course we’ll talk about ensemble methods that build on this idea that the average of a lot of models might be better than one

Goldilocks Principle

Image adapted from MollyMooCrafts

• Bias refers to the error introduced by approximating a real-world problem, which may be complex, by a simplified model.

• Variance refers to the error introduced due to the model’s sensitivity to the random noise or fluctuations in the training data.

• we want to strike a balance between two types of errors that affect the performance of predictive models: variance and bias.

• Achieving the right balance between these two factors is essential for building models that generalize well to new, unseen data.

Bias Variance Tradeoff

Bias-variance Tradeoff

The bias–variance tradeoff is the conflict in trying to simultaneously minimize these two sources of error that prevent supervised learning algorithms from generalizing beyond their training set:[1][2][3]

• Since the test MSE is comprised of both bias and variance we want to try to reduce both of them.

• However, as we decrease bias, we tend to increase variance and vice versa.

• We don’t want our models to be too simple (bias) which could lead to underfitting
• at the same time We don’t want our models to be too flexible (variable) which could lead to overfitting
• We want our algorithms to be generalizable
• dont just do well on classifying the pictures of cats and dogs we’ve learnt with
• we want classify new images into cat or dog with good accuracy

iClicker

iClicker

iClicker

What does high bias in a model typically lead to?

A. Overfitting

B. Underfitting

C. Good generalization

D. High variance

Correct answer: B. Underfitting

iClicker

iClicker

Which of the following best describes a high variance model?

A. The model performs well on new data.

B. The model has low flexibility.

C. The model captures noise in the training data.

D. The model makes strong assumptions about the data.

Correct answer: C. The model captures noise in the training data.

iClicker

iClicker

What is the primary goal when managing the bias-variance tradeoff?

A. To maximize bias and minimize variance.

B. To eliminate variance completely.

C. To minimize both bias and variance equally.

D. To find a balance that minimizes overall prediction error.

Correct answer: D. To find a balance that minimizes overall prediction error.

iClicker

iClicker

As model flexibility increase, the bias tends to __________________

A. decrease

B. increase

C. stay the same

D. it depends

Correct answer: A. decrease

iClicker

iClicker

As model flexibility increase, the variance tends to __________________

A. decrease

B. increase

C. stay the same

D. it depends

Correct answer: B. increase

Bias-Variance Tradeoff visualized

iClicker

iClicker

As model flexibility increase, the MSE training will tends to __________________

A. decrease

B. increase

C. stay the same

D. it depends

Correct answer: A. decrease

iClicker

iClicker

As model flexibility increase, the MSE test will tends to __________________

A. decrease

B. increase

C. stay the same

D. it depends

Correct answer: D. it depends

MSE visualized

Looking Forward

• To manage the variance-bias tradeoff effectively, you can employ techniques such as cross-validation, regularization, feature selection, and ensemble methods (e.g., bagging and boosting), all of which will be discussed throughout the course.

• These techniques help you find the optimal model complexity and reduce overfitting or underfitting, ultimately improving a model’s generalization performance.

ISLR Simulations

The ISLR textbook (section 2.2.2) has similar simulations that I encourage to go through on your own. You can skim through the slides here which we will summarize in class only if time permits.

if we have time go through these (otherwise skip)

Section 2.2.2

• The following simulations are taken from your ISLR2 textbook.

• Looking at a variety of different data we can gain some general insights on \(\text{MSE}_{Te}\) and \(\text{MSE}_{Tr}\)

• Again, the particulars about the specific models (linear regression and loess smoothing splines) are not important.

• What is important is it understand how the level of flexibility impact the bias and variance of the models (and therefore the MSE).

Figure 2.9 (left plot)

ISLR Figure 2.9 (left plot) Data simulated from \(f\), shown in black. Three estimates of \(f\) are shown: the linear regression line (orang ecurve), and two smoothing spline fits (blue and green curves). jump to right-hand panel

Here is a simulation from the texbook - we have the truth in black - simple model in orange - medium flexible model in blue - very flexible model in green

The methods have varying level of flexibility: 1. orange least flexible 1. medium flexibility 1. - .content-box-green[most flexible]

The more flexible the model, the closer the curve fits to the observed data.

Figure 2.9 (left plot).

Orange curve

• High bias, low variance

• Low Variance the oranage fit would not have much variability from training set to training set

• High Bias it systematically underestimate between 40–80 and overestimate towards the boundaries, for example.

Variance better to see in repeated fits (visuals on this to come)

Figure 2.9 (left plot).

Green curve

• Low bias, high variance

• As the green curve is the most flexible, it matches the training data very closely

• However, it is much more wiggly than \(f\) (ie the “true” generating black curve)

High Variance You can imagine if we simulated more data from this the green curve could look quite different

Overfitting much more complicated than it needs to be

Low Bias you cant visualize this here, but again thinking of averages this model would do OK

Figure 2.9 (left plot).

Blue curve

• Low bias, low variance

• The blue curve strikes the balance between low variance and low bias

• As one may expect, the average fitted curve is quite similar to \(f\) (ie the “true” generating black curve)

• draw similar conclusions as our last simulation…
• You can imagine we can move the span dial up and down to get slightly more wiggly lines than the blue, but less wiggly than the gree

Estimated test MSE

• To estimate the expected \(\text{MSE}_{Tr}\) and \(\text{MSE}_{Te}\) their values over a very large collection of data sets.

• The average training MSE and testing MSE is plotted in gray and red, respectively, as a function of flexibility.

• The flexibility of these models are mapped to numeric values on the \(x\)-axis (lower values indicate less flexibility).

• Squares (🟧, 🟦, 🟩) indicate the MSEs associated with the corresponding orange, blue, and green models.

Figure 2.9 (right)

ILSR Figure 2.9 (right plot) Training MSE (grey curve), test MSE (red curve), and minimum possible test MSE over all methods (dashed line). Squares represent the training and test MSEs for the three fits shown in the left-hand panel.

The irreducible error serves as a limit to the model’s predictive performance because it represents the natural variability and unpredictability in real-world data.

Test MSE

The orange and green models have high \(\text{MSE}_{Te}\) but for different reasons

• orange is underfitting
• green is overfitting

Blue is close to optimal

1. The orange is not flexible enough to estimate \(f\)
2. Green fits too flexible close to the training set and is not generalizable to test data

• Blue (unsurprisingly) has the lowest \(\text{MSE}_{Te}\)

U-shape - Initially \(MSE_{TEST} \downarrow \text{ as flexibility } \uparrow\) - sweet spot (best “blue” model) - \(MSE_{TEST} \uparrow \text{ as flexibility } \uparrow\)

The orange and green models both have high test MSE but for different reasons - orange too simple (not flexible enough) - green overly flexible (fit to close to training)

The blue curve minimizes the test MSE, which should not be surprising given that visually it appears to estimate f the best

ISLR Figure 2.9 (right plot) Test MSE is displayed using the U-shaped (red curve) and the minimum possible test MSE over all methods (dashed line)

Minimizing test MSE

• The horizontal dashed line indicates \(\text{Var}(\epsilon)\), the irreducible error

• This line corresponds to the lowest achievable test MSE among all possible methods.

• Hence, the “blue model” is close to optimal.

The blue curve minimizes the test MSE, which should not be surprising given that visually it appears to estimate f the best

ISLR Figure 2.9 (right plot) Test MSE is displayed using the U-shaped (red curve) and the minimum possible test MSE over all methods (dashed line)

Training MSE

The green curve has the lowest training MSE of all three methods, since it corresponds to the most flexible of the three curves fit in the left-hand panel

\(MSE_{TRAIN} \downarrow \text{ as flexibility } \uparrow\)

This is a fundamental property of statistical learning that holds regardless of the particular data set at hand and regardless of the SL method.

• As we increase flexibility the TRAINING MSE tends to go down
• makes sense since we will tend to fit closer and closer to the training data

Green

• The most flexible green model fits very closely to the training data, therefore MSE training is small.
• This curve much more wiggly than the true f so for new data, the curve does not predict well on average, hence MSE test is large

ISLR Figure 2.9 (right plot) Training MSE is depicted using the gray curve and the minimum possible test MSE over all methods (dashed line)

Training MSE

The orange curve has the highest training MSE of all three methods, since even on the training set, it is not flexible enough to approximate the underlying relationship

ISLR Figure 2.9 (right plot) Training MSE is depicted using the gray curve and the minimum possible test MSE over all methods (dashed line)

Training MSE

The blue curve obtains a similar training and testing MSE.

iClicker

iClicker

If the MSE on the training set is much lower than the MSE on the testing set, what does this indicate about the model?

A. The model is underfitting.

B. The model is overfitting.

C. The model is well-generalized.

D. The model has high bias.

Correct answer: B. The model is overfitting.

General Trends

• The U-shaped test MSE is a fundamental property of statistical learning that holds regardless of the particular dataset and statistical method being used.

• As model flexibility increases, the training MSE will decrease, but the test MSE may not.

• When a method yields a small training MSE but a large test MSE, is is said to be overfitting the data.

• We almost always expect the training MSE to be smaller than the test MSE.

• The training MSE declines monotonically as flexibility increases.

• When we overfit the training data, the test MSE will be very large because the supposed patterns that the method found in the training data simply don’t exist in the test data.

• We almost always expect the training MSE to be smaller than the test MSE. because most statistical learning methods either directly or indirectly seek to minimize the training MSE.