Data 311: Machine Learning

Lecture 2: Notation and Terminology

Dr. Irene Vrbik

University of British Columbia Okanagan

Quote

All models are wrong but some are useful.

• George Edward Pelham Box

Image take from [1]

Image from ILSR

• a famous quote often attributed to the British statistician George E. P. Box

• ILSR authors argue that underlying distribution is never truly linear, though linear regression is extremely useful both conceptually and practically.

• e.g. if we try to predict the number of tourist in Niagara Falls on a given day, will be wrong more that it is right and it will never represent the exact real behaviour. Having said that, even if a model cannot describe exactly the reality it could be very helpful if it is close enough.

Overview

• Statistical vs. Machine learning
• Prediction vs. Inference
• Supervised vs Unsupervised
• Classification vs. Regression
• Notation¹
• Training vs. Testing

Today we will be looking at notation that may or may not differ from those used in courses you have previously taken in Statistics.

1. as used in ISLR2 Notation and Simple Matrix Algebra (pg 9–12).

What is Statistical Learning?

This is the first two lines from your textbook Introduction to Statistical Learning, Ed. 2, I will henceforth refer to as ISLR2.

“Statistical learning refers to a vast set of tools for understanding data. These tools can be classified as supervised or unsupervised.”

Why is our textbook SL vs ML

Statistical vs. Machine learning

¹

1. Some discussion on the topic: [1], [2] [3], blogs: [4], [5], stackexchange: [6], videos: [7] [8], University websites [8], towardsdatascience [9]

Statistical vs. Machine learning

• Last class, we described ML as systems that learn without being explicitly programmed.

• In the previous example, I can write a program that says, may X to Y where Y = 2*X + 5. A regression model, will automatically learn a slope and intercept (i.e. the program that maps x to y) from data.

• so in ML we learn the programs, rather than write them from scratch.

• It is impossible to deny that statistical modelling/learning and machine learning are intertwined.

• Take linear regression for example, which is a type of ML algorithm that was developed based on the principles of statistics.

• and while terms are not necessarily precise, for the textbook I think the name is mostly rooted in some historical context and reflect different cultures rather than being some well defined notion that you can reason about. (authors are statisticians)

• However, a distinction often made is that statistical and machine learning serve a different purpose.

Two Cultures¹

• this culture is adopted from this paper from 2001 which doesn’t use the term ML, but captures the right idea

• each camp is trying to observe how nature maps x to y and they want to be able to reproduce that x

The paper argues the Statistical approach

• Led to irrelevant theory and questionable scientific conclusions;
• Kept statisticians from using more suitable algorithmic models;
• Prevented statisticians from working on exciting new problems;

Rule of thumb: more complex the date, the more complex the algorithm

• Statistical learning tends to focus on inference

• The statisticians will attempt to model this natural phenomenon through some statistical process or stochastic model.

• they want to model the process that gave rise to the data that you see

• They want are interested in how nature is associating the response variables y to the input variables x., e.g. slope

• use for inference to hypothetically draw more samples and reason about the entire population

• Machine learning tends to focus on prediction

• nature is difficult to model and we don’t really care if we have an accurate model of it.

• Goal: To be able to accurate say what the y responses are going to be to future input variables x;

• how they do this is to measure the performance in predicting the outputs that we’ve seen from given inputs

• So it’s not important that the algorithm actually reflects the real process (we don’t claim that it does),

• the claim is that it does a pretty good job making predictions anyway.

Other thoughts by Pedram Jahangiry YT Machine Learning Concepts

1. Leo Breiman. “Statistical Modeling: The Two Cultures.” Statist. Sci. 16 (3) 199 - 231, August 2001. https://doi.org/10.1214/ss/1009213726

Prediction vs. Inference

• Goal of prediction¹: To be able to predict what the responses are going to be to future input variables

• Goal of inference:

  1. To gain insights about the underlying data generation process, relationships between variables, or the impact of certain factors.
  2. To draw conclusions or make statements about the population or data generating mechanism.

1. Leo Breiman. “Statistical Modeling: The Two Cultures.” Statist. Sci. 16 (3) 199 - 231, August 2001. https://doi.org/10.1214/ss/1009213726

DESCRIBE PICTURE - Not one of the other, often both with varying degree of concentration - This course is going to be geared more towards making predictions.
- Hence more ML than SL - This definition/take is highly variable (some use as synomyms)

Motivating Examples

ILSR2 package

• The following data sets are from the ISLR2 supported files.
• They will serve as motivating examples throughout this course.
• The easiest way to access them in through the ISLR2 R package

# install.packages("ISLR2") 
library(ISLR2)
attach(Wage)

Wage

• The Wage data set comprise the wages from 3000 males from the Atlantic regions of the United States.

• There are 11 variables (type ?Wage for details) :

  • year, age, maritl, race, education, region, jobclass, health, health_ins, logwage, and wage

• When analyzing this data, we might have different types of questions in mind …

The Wage data involves predicting a continuous or quantitative output value.

• We might use this data to understand the relationship between an employee’s age and wage, for example.

• We might also/instead use this data to predict the wage of an employee based on age and eduction, for example.

• We will refer to wage as our quantitative .display[output] value.

Visualization adapted from stackoverflow discussion

In reality there is three goals:

1. Exploratory Data Analysis (EDA): refers to process of investigating, summarizing, and analyzing the main characteristics of data sets.

2. Inference: refers to the process of inferring properties of an underlying population and learning about the mechanism or phenomenon by which the data was generated.

3. Prediction: refers to the process of using historical data to predict the outcomes for new data points.

DESCRIBE PICTURE These goals can be placed within this triangle where each vertex corresponds to a different type of question source

1. Exploratory Questions (summarize main characteristics)
2. Predictive questions (use the data we have to predict about new data)
3. Inferential questions (use this sample data to generalize to a population)

DESCRIBE PICTURE - Exploratory Questions aim to present information in a more digestible and concise way.
- usually a visualization or plot - it doesn’t look to make conclusion or implications about the discrepancies in wage for example. - Exploratory data analysis or EDA is often the first step when looking at a new data set/problem

Exploration

What is the age range in this data set?: range(age) = 18, 80

How does wage differ across education status?

• gives us a birds eye view of what the data looks like,
• Next step: statistical model.
• either used to
  • infer something about the relationships within the data (Inferential question)
  • or predict future values (predictive questions)
  • Often, both.

DESCRIBE PICTURE Answering Inferential questions - attempts to understand the relationships within the data. - often used to inform decisions.
- we want interpretable models that will facilitate making these connections. - A black box type of procedure where we feed in education status and you get a predicted wage would not be ideal because we get no sense of the inner-workings of that model - does more education equate to higher wages–if so by how much?

DESCRIBE PICTURE - we can make these models more and more complex, however, the more complex we make them, the harder they may become at interpreting

Inference

• the lines represent a statistical (regression) model.
• here we might want to infer something about the population from which it was sampled.
• it looks like the more education you have, the higher a wage you can expect
• also it appears that your wage tends to increase with age and experience

Note that these models can be used for prediction and often are, however, based on our Question (i.e. purpose) I would argue this leans towards inference, as opposed to prediction.

DESCRIBE PICTURE Using that same model we might ask a (predictive) question like: what is the expected wage of a 24-year old with an advanced degree?

• next slide for answer

Predictive

what is the expected wage of a 24-year old with an advanced degree? - go to 24 on the x-axis, find the line corresponding to advanced degree (magenta) and predict an average wage of 140ish - I have been making conclusion from these visualization but in practice we have more tangible things to work with like estimates of parameters, confidence intervals, hypothesis tests, etc - Regression is example of “ML” rooting in Statistical modeling

DESCRIBE PICTURE Here the inference and prediction question used the same model, however, when our sole goal is prediction we might not care to interpret these models so a more complicated “ML” algorithm (use NN black box) may be preferred.

DESCRIBE PICTURE statistical learning leans more to the purpley-red region; - methods for inference which seem to be motivated by traditional statistical tools.

Machine Learning more data driven based than hypothesis based - more focused on forecasting (i.e. predicting) future outcomes. - cares less about the interpretibility of the model (black box methods ok)

Supervised vs Unsupervised

Supervised Learning

• Supervised learning is characterized by the presence of “answers” in the data set which are utilized to supervise the algorithm.

• To put another way, the data is comprised of both inputs \(X\) and outputs \(Y\).

  • \(Y\) can be categorical, eg. cat/dog, spam/not spam, positive/negative/neutral
  • \(Y\) can be numeric eg. wage, stock price

Classification vs. Regression

Depending on the format of \(Y\) (i.e. categorical or numeric), supervised learning will perform one of the following tasks:

1. Classification: for predicting a discrete category/label based on the input data, or

• based on a picture (input) is this a cat or a dog
• in the supervised setting the algorithm learns from known labels.
• the algorithm starts off with a buch of labeled images of cats and dogs, learns from that and uses the fitted model to predict on images it has never seen before.
• some of these will be associated with a probability some just with an outcome

2. Regression: for predicting a numerical values based on the input data.

for numeric outcomes. So i give you an age and education level, you predict the wage

Unsupervised Learning

• Unsupervised learning attempts to learn relationships and patterns from data that are not labeled in any way.

• In other words, we have only inputs \(X\) and no \(Y\).

• In cat/dog example our pictures are unlabelled data - i just give you a stack of pictures and ask you to categorize them. I may not even tell you how many groups you can expect to see in this data set
• (clustering rather than classification)

• Unsupervised learning is a more challenging task than supervised.

• Basically learning without a teacher.

• For the meantime, we will focus on the (easier) task of supervised learning before exploring the unsupervised learning.

Source: https://vas3k.com/blog/machine_learning/

DESCRIBE PICTURE

Supervised - pre-catgorized –> Classification - in our cats and dogs example, tell you which pictures are cats and which are dogs rather than just giving you a pile of pictures and asking you to sort them - numerical –> Regression - we might choose Wage as our numerical variable of interest

Unsupervised - not labeled in any way - stacks of pictures of cats and dogs and i ask you to sort them as best you can (no pre-determined number of piles either)

For the meantime, we will focus on the (easier) task of supervised learning before exploring the unsupervised learning.

Notation

Notation is not standard across different disciplines, courses, or textbooks. We adopt the same notation used in ISLR2:

• \(n\): the number of distinct observations in our sample
• \(p\): the number of variables available for making predictions

\(n\)/\(p\) is often BIG in ML (on the order of thousands or even millions)

Example: Wage has \(n\)=3000 observations (i.e. male workers in the Mid-Atlantic region) and \(p\)= 11 variables¹: year, age, maritl, race, education, region, jobclass, health, health_ins, logwage, wage.

• marital status (never, widowed, divorced, separated)
• region of the country
• jobclass (industrial, information)
• health (factor: < good, > very good)
• health_ins (has insurance? logical)
• log of wage
• raw wage

1. some places only use \(p\) to count the non-output variables (in this case 9)

Matrices, Vectors, and scalars

The theory will often involve matrices, vectors and scalars

Matrices

Let \(\textbf{X}\) define an \(n \times p\) matrix whose ( \(i\) , \(j\) )^th element is \(x_{ij}\).

\[\begin{array}{cc} & \begin{array}{cccc} \text{col }1 & \text{col } 2 & & \text{col } p \end{array} \\ \begin{array}{cccc} \text{row }1 \\ \text{row }2 \\ \\ \text{row }n\end{array} & \left( \begin{array}{cccc} x_{11} & x_{12} & \dots & x_{1p} \\ x_{21} & x_{22} & \dots & x_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \dots & x_{np} \end{array} \right)\end{array}\]

• we will use capital bold for matrices
• lower-case normal with two subscripts to denote elements in the matrix
• eg. \(x_{21}\) is the element in the 2nd row, first column

Some may find it helpful to think of \(\textbf{X}\) as a spreadsheet of numbers with \(n\) rows and \(p\) columns. N.B. the first index (\(i\)) of \(x_{ij}\) is the row and second index (\(j\)) is the column)

• in excel we reference rows by number and columns be letter. so \(x_21\) would be cell A2 in excel.
• note the change in order in referencing
  • Excel uses letter then number (column then row)
  • this notation uses row then column (both by number

Vectors

• In reference to matrix \(\mathbf{X}\) we will either be referencing a row vector, \(x_i\), or a column vector \(\textbf{x}_j\).

• Notice the slight change in little x font here:

• rows vector \(x_i\) is curly and not bold

• column vector \(\textbf{x}_j\) is bold and straight

Row Vector

• We refer to the \(i\)^throw of \(\textbf{X}\) using \(x_i\)

• Hence, \(\textbf{X}\) is comprised of the \(n\) row vectors \(x_1, x_2, \dots, x_n\) where \(x_i\) is a vector of length \(p\).

• Typically, \(x_i\) stores all the variable measurements for the \(i\)th observation.

• Vectors are by default represented as columns:

\[\begin{equation} x_i = \left( \begin{array}{c} x_{i1} \\ x_{i2} \\ \vdots \\ x_{i2} \end{array} \right) \end{equation}\]

• lower-case normal x with a single subscript to denote a row vector
• In `Wage \(x_{2\cot}\) references the age, education, etc for a second individual in our data frame and will be of length \(p\)=10
• Even though this is a row vector it will be represented vertically.

Visualization of row vector \(x_i\)

Here $x_i$ is a vector of length $p$, containing the $p$ variable measurements for the $i$th observation.

Rows in R

For example, for the Wage data, \(x_7\) is a vector of length 11, consisting of year, age, race, and other values for the 7th individual.

Wage[7,] # extracts the 7th row

Column Vector

• We refer to the \(j\)^thcolumn of \(\textbf{X}\) using \(\textbf{x}_j\)

• Hence, \(\textbf{X}\) is comprised of the \(p\) column vectors \(\textbf{x}_1, \textbf{x}_2, \dots, \textbf{x}_p\) where \(\textbf{x}_j\) is a vector of length \(n\).

• Typically, \(\textbf{x}_j\) stores the measurements of a variable for all of the \(n\) observations.

\[\begin{equation} \textbf{x}_j = \left( \begin{array}{c} x_{1j} \\ x_{2j} \\ \vdots \\ x_{nj} \end{array} \right) \end{equation}\]

• lower-case bold with a single subscript to denote columns of X
• In the wage data set, this might be the vector of all of the ages for example.
• this vector would be the same length as the number of observations in our data set (hence \(n\))

Visualization of column vector \(= \textbf{x}_j\)

Notice the slight change in little x font here. (rows are more curley) and columns are bold

\(= \textbf{x}_j\)

Columns in R

For example, for the Wage data, \(\textbf{x}_1\) contains the \(n = 3000\) values for year (that year that wage information was recorded for each worker in our data set).

Wage[,1] # extracts the first column (year)

   [1] 2006 2004 2003 2003 2005 2008 2009 2008 2006 2004 2007 2007 2005 2003
  [15] 2009 2009 2003 2006 2007 2003 2003 2005 2009 2007 2006 2005 2006 2004
  [29] 2008 2003 2004 2005 2008 2005 2006 2003 2006 2004 2006 2003 2004 2007
  [43] 2004 2005 2004 2008 2009 2007 2006 2006 2005 2008 2004 2003 2009 2004
  [57] 2006 2003 2004 2003 2005 2006 2005 2004 2003 2003 2009 2003 2004 2008
  [71] 2004 2006 2006 2006 2009 2003 2003 2006 2003 2008 2005 2004 2009 2003
  [85] 2007 2006 2004 2009 2009 2009 2004 2003 2003 2005 2006 2009 2003 2006
  [99] 2009 2005 2009 2008 2004 2005 2008 2006 2007 2003 2007 2006 2009 2008
 [113] 2004 2006 2003 2007 2009 2006 2008 2005 2006 2009 2006 2008 2009 2005
 [127] 2004 2004 2009 2003 2007 2008 2007 2008 2005 2004 2009 2007 2004 2003
 [141] 2004 2006 2003 2007 2005 2004 2005 2004 2007 2004 2007 2008 2003 2003
 [155] 2003 2008 2005 2004 2003 2005 2007 2008 2008 2007 2003 2007 2003 2004
 [169] 2009 2006 2005 2008 2007 2004 2009 2006 2003 2004 2003 2006 2006 2005
 [183] 2006 2009 2003 2006 2004 2008 2006 2005 2004 2005 2004 2003 2006 2009
 [197] 2008 2006 2005 2004 2006 2006 2006 2003 2007 2007 2007 2006 2005 2007
 [211] 2005 2006 2008 2009 2008 2003 2008 2008 2009 2009 2008 2003 2006 2004
 [225] 2005 2003 2009 2006 2007 2003 2003 2006 2006 2007 2004 2009 2004 2006
 [239] 2007 2007 2009 2006 2006 2004 2005 2007 2004 2009 2006 2007 2003 2006
 [253] 2005 2007 2008 2004 2004 2008 2008 2009 2007 2003 2009 2008 2004 2009
 [267] 2007 2006 2004 2003 2007 2007 2003 2005 2007 2008 2004 2003 2006 2005
 [281] 2003 2004 2008 2009 2009 2005 2004 2005 2005 2007 2004 2005 2009 2003
 [295] 2006 2004 2007 2003 2005 2009 2003 2003 2005 2004 2005 2006 2007 2006
 [309] 2004 2006 2007 2007 2009 2004 2007 2007 2007 2005 2006 2008 2004 2009
 [323] 2005 2009 2008 2003 2003 2008 2003 2005 2004 2009 2009 2008 2007 2003
 [337] 2005 2004 2008 2005 2004 2004 2005 2004 2007 2007 2008 2007 2003 2005
 [351] 2004 2009 2005 2009 2005 2003 2004 2006 2009 2007 2007 2003 2004 2005
 [365] 2006 2003 2003 2008 2009 2004 2007 2006 2004 2004 2003 2004 2008 2009
 [379] 2008 2004 2004 2008 2007 2005 2004 2004 2003 2005 2004 2008 2008 2008
 [393] 2005 2008 2003 2007 2009 2009 2003 2004 2006 2005 2004 2009 2004 2009
 [407] 2004 2003 2006 2003 2003 2009 2003 2008 2006 2005 2009 2009 2004 2005
 [421] 2005 2005 2009 2004 2006 2008 2003 2007 2005 2009 2006 2005 2009 2007
 [435] 2003 2003 2007 2007 2005 2005 2009 2004 2004 2006 2005 2009 2007 2003
 [449] 2003 2005 2009 2007 2009 2007 2007 2009 2007 2005 2008 2004 2005 2008
 [463] 2006 2008 2009 2008 2005 2003 2003 2004 2006 2005 2003 2003 2006 2007
 [477] 2005 2006 2009 2003 2007 2006 2005 2009 2004 2009 2009 2003 2007 2007
 [491] 2008 2004 2007 2006 2005 2003 2007 2005 2009 2003 2003 2005 2003 2003
 [505] 2003 2008 2005 2003 2004 2007 2003 2004 2004 2006 2004 2006 2006 2009
 [519] 2004 2004 2003 2003 2007 2005 2005 2009 2003 2007 2003 2009 2003 2004
 [533] 2009 2003 2007 2006 2005 2005 2004 2003 2005 2009 2006 2009 2008 2003
 [547] 2007 2007 2005 2007 2007 2003 2009 2003 2008 2007 2003 2006 2007 2007
 [561] 2003 2009 2007 2006 2008 2004 2004 2005 2004 2003 2004 2004 2007 2008
 [575] 2003 2007 2006 2005 2006 2008 2004 2008 2007 2004 2005 2008 2003 2007
 [589] 2007 2005 2008 2003 2004 2004 2006 2006 2004 2009 2009 2009 2005 2003
 [603] 2008 2005 2006 2004 2006 2009 2006 2006 2005 2008 2008 2009 2006 2006
 [617] 2006 2007 2003 2008 2008 2007 2005 2006 2008 2008 2009 2006 2007 2006
 [631] 2004 2004 2005 2006 2007 2008 2008 2009 2004 2003 2008 2009 2007 2009
 [645] 2008 2003 2005 2007 2005 2003 2009 2004 2007 2006 2008 2006 2003 2006
 [659] 2008 2005 2003 2004 2006 2003 2009 2006 2008 2009 2003 2004 2007 2008
 [673] 2004 2004 2007 2007 2008 2003 2003 2007 2003 2004 2004 2008 2007 2007
 [687] 2009 2003 2009 2006 2007 2003 2003 2005 2003 2008 2005 2009 2005 2005
 [701] 2003 2004 2008 2004 2008 2003 2006 2009 2003 2004 2008 2007 2007 2005
 [715] 2007 2005 2005 2005 2003 2006 2008 2007 2008 2004 2007 2005 2005 2008
 [729] 2004 2008 2009 2006 2004 2003 2007 2009 2004 2009 2005 2006 2008 2008
 [743] 2008 2007 2007 2008 2003 2003 2007 2005 2004 2003 2008 2009 2004 2008
 [757] 2006 2005 2005 2007 2006 2003 2005 2006 2004 2005 2005 2006 2008 2008
 [771] 2004 2003 2008 2004 2006 2008 2003 2008 2003 2007 2004 2008 2008 2003
 [785] 2008 2004 2005 2003 2003 2009 2007 2009 2004 2006 2008 2006 2003 2009
 [799] 2009 2007 2003 2005 2006 2003 2007 2006 2005 2003 2008 2004 2003 2007
 [813] 2006 2003 2007 2007 2007 2007 2009 2006 2004 2008 2003 2004 2009 2005
 [827] 2005 2007 2009 2007 2007 2003 2006 2007 2008 2009 2003 2007 2007 2003
 [841] 2004 2004 2004 2008 2009 2005 2003 2007 2004 2009 2005 2004 2009 2004
 [855] 2005 2008 2007 2003 2004 2006 2009 2009 2005 2007 2004 2005 2005 2003
 [869] 2005 2003 2007 2005 2009 2005 2003 2009 2006 2006 2005 2009 2006 2006
 [883] 2005 2003 2008 2003 2006 2006 2004 2003 2004 2005 2005 2009 2004 2009
 [897] 2004 2003 2004 2003 2004 2009 2009 2008 2006 2007 2007 2004 2006 2008
 [911] 2003 2004 2003 2005 2003 2005 2009 2004 2004 2004 2005 2004 2005 2004
 [925] 2008 2005 2005 2005 2008 2006 2009 2003 2009 2003 2003 2008 2004 2003
 [939] 2003 2005 2008 2007 2007 2009 2005 2009 2003 2003 2004 2005 2006 2009
 [953] 2009 2009 2007 2004 2003 2008 2008 2004 2006 2006 2005 2006 2007 2007
 [967] 2006 2008 2007 2008 2006 2007 2008 2009 2006 2005 2003 2007 2006 2004
 [981] 2003 2004 2009 2008 2009 2006 2007 2003 2003 2009 2005 2005 2008 2006
 [995] 2004 2007 2006 2005 2008 2003 2007 2007 2004 2005 2005 2007 2006 2005
[1009] 2007 2004 2003 2009 2006 2007 2009 2007 2004 2006 2006 2005 2006 2003
[1023] 2007 2007 2003 2009 2007 2006 2007 2008 2004 2003 2006 2006 2004 2008
[1037] 2007 2006 2009 2004 2008 2008 2007 2006 2009 2009 2003 2005 2003 2009
[1051] 2007 2006 2007 2006 2005 2005 2004 2008 2008 2003 2004 2006 2008 2007
[1065] 2005 2007 2004 2008 2005 2007 2005 2006 2006 2006 2003 2004 2007 2004
[1079] 2009 2004 2009 2006 2006 2005 2005 2003 2004 2005 2006 2004 2008 2005
[1093] 2003 2006 2003 2009 2005 2008 2004 2003 2004 2005 2007 2005 2004 2004
[1107] 2004 2006 2003 2005 2008 2004 2006 2007 2007 2004 2007 2006 2003 2009
[1121] 2003 2007 2003 2003 2007 2005 2005 2007 2005 2008 2007 2009 2005 2008
[1135] 2007 2003 2006 2004 2008 2004 2006 2008 2003 2006 2008 2006 2004 2007
[1149] 2008 2004 2009 2003 2007 2009 2004 2004 2005 2004 2006 2006 2004 2008
[1163] 2005 2004 2006 2005 2007 2007 2004 2004 2008 2009 2004 2005 2003 2007
[1177] 2008 2009 2008 2005 2003 2008 2003 2009 2007 2009 2009 2006 2003 2006
[1191] 2003 2006 2006 2009 2007 2007 2009 2006 2004 2007 2005 2004 2005 2006
[1205] 2007 2007 2007 2004 2003 2003 2003 2003 2006 2009 2006 2003 2008 2005
[1219] 2003 2003 2007 2003 2006 2009 2003 2003 2009 2009 2005 2007 2006 2003
[1233] 2005 2006 2009 2004 2005 2007 2004 2008 2005 2003 2006 2008 2004 2003
[1247] 2009 2005 2003 2004 2004 2008 2003 2004 2005 2007 2004 2006 2005 2009
[1261] 2008 2003 2003 2005 2007 2007 2004 2005 2005 2004 2004 2003 2003 2008
[1275] 2009 2004 2003 2008 2006 2005 2009 2004 2003 2007 2008 2005 2003 2004
[1289] 2008 2005 2004 2004 2008 2009 2005 2008 2004 2005 2006 2009 2009 2009
[1303] 2006 2004 2006 2009 2003 2008 2009 2003 2005 2006 2005 2005 2007 2008
[1317] 2003 2004 2009 2003 2007 2005 2006 2005 2006 2004 2003 2004 2003 2004
[1331] 2008 2004 2005 2004 2009 2003 2008 2003 2008 2003 2006 2008 2003 2009
[1345] 2004 2009 2007 2005 2003 2003 2007 2006 2008 2007 2003 2008 2009 2004
[1359] 2005 2003 2006 2005 2007 2004 2004 2005 2006 2009 2009 2004 2003 2004
[1373] 2003 2004 2006 2008 2009 2003 2005 2008 2004 2009 2005 2007 2005 2005
[1387] 2004 2007 2003 2007 2008 2005 2004 2007 2003 2003 2008 2003 2006 2008
[1401] 2003 2008 2006 2009 2004 2008 2006 2003 2007 2004 2009 2008 2004 2005
[1415] 2003 2004 2007 2007 2009 2009 2004 2006 2009 2007 2005 2005 2009 2005
[1429] 2004 2008 2003 2005 2006 2006 2004 2005 2004 2004 2003 2004 2003 2005
[1443] 2003 2004 2003 2003 2006 2004 2007 2008 2005 2008 2005 2003 2006 2006
[1457] 2006 2008 2006 2004 2009 2003 2003 2007 2009 2004 2005 2003 2006 2004
[1471] 2008 2003 2006 2006 2007 2004 2003 2004 2004 2005 2009 2006 2005 2005
[1485] 2008 2007 2008 2003 2003 2007 2004 2008 2007 2008 2008 2003 2004 2008
[1499] 2008 2004 2003 2003 2004 2006 2009 2009 2006 2003 2003 2008 2008 2004
[1513] 2008 2007 2005 2003 2003 2006 2003 2006 2004 2003 2005 2006 2007 2007
[1527] 2009 2006 2006 2004 2004 2003 2004 2007 2003 2007 2005 2004 2007 2006
[1541] 2004 2008 2008 2007 2007 2008 2008 2008 2004 2005 2007 2005 2005 2003
[1555] 2003 2009 2006 2008 2003 2005 2008 2008 2004 2007 2008 2003 2008 2004
[1569] 2009 2004 2007 2004 2008 2004 2009 2005 2006 2003 2003 2006 2006 2009
[1583] 2007 2004 2004 2004 2004 2008 2003 2005 2005 2003 2009 2006 2003 2007
[1597] 2008 2007 2005 2005 2008 2009 2006 2008 2007 2007 2005 2009 2003 2009
[1611] 2009 2008 2006 2003 2008 2003 2006 2006 2004 2006 2008 2004 2004 2007
[1625] 2005 2007 2005 2006 2007 2004 2003 2007 2003 2009 2003 2008 2008 2009
[1639] 2009 2003 2003 2006 2005 2005 2009 2005 2008 2004 2005 2004 2005 2004
[1653] 2009 2005 2004 2007 2004 2004 2009 2005 2003 2005 2003 2009 2003 2003
[1667] 2006 2004 2003 2005 2009 2009 2003 2004 2004 2008 2006 2004 2006 2003
[1681] 2007 2009 2009 2008 2003 2005 2008 2009 2004 2009 2003 2005 2003 2003
[1695] 2008 2005 2009 2006 2007 2003 2005 2005 2004 2007 2007 2009 2009 2006
[1709] 2006 2004 2005 2007 2005 2005 2007 2007 2003 2009 2009 2004 2008 2005
[1723] 2008 2004 2003 2009 2004 2007 2004 2003 2005 2005 2004 2009 2005 2003
[1737] 2004 2009 2003 2005 2004 2007 2004 2006 2008 2007 2008 2004 2005 2007
[1751] 2009 2003 2008 2009 2005 2007 2003 2005 2003 2004 2004 2007 2003 2007
[1765] 2006 2006 2003 2009 2005 2003 2004 2008 2009 2007 2009 2003 2008 2005
[1779] 2006 2006 2004 2006 2004 2009 2005 2009 2003 2004 2006 2006 2005 2004
[1793] 2003 2004 2005 2009 2004 2006 2003 2007 2009 2003 2003 2005 2005 2008
[1807] 2005 2006 2006 2004 2005 2003 2006 2006 2009 2003 2006 2003 2006 2008
[1821] 2006 2006 2004 2006 2008 2008 2005 2003 2008 2003 2006 2007 2009 2008
[1835] 2007 2003 2005 2005 2008 2009 2004 2007 2003 2006 2008 2003 2003 2009
[1849] 2007 2007 2003 2005 2005 2005 2004 2003 2005 2005 2009 2008 2005 2005
[1863] 2006 2006 2003 2004 2008 2003 2005 2004 2007 2007 2009 2007 2006 2003
[1877] 2003 2005 2008 2009 2009 2009 2006 2009 2004 2004 2004 2004 2003 2009
[1891] 2009 2009 2009 2009 2005 2009 2006 2008 2009 2005 2004 2008 2007 2008
[1905] 2007 2009 2007 2004 2004 2004 2007 2008 2005 2005 2008 2004 2005 2009
[1919] 2005 2007 2007 2006 2005 2007 2004 2005 2007 2009 2007 2008 2003 2004
[1933] 2007 2008 2005 2005 2004 2003 2009 2005 2008 2003 2003 2008 2008 2009
[1947] 2004 2006 2005 2008 2006 2009 2004 2009 2009 2004 2004 2003 2004 2005
[1961] 2007 2008 2003 2006 2008 2007 2005 2003 2004 2006 2004 2004 2003 2008
[1975] 2009 2009 2004 2005 2006 2006 2006 2008 2008 2005 2008 2009 2004 2006
[1989] 2004 2007 2008 2005 2007 2003 2009 2004 2007 2005 2007 2009 2003 2005
[2003] 2003 2006 2004 2009 2005 2009 2005 2005 2007 2004 2003 2004 2006 2006
[2017] 2005 2008 2008 2006 2008 2004 2005 2005 2008 2004 2005 2007 2006 2004
[2031] 2006 2009 2004 2003 2004 2009 2008 2006 2008 2003 2005 2009 2008 2008
[2045] 2006 2005 2007 2009 2003 2003 2009 2006 2006 2003 2007 2008 2004 2005
[2059] 2004 2005 2004 2009 2006 2007 2009 2009 2006 2009 2005 2008 2007 2005
[2073] 2005 2006 2008 2005 2003 2003 2004 2008 2004 2004 2004 2004 2007 2006
[2087] 2006 2004 2003 2006 2009 2008 2004 2008 2009 2008 2006 2005 2003 2004
[2101] 2008 2008 2005 2007 2006 2005 2006 2005 2009 2005 2006 2006 2006 2003
[2115] 2004 2005 2004 2004 2008 2009 2003 2005 2004 2006 2009 2006 2003 2007
[2129] 2004 2005 2006 2008 2008 2008 2009 2007 2009 2008 2003 2008 2005 2006
[2143] 2003 2003 2007 2003 2008 2007 2003 2006 2008 2007 2005 2009 2007 2005
[2157] 2008 2007 2007 2005 2003 2005 2008 2006 2004 2004 2009 2003 2006 2007
[2171] 2003 2004 2003 2004 2004 2004 2007 2005 2007 2009 2005 2003 2008 2003
[2185] 2004 2003 2007 2007 2003 2004 2004 2005 2005 2008 2004 2007 2005 2009
[2199] 2007 2004 2004 2004 2006 2006 2003 2008 2009 2005 2008 2003 2005 2006
[2213] 2003 2004 2006 2008 2008 2006 2004 2004 2003 2004 2008 2009 2005 2004
[2227] 2004 2009 2006 2006 2003 2009 2007 2005 2009 2009 2007 2008 2007 2006
[2241] 2004 2005 2005 2005 2004 2009 2003 2008 2008 2009 2004 2008 2005 2008
[2255] 2005 2003 2007 2005 2003 2006 2009 2003 2008 2003 2008 2009 2007 2006
[2269] 2005 2003 2003 2004 2006 2009 2005 2009 2004 2003 2009 2006 2004 2003
[2283] 2007 2004 2003 2003 2007 2004 2007 2009 2007 2008 2004 2005 2009 2008
[2297] 2005 2006 2004 2004 2007 2008 2003 2009 2005 2005 2007 2006 2009 2004
[2311] 2005 2005 2008 2003 2008 2005 2009 2005 2009 2003 2005 2009 2007 2004
[2325] 2003 2003 2004 2004 2005 2008 2004 2009 2008 2003 2006 2006 2004 2003
[2339] 2003 2008 2003 2006 2003 2009 2003 2009 2004 2003 2004 2007 2008 2009
[2353] 2004 2007 2006 2008 2009 2003 2005 2006 2003 2007 2006 2006 2009 2009
[2367] 2008 2003 2008 2003 2004 2003 2004 2004 2007 2006 2009 2006 2006 2005
[2381] 2003 2008 2004 2004 2005 2009 2008 2009 2008 2006 2005 2008 2008 2008
[2395] 2005 2009 2006 2005 2008 2005 2008 2006 2009 2009 2007 2004 2004 2007
[2409] 2004 2003 2006 2003 2005 2004 2003 2008 2009 2006 2007 2004 2008 2004
[2423] 2006 2006 2003 2009 2008 2009 2006 2009 2003 2008 2007 2003 2004 2003
[2437] 2008 2007 2004 2008 2004 2007 2004 2009 2006 2009 2007 2007 2008 2006
[2451] 2004 2008 2003 2007 2006 2004 2008 2009 2009 2008 2007 2005 2005 2004
[2465] 2003 2008 2003 2003 2004 2008 2006 2004 2008 2009 2004 2008 2009 2003
[2479] 2005 2003 2005 2007 2004 2006 2008 2007 2006 2009 2004 2005 2004 2007
[2493] 2009 2008 2009 2005 2005 2009 2005 2007 2006 2008 2004 2008 2006 2004
[2507] 2005 2003 2005 2004 2008 2008 2003 2005 2007 2004 2003 2009 2005 2005
[2521] 2004 2004 2003 2009 2003 2003 2004 2004 2005 2006 2004 2008 2005 2004
[2535] 2004 2009 2003 2004 2003 2008 2007 2007 2005 2008 2003 2008 2006 2005
[2549] 2007 2009 2003 2003 2005 2009 2009 2004 2008 2005 2003 2003 2008 2003
[2563] 2005 2004 2005 2005 2005 2006 2005 2007 2007 2007 2005 2009 2008 2008
[2577] 2003 2003 2003 2008 2006 2003 2008 2004 2005 2007 2009 2007 2005 2003
[2591] 2009 2003 2009 2009 2006 2004 2005 2007 2009 2005 2006 2005 2008 2005
[2605] 2008 2008 2007 2007 2009 2006 2003 2006 2003 2008 2006 2007 2007 2003
[2619] 2008 2005 2007 2008 2003 2008 2005 2003 2003 2008 2003 2004 2005 2007
[2633] 2008 2009 2008 2005 2009 2006 2008 2005 2008 2009 2005 2004 2005 2009
[2647] 2004 2006 2003 2003 2006 2007 2004 2003 2005 2006 2008 2005 2007 2003
[2661] 2008 2006 2003 2006 2003 2005 2004 2006 2004 2009 2003 2007 2003 2007
[2675] 2003 2005 2009 2004 2006 2006 2006 2004 2006 2009 2006 2005 2006 2009
[2689] 2009 2003 2006 2005 2005 2006 2009 2006 2009 2008 2008 2003 2004 2004
[2703] 2006 2004 2005 2003 2005 2004 2004 2006 2008 2007 2006 2008 2003 2007
[2717] 2009 2007 2006 2004 2009 2003 2005 2008 2003 2007 2007 2008 2003 2003
[2731] 2005 2003 2005 2003 2004 2003 2006 2009 2007 2005 2009 2008 2008 2003
[2745] 2009 2005 2007 2006 2004 2004 2008 2006 2009 2009 2006 2007 2003 2004
[2759] 2006 2004 2003 2005 2003 2003 2005 2006 2003 2007 2003 2009 2007 2007
[2773] 2009 2003 2006 2008 2006 2005 2003 2008 2003 2004 2009 2006 2009 2005
[2787] 2008 2004 2008 2008 2006 2004 2004 2004 2005 2009 2003 2008 2006 2003
[2801] 2005 2007 2003 2005 2003 2004 2003 2005 2008 2004 2005 2005 2005 2004
[2815] 2004 2007 2005 2009 2004 2005 2003 2008 2003 2005 2005 2008 2004 2005
[2829] 2004 2008 2007 2005 2004 2003 2009 2006 2004 2009 2009 2004 2008 2009
[2843] 2006 2009 2007 2005 2005 2005 2004 2003 2005 2009 2003 2006 2009 2005
[2857] 2008 2003 2004 2007 2008 2009 2008 2009 2009 2003 2004 2008 2003 2004
[2871] 2004 2003 2009 2003 2008 2003 2004 2004 2006 2008 2007 2004 2007 2005
[2885] 2003 2006 2007 2004 2006 2007 2007 2009 2005 2003 2004 2006 2003 2005
[2899] 2003 2003 2009 2003 2005 2007 2009 2004 2006 2005 2005 2007 2007 2003
[2913] 2007 2003 2006 2004 2008 2003 2003 2009 2004 2006 2007 2007 2005 2005
[2927] 2009 2007 2003 2006 2003 2005 2005 2007 2007 2003 2007 2008 2008 2009
[2941] 2008 2006 2009 2005 2004 2004 2007 2007 2006 2009 2007 2009 2003 2004
[2955] 2003 2003 2003 2007 2007 2006 2004 2007 2009 2006 2004 2007 2003 2003
[2969] 2008 2009 2007 2003 2009 2005 2003 2008 2006 2007 2003 2009 2007 2003
[2983] 2005 2003 2008 2007 2003 2007 2007 2008 2009 2003 2007 2006 2009 2008
[2997] 2007 2005 2005 2009

Matrices revisited

Using the row and column notation just presented, the matrix \(X\) can be written:

\[\begin{equation} \textbf{X} = \left( \begin{array}{c} x_{1}^T \\ x_{2}^T \\ \vdots \\ x_{n}^T \end{array} \right) = (\textbf{x}_1, \textbf{x}_2, \dots, \textbf{x}_p) \end{equation}\]

The \({}^T\) notation denotes the transpose of a matrix or vector, eg \({x}_{i}^T =({x}_{i1}, {x}_{i2}, \dots {x}_{ip})\).

you can think of a matrix as: - \(n\) row vectors or - \(p\) column vectors

remember, since vectors are represented columnwise, we need to transpose the row vectors before stacking them on one another - so \(x_i^T\) denotes the row matrix in its more natural horizontal representation and - \(x_1^T\) stacked on \(x_2^T\), stacked on \(x_3^T\), etc will give us back our matrix capital bold-face X

Output vs. Input Variables

• We use \(x\) to denote our input variable(s) and \(y\) to denote our output variable¹

• For instance, \(y_i\) may refer to the wage of the \(i\)th observation in the Wage data set, whose observed features are stored in \(x_i\).

• The collection of all \(n\) observed outcomes form the vector \(\textbf{y} = (y_1,y_2, \dots, y_n)^T\)

\[\begin{equation} \textbf{y} = \left( \begin{array}{c} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{array} \right) \end{equation}\]

In terms of choice of letters for denoting random varialbes we typically use: - \(x\) for input or predictor variable (features, independent variables) - \(y\) for output or response variable ( dependent variables) - \(y\) is a column vector of length \(n\), we have one “answer” for each observation in our data set

\(N.B.\) \(y\) will only be available in the Supervised setting

1. \(\textbf{y}\) only available in the Supervised setting!

Paired data

Our observed data consists of \(\{(x_1, y_1)\), \((x_2, y_2)\), \(\dots\), \((x_n, y_n)\}\), where each \(x_i\) is a vector of length \(p\).

Visualization of inputs and outputs in tabular data

Assuming we’re in the supervised setting…

• we can think of our observations as a collection of \((x, y)\) pairs

So back to the wage example, we might choose wage to be our output variable \(\textbf{y}\) and use the remaining 10 variables as our input variables \(\textbf{x}\)

Summary

Note there is a potential for ambiguity in the usage of \(a\) (as a vector) and \(a\) as a scalar

Notation	Description	Representation
\(n\)	lower case “n”	number of samples
\(\textbf{x}\), \(\textbf{y}\), \(\textbf{a}\)	lower-case bold	vectors of length \(n\)
\(x\), \(x_i\), \(a\)	lower-case normal	vectors of length \(\neq n\)
\(\textbf{X}\), \(\textbf{A}_{n\times p}\)	capital bold letters	matrix
\(a\)	lower-case normal (beginning of alphabet)	scalars
\(X\)	capital (end of alphabet)	random variables

Why \(a\)?

• vectors that are not of length \(n\) (such as feature vectors of length \(p\)) will be denoted in lower case normal font. - In the rare cases in which these two uses for lower case normal font lead to ambiguity, we will clarify which use is intended.

Ambiguous cases

In the rare cases the use of lower case normal font leads to ambiguity, clarification will be provided using the following notation:

• \(a\) \(\exists \mathbf{R}\) if \(a\) scalar or \(a \in \mathbb{R}\)
• \(a\) \(\exists \mathbf{R}^k\) if \(a\) is a vector of length \(k\) or \(a \in \mathbb{R}^k\)

We will indicate an \(r \times s\) matrix using:

• \(\textbf{A} \exists\) \(\mathbf{R}^{r\times s}\) or \(\textbf{A} \in \mathbb{R}^{r \times s}\) or \(\textbf{A}_{r \times s}\)

• For completion we use A in R^{r times s} to denote the dimension of a matrix.
  
• Alternatively we may specify the dimension of a matrix using subscripts
• I will avoided matrix algebra whenever possible.
• I will brush up on some (basic) matrix algebra when we need it

Statistical Learning

• SL and ML i will use as synomyms

• this goes through some of chapter

General Model

\[Y = f(X) + \epsilon\]

We assume that data arises from the above formula where:

• \(X=(X_1, X_2, \ldots, X_p)\) are inputs (also referred to as predictors, features, independent variables, among others)

• \(Y\) is the output (also referred to as response, dependent variable, among others)

• \(\epsilon\) is the error term (independent of \(X\) and with mean 0)

• our general learning task can be summarized by this formula

• In this formulation, \(f\) represents the systematic information that \(X\) provides about \(Y\) and

• \(epsilon\) captures the inherent randomness that our model will never be able to capture

• the idea is that this process has some generative model \(f\) represents the systematic information that \(X\) provides about \(Y\)
  

• epsilon captures the random behaviour that we’ll never be able to account for

• ML wants to learn about this mechanism \(f\) that mapped our input variables \(X\) to an output variable \(Y\)

Use of Capital Letters

\[Y = f(X) + \epsilon\]

• Note that we are using capital letters¹ to denote random variables in this context.

• For ease of notation in this section, we use \(X\) to denote the input variable(s) which we distinguish using subscripts.

• eg. using the Wage data set we might consider the input variables: \(X = (X_1, X_2)\) where \(X_1\)= year and \(X_2\)= age and response variable \(Y\) = wage.

• The model this model could be regression, KNN, SVM, gaussian mixture models etc.. and will tell the computer how it should find patterns in the data.

1. We denote observed or sampled data by lower-case.

Goal of Statistical Learning

\[Y = f(X) + \epsilon\]

• The function \(f\) that connects the input variable to the output variable is in generally unknown.

If we knew this we wouldn’t need ML

• Statistical learning is concerned with estimating \(f\).

• we don’t know this and what to estimate with some model
• This model \(f\) can be linear regression, KNN, random forests, svm, gaussian mixture model, etc

• Estimates of \(f\) are denoted by \(\hat{f}\), and \(\hat{Y}\) will represent the resulting prediction for \(Y\).

We will use these estimates a couple of different ways throughout the course - predictions on “new” data for which we don’t know the true “answer” - predictions on “new” data for which we do know the true “answer” to be used to see how well we’re doing

Adapted from ISLR

Example: Bacteria Growth

The growth of bacterial in a population occurs in an exponential manner as simulated here.

This is simulated data and the function used to generate these points were indeed based on the exp function in R (discussed more in lab).

This is a Simulation!

Important Note: For real (ie. non-simulated data) data, the function \(f\) is generally unknown and must estimated based on the observed points.

• based on only the points, we want to try and discover that underlying exponential red curve

• Importantly, we won’t generally know the underlying relationship (i.e. exponential)

We don’t know the “shape” of \(f(x)\)

Recall

Reasons for finding \(f\) fall into two primary categories:

• Prediction: with inputs \(X\) available, our concern is predicting the output \(Y\).

• Inference: we want to understand the relationship between \(X\) and \(Y\).

Often, we will be interested in both, perhaps to varying extent. Our goal will dictate what choices for \(f\) will be “better” for us.

Motivated by Prediction

• Motivation for prediction problems often stem from the situation where \(X\) is cheap but \(Y\) is “expensive”.

• “expense” could be monetary
  • running an expensive laser for each data point
• could be time insensitive:
  • collecting data from far-away satellites
• could be labour intensive
  • image categorization needing human sorting

• Using a complicated black box method for \(f\) may be preferable if our many focus is prediction since we are not particularly concerned with the exact form of \(\hat f\).

• We don’t care what are spam detector is looking for or how it works, we just care to sort out our genuine e-mail from spam.

• Generally speaking, the more complex our algorithm, the harder it will be to interpret.

sometimes its better to sacrifice interpretibility to gain an extra 2% accuracy which might equte to an additional 2 billion in revenue.

Motivated by Inference

• Inference aims to answer how \(Y\) affected by \(X\)

• Since \(\hat f\) is used to model this relationship, we don’t want a black box, we want to understand its nuts and bolts.

• Historically, most methods for estimating f have taken a linear form.
• In some situations, such an assumption is reasonable or even desirable.
• But often the true relationship is more complicated, in which case a linear model may not provide an accurate representation of the relationship between the input and output variables.

Some related question may include:

1. Which input variables (predictors) are associated with output variables (response)?

Variable Selection - do i use all my variables or will a subset of them suffice

1. What is the relationship between the response and each predictor? e.g. positive, negative, linear

• Should I expect an increase in wage if i obtain a higher level of education?
• is bacteria increasing linearly or exponentially? at what rate?

Takeaway Message

When choosing the “best” model for a problem, it is important to keep the underlying task in mind.

i.e. prediction or inference - do we care more about getting the right “answer” - or determining the rules to get at that answer

Generally speaking:

• complicated models are often better at prediction but harder to understand;

• simpler models tend to be easier to interpret but will not necessary make accurate predictions.

• If we are mainly interested in inference, then simpler models are much more interpretable.
• If we are only interested in prediction, and the interpretability of the predictive model is simply not of interest.

In many cases a balancing act between the interpretation and accuracy of prediction is needed.

Interpretability vs. accuracy tradeoff

Inspired by Fig 2.7 of ISLR2 (pg 25)

• be aware of the inherent trade-off between accuracy and interpretability in predictive performance
• there will be times when neural networks are needed (eg classification of pictures, videos)

Other pitfalls and considerations:

Why not fit the most complicated model if we are just concerned with prediction? - KISS only use as complicated a model as needed (otherwise you’ll risk “overfitting”) - in other words your findings wont be generalizable to “new” data

General Workflow

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• old data = data we have on hand

Step 1: pick a model

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

pick a model \(f\). throughout the course we will be looking at a variety of ways to do this.

Step 2: Train the model

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• “learn” algorithm/process from data = “train” the model
• trained model = \(\hat{f}\).
• now i want to test that model …

Step 3: Test the model

Get New Data

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• get some new data and feed it into that trained model \(\hat{f}\)

Step 3: Test the model

Make predictions

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• this will produce predictions \(\hat{y}\)

Step 4: Assess

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• we can compare \(\hat{y}\) with the true \(y\) to see how we did
• in future lectures we will discuss different metrics for answering that

Step 5: Rinse Repeat

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• based on the results of that check, we can improve our model
• Iterative process!

Question: Why is it important to have new data?

• don’t want to overfit. E.g. cats/dogs are we learning the pattern or are we just memorizing the answers on the pictures we have on hand

Question: any problems with this workflow

Hint: step 3. Usually we only have one data set on hand.

Training vs. Testing

• If we do not have access to another “new” data set (as we often don’t), we can divide our data (randomly) into two non-overlapping sets:

  1. The training set will be the “old” data used to fit the model
  2. The testing set is strictly used to evaluate performance (i.e. the “new data”)

Comments

• A common split would be 80% of data for fitting and 20% for testing; however, the decision is very dependent on the problem and data available.

A word of warning:

• If training set is small, model fit may be poor
• If testing set is small, performance metrics may be unreliable

Workflow Summary

This is the general workflow we’ll be using throughout this course.

Next class we will talk about how we go about doing step 4.