Data 311: Machine Learning

Lecture 2: Notation and Terminology

Dr. Irene Vrbik

University of British Columbia Okanagan

Outcomes

• Define Statistical vs. Machine learning
• Differentiate between Inference vs. Prediction
• Understand the difference between Supervised vs Unsupervised Learning
• Distinguish between Classification vs. Regression problems
• Recognize and correctly interpret the common Notation¹
• Training vs. Testing

Today we will be looking at notation that may or may not differ from those used in courses you have previously taken in Statistics.

1. as used in ISLR2 Notation and Simple Matrix Algebra (pg 9–12).

Motivating Example

• The following data sets are from the ISLR2 supported files.
• They will serve as motivating examples throughout this course.
• The easiest way to access them in through the ISLR2 R package

# install.packages("ISLR2") 
library(ISLR2)
attach(Wage)

Wage

• The Wage data set comprise the wages from 3000 males from the Atlantic regions of the United States.

• There are 11 variables (type ?Wage for details) :

  • year, age, maritl, race, education, region, jobclass, health, health_ins, logwage, and wage

• When analyzing this data, we might have different types of questions in mind …

The Wage data involves predicting a continuous or quantitative output value.

• We might use this data to understand the relationship between an employee’s age and wage, for example.

• We might also/instead use this data to predict the wage of an employee based on age and eduction, for example.

• We will refer to wage as our quantitative .display[output] value.

Visualization adapted from stackoverflow discussion

In reality there is three goals:

1. Exploratory Data Analysis (EDA): refers to process of investigating, summarizing, and analyzing the main characteristics of data sets.

2. Inference: refers to the process of inferring properties of an underlying population and learning about the mechanism or phenomenon by which the data was generated.

3. Prediction: refers to the process of using historical data to predict the outcomes for new data points.

DESCRIBE PICTURE These goals can be placed within this triangle where each vertex corresponds to a different type of question source

1. Exploratory Questions (summarize main characteristics)
2. Predictive questions (use the data we have to predict about new data)
3. Inferential questions (use this sample data to generalize to a population)

DESCRIBE PICTURE - Exploratory Questions aim to present information in a more digestible and concise way.
- usually a visualization or plot - it doesn’t look to make conclusion or implications about the discrepancies in wage for example. - Exploratory data analysis or EDA is often the first step when looking at a new data set/problem

Exploration

What is the age range in this data set?: range(age) = 18, 80

How does wage differ across education status?

• gives us a birds eye view of what the data looks like,
• Next step: statistical model.
• either used to
  • infer something about the relationships within the data (Inferential question)
  • or predict future values (predictive questions)
  • Often, both.

DESCRIBE PICTURE Answering Inferential questions - attempts to understand the relationships within the data. - often used to inform decisions.
- we want interpretable models that will facilitate making these connections. - A black box type of procedure where we feed in education status and you get a predicted wage would not be ideal because we get no sense of the inner-workings of that model - does more education equate to higher wages–if so by how much?

DESCRIBE PICTURE - we can make these models more and more complex, however, the more complex we make them, the harder they may become at interpreting

DESCRIBE PICTURE Using that same model we might ask a (predictive) question like: what is the expected wage of a 24-year old with an advanced degree?

• I want you to keep in mind that the “right” model for a given data set has to do both with what kind of data we have AND what type of question we are asking…

Introduction

Data:

• input \(x\) (features, independent variables)
• output \(y\) (response/dependent variables)

Input Variables (x): Features like size of the house, number of bedrooms, location, age of the house, and recent sales in the area.

Output Variable (y): The selling price of the house.

Inside the box we observe some natural pheonomenon that determines the selling price of a house

• market dynamics, location, buyer behaviour, etc

Source: Breiman, L. (2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 16(3), 199-231. 🔗Link

Approaches

There are two goals in analyzing this data…

Data Modeling Culture

Algorithmic Modeling Culture

Each camp is trying to observe how nature maps x to y and they want to be able to reproduce that y.

Data Modeling Culture:

• the black box is simplified by assuming a specific, often linear, mathematical model for the relationship between x and y

• This approach relies on predefined assumptions about how inputs are related to outputs.

Algorithmic Modeling Culture:

• treats the inside of the black box as complex and unknown (market dynamics, location, buyer behaviour, etc), without making strong assumptions about its structure

• It uses flexible algorithms to learn the relationship between x and y directly from the data.

• nature is difficult to model we don’t really care about the nuts and bolts if we obtain a good estimat of y

Two Cultures

Data Modeling Culture:

Assumes data is generated by a specific stochastic process that they wish to model.

may be preferred for inference

Algorithmic Modeling:

Often treats the data generation process as unknown or complex.

may be preferred when prediction is your goal

Each camp is trying to observe how nature maps x to y and they want to be able to reproduce that y but theyre going to do this is slightly different ways

Data Modeling (Statistical Learning) Culture

• The statisticians will attempt to model this natural phenomenon through some statistical process or stochastic model.

• they want to model the process that gave rise to the data and understand how nature is associating the input(s) x with the output y

• Favors traditional statistical models (e.g., linear regression, generalized linear models) chosen based on assumptions about the data.

• e.g. if your house has an extra bedroom, you can expect your selling price to go up by $X. (theoretical mode could be a linear regression model)

• Critique: often leads to rigid assumptions and irrelevant theories.

Algorithmic Modeling (ML) Culture

• Treats the data mechanism as complex and unknown. nature is difficult to model we don’t really care if we have an accurate model of it.

• Goal: To be able to accurate say what the y responses are going to be to future input variables x;

• Leverages powerful machine learning algorithms (e.g., random forests, neural networks) with greater flexibility to model complex relationships.

• Benefit: This often achieves better predictive accuracy; however those models may be complex for humans to interpret

I’m not trying to be pedantic about the difference between machine learning and statistical learning. Instead, I want to emphasize the importance of understanding the concepts of prediction and inference, as these goals will dictate which model you should choose. Depending on your goals, either culture might be more suitable.

Algorithmic Modeling (ML) Culture: Focuses on predictive algorithms, often treating the data generation process as unknown or complex.

Other thoughts by Pedram Jahangiry YT Machine Learning Concepts

Source: Leo Breiman. “Statistical Modeling: The Two Cultures.” Statist. Sci. 16 (3) 199 - 231, August 2001. https://doi.org/10.1214/ss/1009213726

Inference vs. Prediction

Inference goal: we might fit a linear model to understand how specific features (eg. square footage, number of bedrooms, or proximity to amenities) influence house prices.

Prediction goal: we might use random forests with the same features to predict housing prices, with less emphasis on interpreting each feature’s impact.

x = … pool

Inferential questions might include:

• How much does each additional square foot add to the value of a house, on average?

• Are houses in urban areas significantly more expensive than those in suburban areas, controlling for other factors?

• What is the impact of a newly renovated kitchen on house prices, after accounting for the size and location of the house?

Prediction question

Instead of seeking to understand or quantify this impact precisely, you would reframe the goal towards predicting house prices using the number of bedrooms and other features, with less emphasis on the interpretability of each feature’s impact.

• These models may capture complex interactions between variables without explicitly estimating or interpreting individual effects.

Inference

Objective: To understand the relationship between \(x\) and \(y\) in order to draw conclusions about the population or data generating mechanism.

Focus: Emphasizes the interpretability and simplicity of the model

Prediction

Objective: To accurately forecast the output (response \(y\)) for new, unseen data points based on existing patterns in the data.

Focus: Prioritizes accuracy of predictions, often using “black box” algorithms

Examples of Inference: Determining how factors like education and experience impact salary, or understanding the effect of a medication on patient recovery rates.

Examples of Prediction models: (e.g., random forests, neural networks)

Goal of inference:

1. To gain insights about the underlying data generation process, relationships between variables, or the impact of certain factors.
2. To draw conclusions or make statements about the population or data generating mechanism.

Goal of prediction: To be able to predict what the responses are going to be to future input variables.

DESCRIBE PICTURE - Not one of the other, often both with varying degree of concentration - This course is going to be geared more towards making predictions.
- Hence more ML than SL - This definition/take is highly variable (some use as synomyms)

Examples

Inference Examples:

• Estimating the effect of a drug on blood pressure

• testing if there is a significant difference between groups

• determining the association between smoking and lung cancer

Prediction Examples:

• Predicting house prices based on features like size and location

• forecasting stock prices

• classifying whether an email is spam or not

• weather forecast

Inference examples:

• difference between groups (e.g. rural vs. urban)

• eg. notoriously tricky stock market prediction

  • algorithmic modeling (like random forests or support vector machines) can analyze vast historical data (stock prices, economic indicators, and news sentiment) without assuming a specific model.
  • uncover complex, non-linear patterns that are not readily apparent with traditional methods

My take on the two cultures

Statistical Learning

Like the Data Modeling Culture “Statistical Learning” assumes a specific stochastic model (e.g., linear regression, logistic regression). They prioritize interpretability and understanding the relationship between variables.

• Primarily rooted in Statistics

Machine Learning

Like the Algorithmic Modeling Culture, “Machine Learning” emphasizes predictive accuracy and the ability to handle complex, high-dimensional data without predefined assumptions about the data structure.

• Primarily rooted in Computer Science

• adopted from this paper from 2001 which doesn’t use the term ML, but captures the right idea:
• we will be touching on both throughout the course
• while i don’t care too much about distinguishing between the two terms, i will try to focus on the different goals they usually tend to focus on

Statistical vs. Machine learning

• The authors of ISLR chose “Statistical Learning” to emphasize their statistics-focused approach to machine learning, as opposed to a computer science emphasis.¹

the authors are established statistians from standford

• A distinction often made is that statistical and machine learning serve a different purpose.

• Statistical learning tends to focus on inference, while Machine learning tends to focus on prediction.

in the youtube video two of the authors (experienced statisticians and professors at Stanford) say that statistical learning evolved in the 1980s alongside the rise of machine learning and neural networks, driven by developments in computer science.

is there take on Machine Learning which is more heavily rooted in computer science

1. Source: Statistical Learning: 1.1 Opening Remarks: https://youtu.be/LvySJGj-88U?si=q37Css-rgxkhpjUP

DESCRIBE PICTURE statistical learning leans more to the purpley-red region; - methods for inference which seem to be motivated by traditional statistical tools.

Machine Learning more data driven based than hypothesis based - more focused on forecasting (i.e. predicting) future outcomes. - cares less about the interpretibility of the model (black box methods ok)

Takeaway Message

• No single approach is universally superior
• Both “camps” have their strengths and weaknesses
• Our goal is to adopt a broad toolkit to handle a wide range of problems

Different Philosophies and Goals:

• Inference which favours statistical learning rooted in statistics

• Prediction which tends to favour Machine Learning models originating from computer science

SL:

• focuses on understanding the relationship between variables through interpretable models.
  
• It often assumes specific structures (e.g., linear models) and prioritizes inference and interpretability.

ML:

• it emphasizes predictive accuracy and the ability to handle complex, high-dimensional data.
• It uses algorithmic models (e.g., decision trees, neural networks) that don’t require explicit assumptions about data structure.

iClicker Question

Which of the following is a strength of the algorithmic modeling culture compared to the data modeling culture?

1. Excels in predictive accuracy but lacks interpretability

2. Emphasizes interpretability and simplicity

3. Relies on strong assumptions about data structure.

4. Focuses mainly on inference

Correct answer: A

Supervised vs Unsupervised

Supervised Learning

• Supervised learning is characterized by the presence of “answers” in the data set which are utilized to supervise the algorithm.

• To put another way, the data is comprised of both inputs (AKA predictor variables or features) \(X\) and outputs (AKA response variable) \(Y\) (our “answers”).

  • \(Y\) can be categorical, eg. cat/dog, spam/not spam, positive/negative/neutral
  • \(Y\) can be numeric eg. wage, stock price

these “answers” are often refereed to as “labels” Write that down!

Classification vs. Regression

Depending on the format of \(Y\) (i.e. categorical or numeric), supervised learning will perform one of the following tasks:

1. Classification: for predicting a discrete category/label based on the input data, or

• based on a picture (input) is this a cat or a dog
• in the supervised setting the algorithm learns from known labels.
• the algorithm starts off with a buch of labeled images of cats and dogs, learns from that and uses the fitted model to predict on images it has never seen before.
• some of these will be associated with a probability some just with an outcome

2. Regression: for predicting a numerical values based on the input data.

for numeric outcomes. So i give you an age and education level, you predict the wage

Unsupervised Learning

• Unsupervised learning attempts to learn relationships and patterns from data that are not labeled in any way.

• In other words, we have only inputs \(X\) and no \(Y\).

• In cat/dog example our pictures are unlabelled data - i just give you a stack of pictures and ask you to categorize them. I may not even tell you how many groups you can expect to see in this data set
• (clustering rather than classification)

• Unsupervised learning is a more challenging task than supervised.

• Basically learning without a teacher.

• For the meantime, we will focus on the (easier) task of supervised learning before exploring the unsupervised learning.

Source: https://vas3k.com/blog/machine_learning/

DESCRIBE PICTURE

Supervised - pre-catgorized –> Classification - in our cats and dogs example, tell you which pictures are cats and which are dogs rather than just giving you a pile of pictures and asking you to sort them - numerical –> Regression - we might choose Wage as our numerical variable of interest

Unsupervised - not labeled in any way - stacks of pictures of cats and dogs and i ask you to sort them as best you can (no pre-determined number of piles either)

For the meantime, we will focus on the (easier) task of supervised learning before exploring the unsupervised learning.

iClicker Question

What is the main difference between supervised and unsupervised learning?

1. Supervised learning only works with numbers, while unsupervised learning only works with text.

2. Supervised learning predicts categories, while unsupervised learning predicts continuous values.

3. Supervised learning and unsupervised learning are the same; they both use labeled data to make predictions.

4. Supervised learning uses labeled examples, while unsupervised learning tries to find patterns within unlabelled data.

Correct answer: D

iClicker Question

What is the main difference between classification and regression?

1. Classification only works with numbers, while regression only works with text.

2. Classification predicts categories, while regression predicts continuous values.

3. Classification is used for organizing data, while regression is used for finding patterns.

4. Classification and regression are the same; they both predict categories.

Correct answer: B

Notation

This section will cover some basic notation and refresh our memory on Matrices, Vectors, and scalars

The theory will often involve matrices, vectors and scalars

Wage dataset

Let’s consider the Wage dataset from the ISLR2 package.

# install.packages("ISLR2") 
library(ISLR2)
attach(Wage)

• The Wage data set comprise the wages from 3000 males from the Atlantic regions of the United States.

• There are 11 variables: year, age, maritl, race, education, region, jobclass, health, health_ins, logwage, and wage

The Wage data involves predicting a continuous or quantitative output value.

• We might use this data to understand the relationship between an employee’s age and wage, for example.

• We might also/instead use this data to predict the wage of an employee based on age and eduction, for example.

• We will refer to wage as our quantitative .display[output] value.

To access the details for this dataset, type ?Wage into R.

Notation

Notation is not standard across different disciplines, courses, or textbooks. We adopt the same notation used in ISLR2:

• \(n\): the number of distinct observations in our sample
• \(p\): the number of features available for making predictions.

Number of features

Most places will use \(p\) to denote the total number of columns excluding the response variable while others will use it to count all variables. We we use the former.

\(n\)/\(p\) is often BIG in ML (on the order of thousands or even millions)

Viewing your data

• Structure
• View spreadsheet
• Head

str(Wage)

'data.frame':   3000 obs. of  11 variables:
 $ year      : int  2006 2004 2003 2003 2005 2008 2009 2008 2006 2004 ...
 $ age       : int  18 24 45 43 50 54 44 30 41 52 ...
 $ maritl    : Factor w/ 5 levels "1. Never Married",..: 1 1 2 2 4 2 2 1 1 2 ...
 $ race      : Factor w/ 4 levels "1. White","2. Black",..: 1 1 1 3 1 1 4 3 2 1 ...
 $ education : Factor w/ 5 levels "1. < HS Grad",..: 1 4 3 4 2 4 3 3 3 2 ...
 $ region    : Factor w/ 9 levels "1. New England",..: 2 2 2 2 2 2 2 2 2 2 ...
 $ jobclass  : Factor w/ 2 levels "1. Industrial",..: 1 2 1 2 2 2 1 2 2 2 ...
 $ health    : Factor w/ 2 levels "1. <=Good","2. >=Very Good": 1 2 1 2 1 2 2 1 2 2 ...
 $ health_ins: Factor w/ 2 levels "1. Yes","2. No": 2 2 1 1 1 1 1 1 1 1 ...
 $ logwage   : num  4.32 4.26 4.88 5.04 4.32 ...
 $ wage      : num  75 70.5 131 154.7 75 ...

The following will open a window similar to an Excel spreadsheet:

View(Wage)

The following shows the first 6 rows, but if your data is very wide (i.e. there are a lot of columns) it can be difficult to view

head(Wage)

iClicker

What is \(n\) the Wage data.

1. 10

2. 11

3. 2999

4. 3000

5. None of the above

Correct Answer: D; Wage has \(n\)=3000 observations (i.e. male workers in the Mid-Atlantic region)

iClicker

What is \(p\) for the Wage data?

1. 10

2. 11

3. 2999

4. 3000

5. None of the above

Correct Answer: E; Wage has \(p\)= 9 variables: year, age, maritl, race, education, region, jobclass, health, health_ins. N.B. logwage and wage are just two versions of the output variable.

• marital status (never, widowed, divorced, separated)
• region of the country
• jobclass (industrial, information)
• health (factor: < good, > very good)
• health_ins (has insurance? logical)
• log of wage
• raw wage

Example: Wage has \(n\)=3000 observations (i.e. male workers in the Mid-Atlantic region) and \(p\)= 10 variables: year, age, maritl, race, education, region, jobclass, health, health_ins, logwage, wage.

Matrices

Let \(\textbf{X}\) define an \(n \times p\) matrix whose ( \(i\) , \(j\) )^th element is \(x_{ij}\).

\[\begin{array}{cc} & \begin{array}{cccc} \text{col }1 & \text{col } 2 & & \text{col } p \end{array} \\ \begin{array}{cccc} \text{row }1 \\ \text{row }2 \\ \\ \text{row }n\end{array} & \left( \begin{array}{cccc} x_{11} & x_{12} & \dots & x_{1p} \\ x_{21} & x_{22} & \dots & x_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \dots & x_{np} \end{array} \right)\end{array}\]

• we will use capital bold for matrices
• lower-case normal with two subscripts to denote elements in the matrix
• eg. \(x_{21}\) is the element in the 2nd row, first column
• first index is the row, second index is the column

Some may find it helpful to think of \(\textbf{X}\) as a spreadsheet of numbers with \(n\) rows and \(p\) columns. N.B. the first index (\(i\)) of \(x_{ij}\) is the row and second index (\(j\)) is the column)

• in excel we reference rows by number and columns be letter. so \(x_21\) would be cell A2 in excel.
• note the change in order in referencing
  • Excel uses letter then number (column then row)
  • this notation uses row then column (both by number

Vectors

• In reference to matrix \(\mathbf{X}\) we will either be referencing a row vector, \(x_i\), or a column vector \(\textbf{x}_j\).

• Notice the slight change in little x font here:

• rows vector \(x_i\) is curly and not bold (typically an example)

• column vector \(\textbf{x}_j\) is bold and straight (typically a feature)

Row Vector

• We refer to the \(i\)^throw of \(\textbf{X}\) using \(x_i\)

• Hence, \(\textbf{X}\) is comprised of the \(n\) row vectors \(x_1, x_2, \dots, x_n\) where \(x_i\) is a vector of length \(p\).

• Typically, \(x_i\) stores all the variable measurements for the \(i\)th observation.

• Vectors are by default represented as columns:

\[\begin{equation} x_i = \left( \begin{array}{c} x_{i1} \\ x_{i2} \\ \vdots \\ x_{i2} \end{array} \right) \end{equation}\]

• lower-case normal x with a single subscript to denote a row vector
• In `Wage \(x_{2\cot}\) references the age, education, etc for a second individual in our data frame and will be of length \(p\)=10
• Even though this is a row vector it will be represented vertically.

Visualization of row vector \(x_i\)

Here $x_i$ is a vector of length $p$, containing the $p$ variable measurements for the $i$th observation.

Rows in R

For example, for the Wage data, \(x_7\) is a vector of length 11, consisting of year, age, race, and other values for the 7th individual.

Wage[7,] # extracts the 7th row

Column Vector

• We refer to the \(j\)^thcolumn of \(\textbf{X}\) using \(\textbf{x}_j\)

• Hence, \(\textbf{X}\) is comprised of the \(p\) column vectors \(\textbf{x}_1, \textbf{x}_2, \dots, \textbf{x}_p\) where \(\textbf{x}_j\) is a vector of length \(n\).

• Typically, \(\textbf{x}_j\) stores the measurements of a variable for all of the \(n\) observations.

\[\begin{equation} \textbf{x}_j = \left( \begin{array}{c} x_{1j} \\ x_{2j} \\ \vdots \\ x_{nj} \end{array} \right) \end{equation}\]

• lower-case bold with a single subscript to denote columns of X
• In the wage data set, this might be the vector of all of the ages for example.
• this vector would be the same length as the number of observations in our data set (hence \(n\))

Visualization of column vector \(= \textbf{x}_j\)

Notice the slight change in little x font here. (rows are more curley) and columns are bold

\(= \textbf{x}_j\)

Columns in R

For example, for the Wage data, \(\textbf{x}_1\) contains the \(n = 3000\) values for year (that year that wage information was recorded for each worker in our data set).

Wage[,1] # extracts the first column (year)

   [1] 2006 2004 2003 2003 2005 2008 2009 2008 2006 2004 2007 2007 2005 2003
  [15] 2009 2009 2003 2006 2007 2003 2003 2005 2009 2007 2006 2005 2006 2004
  [29] 2008 2003 2004 2005 2008 2005 2006 2003 2006 2004 2006 2003 2004 2007
  [43] 2004 2005 2004 2008 2009 2007 2006 2006 2005 2008 2004 2003 2009 2004
  [57] 2006 2003 2004 2003 2005 2006 2005 2004 2003 2003 2009 2003 2004 2008
  [71] 2004 2006 2006 2006 2009 2003 2003 2006 2003 2008 2005 2004 2009 2003
  [85] 2007 2006 2004 2009 2009 2009 2004 2003 2003 2005 2006 2009 2003 2006
  [99] 2009 2005 2009 2008 2004 2005 2008 2006 2007 2003 2007 2006 2009 2008
 [113] 2004 2006 2003 2007 2009 2006 2008 2005 2006 2009 2006 2008 2009 2005
 [127] 2004 2004 2009 2003 2007 2008 2007 2008 2005 2004 2009 2007 2004 2003
 [141] 2004 2006 2003 2007 2005 2004 2005 2004 2007 2004 2007 2008 2003 2003
 [155] 2003 2008 2005 2004 2003 2005 2007 2008 2008 2007 2003 2007 2003 2004
 [169] 2009 2006 2005 2008 2007 2004 2009 2006 2003 2004 2003 2006 2006 2005
 [183] 2006 2009 2003 2006 2004 2008 2006 2005 2004 2005 2004 2003 2006 2009
 [197] 2008 2006 2005 2004 2006 2006 2006 2003 2007 2007 2007 2006 2005 2007
 [211] 2005 2006 2008 2009 2008 2003 2008 2008 2009 2009 2008 2003 2006 2004
 [225] 2005 2003 2009 2006 2007 2003 2003 2006 2006 2007 2004 2009 2004 2006
 [239] 2007 2007 2009 2006 2006 2004 2005 2007 2004 2009 2006 2007 2003 2006
 [253] 2005 2007 2008 2004 2004 2008 2008 2009 2007 2003 2009 2008 2004 2009
 [267] 2007 2006 2004 2003 2007 2007 2003 2005 2007 2008 2004 2003 2006 2005
 [281] 2003 2004 2008 2009 2009 2005 2004 2005 2005 2007 2004 2005 2009 2003
 [295] 2006 2004 2007 2003 2005 2009 2003 2003 2005 2004 2005 2006 2007 2006
 [309] 2004 2006 2007 2007 2009 2004 2007 2007 2007 2005 2006 2008 2004 2009
 [323] 2005 2009 2008 2003 2003 2008 2003 2005 2004 2009 2009 2008 2007 2003
 [337] 2005 2004 2008 2005 2004 2004 2005 2004 2007 2007 2008 2007 2003 2005
 [351] 2004 2009 2005 2009 2005 2003 2004 2006 2009 2007 2007 2003 2004 2005
 [365] 2006 2003 2003 2008 2009 2004 2007 2006 2004 2004 2003 2004 2008 2009
 [379] 2008 2004 2004 2008 2007 2005 2004 2004 2003 2005 2004 2008 2008 2008
 [393] 2005 2008 2003 2007 2009 2009 2003 2004 2006 2005 2004 2009 2004 2009
 [407] 2004 2003 2006 2003 2003 2009 2003 2008 2006 2005 2009 2009 2004 2005
 [421] 2005 2005 2009 2004 2006 2008 2003 2007 2005 2009 2006 2005 2009 2007
 [435] 2003 2003 2007 2007 2005 2005 2009 2004 2004 2006 2005 2009 2007 2003
 [449] 2003 2005 2009 2007 2009 2007 2007 2009 2007 2005 2008 2004 2005 2008
 [463] 2006 2008 2009 2008 2005 2003 2003 2004 2006 2005 2003 2003 2006 2007
 [477] 2005 2006 2009 2003 2007 2006 2005 2009 2004 2009 2009 2003 2007 2007
 [491] 2008 2004 2007 2006 2005 2003 2007 2005 2009 2003 2003 2005 2003 2003
 [505] 2003 2008 2005 2003 2004 2007 2003 2004 2004 2006 2004 2006 2006 2009
 [519] 2004 2004 2003 2003 2007 2005 2005 2009 2003 2007 2003 2009 2003 2004
 [533] 2009 2003 2007 2006 2005 2005 2004 2003 2005 2009 2006 2009 2008 2003
 [547] 2007 2007 2005 2007 2007 2003 2009 2003 2008 2007 2003 2006 2007 2007
 [561] 2003 2009 2007 2006 2008 2004 2004 2005 2004 2003 2004 2004 2007 2008
 [575] 2003 2007 2006 2005 2006 2008 2004 2008 2007 2004 2005 2008 2003 2007
 [589] 2007 2005 2008 2003 2004 2004 2006 2006 2004 2009 2009 2009 2005 2003
 [603] 2008 2005 2006 2004 2006 2009 2006 2006 2005 2008 2008 2009 2006 2006
 [617] 2006 2007 2003 2008 2008 2007 2005 2006 2008 2008 2009 2006 2007 2006
 [631] 2004 2004 2005 2006 2007 2008 2008 2009 2004 2003 2008 2009 2007 2009
 [645] 2008 2003 2005 2007 2005 2003 2009 2004 2007 2006 2008 2006 2003 2006
 [659] 2008 2005 2003 2004 2006 2003 2009 2006 2008 2009 2003 2004 2007 2008
 [673] 2004 2004 2007 2007 2008 2003 2003 2007 2003 2004 2004 2008 2007 2007
 [687] 2009 2003 2009 2006 2007 2003 2003 2005 2003 2008 2005 2009 2005 2005
 [701] 2003 2004 2008 2004 2008 2003 2006 2009 2003 2004 2008 2007 2007 2005
 [715] 2007 2005 2005 2005 2003 2006 2008 2007 2008 2004 2007 2005 2005 2008
 [729] 2004 2008 2009 2006 2004 2003 2007 2009 2004 2009 2005 2006 2008 2008
 [743] 2008 2007 2007 2008 2003 2003 2007 2005 2004 2003 2008 2009 2004 2008
 [757] 2006 2005 2005 2007 2006 2003 2005 2006 2004 2005 2005 2006 2008 2008
 [771] 2004 2003 2008 2004 2006 2008 2003 2008 2003 2007 2004 2008 2008 2003
 [785] 2008 2004 2005 2003 2003 2009 2007 2009 2004 2006 2008 2006 2003 2009
 [799] 2009 2007 2003 2005 2006 2003 2007 2006 2005 2003 2008 2004 2003 2007
 [813] 2006 2003 2007 2007 2007 2007 2009 2006 2004 2008 2003 2004 2009 2005
 [827] 2005 2007 2009 2007 2007 2003 2006 2007 2008 2009 2003 2007 2007 2003
 [841] 2004 2004 2004 2008 2009 2005 2003 2007 2004 2009 2005 2004 2009 2004
 [855] 2005 2008 2007 2003 2004 2006 2009 2009 2005 2007 2004 2005 2005 2003
 [869] 2005 2003 2007 2005 2009 2005 2003 2009 2006 2006 2005 2009 2006 2006
 [883] 2005 2003 2008 2003 2006 2006 2004 2003 2004 2005 2005 2009 2004 2009
 [897] 2004 2003 2004 2003 2004 2009 2009 2008 2006 2007 2007 2004 2006 2008
 [911] 2003 2004 2003 2005 2003 2005 2009 2004 2004 2004 2005 2004 2005 2004
 [925] 2008 2005 2005 2005 2008 2006 2009 2003 2009 2003 2003 2008 2004 2003
 [939] 2003 2005 2008 2007 2007 2009 2005 2009 2003 2003 2004 2005 2006 2009
 [953] 2009 2009 2007 2004 2003 2008 2008 2004 2006 2006 2005 2006 2007 2007
 [967] 2006 2008 2007 2008 2006 2007 2008 2009 2006 2005 2003 2007 2006 2004
 [981] 2003 2004 2009 2008 2009 2006 2007 2003 2003 2009 2005 2005 2008 2006
 [995] 2004 2007 2006 2005 2008 2003 2007 2007 2004 2005 2005 2007 2006 2005
[1009] 2007 2004 2003 2009 2006 2007 2009 2007 2004 2006 2006 2005 2006 2003
[1023] 2007 2007 2003 2009 2007 2006 2007 2008 2004 2003 2006 2006 2004 2008
[1037] 2007 2006 2009 2004 2008 2008 2007 2006 2009 2009 2003 2005 2003 2009
[1051] 2007 2006 2007 2006 2005 2005 2004 2008 2008 2003 2004 2006 2008 2007
[1065] 2005 2007 2004 2008 2005 2007 2005 2006 2006 2006 2003 2004 2007 2004
[1079] 2009 2004 2009 2006 2006 2005 2005 2003 2004 2005 2006 2004 2008 2005
[1093] 2003 2006 2003 2009 2005 2008 2004 2003 2004 2005 2007 2005 2004 2004
[1107] 2004 2006 2003 2005 2008 2004 2006 2007 2007 2004 2007 2006 2003 2009
[1121] 2003 2007 2003 2003 2007 2005 2005 2007 2005 2008 2007 2009 2005 2008
[1135] 2007 2003 2006 2004 2008 2004 2006 2008 2003 2006 2008 2006 2004 2007
[1149] 2008 2004 2009 2003 2007 2009 2004 2004 2005 2004 2006 2006 2004 2008
[1163] 2005 2004 2006 2005 2007 2007 2004 2004 2008 2009 2004 2005 2003 2007
[1177] 2008 2009 2008 2005 2003 2008 2003 2009 2007 2009 2009 2006 2003 2006
[1191] 2003 2006 2006 2009 2007 2007 2009 2006 2004 2007 2005 2004 2005 2006
[1205] 2007 2007 2007 2004 2003 2003 2003 2003 2006 2009 2006 2003 2008 2005
[1219] 2003 2003 2007 2003 2006 2009 2003 2003 2009 2009 2005 2007 2006 2003
[1233] 2005 2006 2009 2004 2005 2007 2004 2008 2005 2003 2006 2008 2004 2003
[1247] 2009 2005 2003 2004 2004 2008 2003 2004 2005 2007 2004 2006 2005 2009
[1261] 2008 2003 2003 2005 2007 2007 2004 2005 2005 2004 2004 2003 2003 2008
[1275] 2009 2004 2003 2008 2006 2005 2009 2004 2003 2007 2008 2005 2003 2004
[1289] 2008 2005 2004 2004 2008 2009 2005 2008 2004 2005 2006 2009 2009 2009
[1303] 2006 2004 2006 2009 2003 2008 2009 2003 2005 2006 2005 2005 2007 2008
[1317] 2003 2004 2009 2003 2007 2005 2006 2005 2006 2004 2003 2004 2003 2004
[1331] 2008 2004 2005 2004 2009 2003 2008 2003 2008 2003 2006 2008 2003 2009
[1345] 2004 2009 2007 2005 2003 2003 2007 2006 2008 2007 2003 2008 2009 2004
[1359] 2005 2003 2006 2005 2007 2004 2004 2005 2006 2009 2009 2004 2003 2004
[1373] 2003 2004 2006 2008 2009 2003 2005 2008 2004 2009 2005 2007 2005 2005
[1387] 2004 2007 2003 2007 2008 2005 2004 2007 2003 2003 2008 2003 2006 2008
[1401] 2003 2008 2006 2009 2004 2008 2006 2003 2007 2004 2009 2008 2004 2005
[1415] 2003 2004 2007 2007 2009 2009 2004 2006 2009 2007 2005 2005 2009 2005
[1429] 2004 2008 2003 2005 2006 2006 2004 2005 2004 2004 2003 2004 2003 2005
[1443] 2003 2004 2003 2003 2006 2004 2007 2008 2005 2008 2005 2003 2006 2006
[1457] 2006 2008 2006 2004 2009 2003 2003 2007 2009 2004 2005 2003 2006 2004
[1471] 2008 2003 2006 2006 2007 2004 2003 2004 2004 2005 2009 2006 2005 2005
[1485] 2008 2007 2008 2003 2003 2007 2004 2008 2007 2008 2008 2003 2004 2008
[1499] 2008 2004 2003 2003 2004 2006 2009 2009 2006 2003 2003 2008 2008 2004
[1513] 2008 2007 2005 2003 2003 2006 2003 2006 2004 2003 2005 2006 2007 2007
[1527] 2009 2006 2006 2004 2004 2003 2004 2007 2003 2007 2005 2004 2007 2006
[1541] 2004 2008 2008 2007 2007 2008 2008 2008 2004 2005 2007 2005 2005 2003
[1555] 2003 2009 2006 2008 2003 2005 2008 2008 2004 2007 2008 2003 2008 2004
[1569] 2009 2004 2007 2004 2008 2004 2009 2005 2006 2003 2003 2006 2006 2009
[1583] 2007 2004 2004 2004 2004 2008 2003 2005 2005 2003 2009 2006 2003 2007
[1597] 2008 2007 2005 2005 2008 2009 2006 2008 2007 2007 2005 2009 2003 2009
[1611] 2009 2008 2006 2003 2008 2003 2006 2006 2004 2006 2008 2004 2004 2007
[1625] 2005 2007 2005 2006 2007 2004 2003 2007 2003 2009 2003 2008 2008 2009
[1639] 2009 2003 2003 2006 2005 2005 2009 2005 2008 2004 2005 2004 2005 2004
[1653] 2009 2005 2004 2007 2004 2004 2009 2005 2003 2005 2003 2009 2003 2003
[1667] 2006 2004 2003 2005 2009 2009 2003 2004 2004 2008 2006 2004 2006 2003
[1681] 2007 2009 2009 2008 2003 2005 2008 2009 2004 2009 2003 2005 2003 2003
[1695] 2008 2005 2009 2006 2007 2003 2005 2005 2004 2007 2007 2009 2009 2006
[1709] 2006 2004 2005 2007 2005 2005 2007 2007 2003 2009 2009 2004 2008 2005
[1723] 2008 2004 2003 2009 2004 2007 2004 2003 2005 2005 2004 2009 2005 2003
[1737] 2004 2009 2003 2005 2004 2007 2004 2006 2008 2007 2008 2004 2005 2007
[1751] 2009 2003 2008 2009 2005 2007 2003 2005 2003 2004 2004 2007 2003 2007
[1765] 2006 2006 2003 2009 2005 2003 2004 2008 2009 2007 2009 2003 2008 2005
[1779] 2006 2006 2004 2006 2004 2009 2005 2009 2003 2004 2006 2006 2005 2004
[1793] 2003 2004 2005 2009 2004 2006 2003 2007 2009 2003 2003 2005 2005 2008
[1807] 2005 2006 2006 2004 2005 2003 2006 2006 2009 2003 2006 2003 2006 2008
[1821] 2006 2006 2004 2006 2008 2008 2005 2003 2008 2003 2006 2007 2009 2008
[1835] 2007 2003 2005 2005 2008 2009 2004 2007 2003 2006 2008 2003 2003 2009
[1849] 2007 2007 2003 2005 2005 2005 2004 2003 2005 2005 2009 2008 2005 2005
[1863] 2006 2006 2003 2004 2008 2003 2005 2004 2007 2007 2009 2007 2006 2003
[1877] 2003 2005 2008 2009 2009 2009 2006 2009 2004 2004 2004 2004 2003 2009
[1891] 2009 2009 2009 2009 2005 2009 2006 2008 2009 2005 2004 2008 2007 2008
[1905] 2007 2009 2007 2004 2004 2004 2007 2008 2005 2005 2008 2004 2005 2009
[1919] 2005 2007 2007 2006 2005 2007 2004 2005 2007 2009 2007 2008 2003 2004
[1933] 2007 2008 2005 2005 2004 2003 2009 2005 2008 2003 2003 2008 2008 2009
[1947] 2004 2006 2005 2008 2006 2009 2004 2009 2009 2004 2004 2003 2004 2005
[1961] 2007 2008 2003 2006 2008 2007 2005 2003 2004 2006 2004 2004 2003 2008
[1975] 2009 2009 2004 2005 2006 2006 2006 2008 2008 2005 2008 2009 2004 2006
[1989] 2004 2007 2008 2005 2007 2003 2009 2004 2007 2005 2007 2009 2003 2005
[2003] 2003 2006 2004 2009 2005 2009 2005 2005 2007 2004 2003 2004 2006 2006
[2017] 2005 2008 2008 2006 2008 2004 2005 2005 2008 2004 2005 2007 2006 2004
[2031] 2006 2009 2004 2003 2004 2009 2008 2006 2008 2003 2005 2009 2008 2008
[2045] 2006 2005 2007 2009 2003 2003 2009 2006 2006 2003 2007 2008 2004 2005
[2059] 2004 2005 2004 2009 2006 2007 2009 2009 2006 2009 2005 2008 2007 2005
[2073] 2005 2006 2008 2005 2003 2003 2004 2008 2004 2004 2004 2004 2007 2006
[2087] 2006 2004 2003 2006 2009 2008 2004 2008 2009 2008 2006 2005 2003 2004
[2101] 2008 2008 2005 2007 2006 2005 2006 2005 2009 2005 2006 2006 2006 2003
[2115] 2004 2005 2004 2004 2008 2009 2003 2005 2004 2006 2009 2006 2003 2007
[2129] 2004 2005 2006 2008 2008 2008 2009 2007 2009 2008 2003 2008 2005 2006
[2143] 2003 2003 2007 2003 2008 2007 2003 2006 2008 2007 2005 2009 2007 2005
[2157] 2008 2007 2007 2005 2003 2005 2008 2006 2004 2004 2009 2003 2006 2007
[2171] 2003 2004 2003 2004 2004 2004 2007 2005 2007 2009 2005 2003 2008 2003
[2185] 2004 2003 2007 2007 2003 2004 2004 2005 2005 2008 2004 2007 2005 2009
[2199] 2007 2004 2004 2004 2006 2006 2003 2008 2009 2005 2008 2003 2005 2006
[2213] 2003 2004 2006 2008 2008 2006 2004 2004 2003 2004 2008 2009 2005 2004
[2227] 2004 2009 2006 2006 2003 2009 2007 2005 2009 2009 2007 2008 2007 2006
[2241] 2004 2005 2005 2005 2004 2009 2003 2008 2008 2009 2004 2008 2005 2008
[2255] 2005 2003 2007 2005 2003 2006 2009 2003 2008 2003 2008 2009 2007 2006
[2269] 2005 2003 2003 2004 2006 2009 2005 2009 2004 2003 2009 2006 2004 2003
[2283] 2007 2004 2003 2003 2007 2004 2007 2009 2007 2008 2004 2005 2009 2008
[2297] 2005 2006 2004 2004 2007 2008 2003 2009 2005 2005 2007 2006 2009 2004
[2311] 2005 2005 2008 2003 2008 2005 2009 2005 2009 2003 2005 2009 2007 2004
[2325] 2003 2003 2004 2004 2005 2008 2004 2009 2008 2003 2006 2006 2004 2003
[2339] 2003 2008 2003 2006 2003 2009 2003 2009 2004 2003 2004 2007 2008 2009
[2353] 2004 2007 2006 2008 2009 2003 2005 2006 2003 2007 2006 2006 2009 2009
[2367] 2008 2003 2008 2003 2004 2003 2004 2004 2007 2006 2009 2006 2006 2005
[2381] 2003 2008 2004 2004 2005 2009 2008 2009 2008 2006 2005 2008 2008 2008
[2395] 2005 2009 2006 2005 2008 2005 2008 2006 2009 2009 2007 2004 2004 2007
[2409] 2004 2003 2006 2003 2005 2004 2003 2008 2009 2006 2007 2004 2008 2004
[2423] 2006 2006 2003 2009 2008 2009 2006 2009 2003 2008 2007 2003 2004 2003
[2437] 2008 2007 2004 2008 2004 2007 2004 2009 2006 2009 2007 2007 2008 2006
[2451] 2004 2008 2003 2007 2006 2004 2008 2009 2009 2008 2007 2005 2005 2004
[2465] 2003 2008 2003 2003 2004 2008 2006 2004 2008 2009 2004 2008 2009 2003
[2479] 2005 2003 2005 2007 2004 2006 2008 2007 2006 2009 2004 2005 2004 2007
[2493] 2009 2008 2009 2005 2005 2009 2005 2007 2006 2008 2004 2008 2006 2004
[2507] 2005 2003 2005 2004 2008 2008 2003 2005 2007 2004 2003 2009 2005 2005
[2521] 2004 2004 2003 2009 2003 2003 2004 2004 2005 2006 2004 2008 2005 2004
[2535] 2004 2009 2003 2004 2003 2008 2007 2007 2005 2008 2003 2008 2006 2005
[2549] 2007 2009 2003 2003 2005 2009 2009 2004 2008 2005 2003 2003 2008 2003
[2563] 2005 2004 2005 2005 2005 2006 2005 2007 2007 2007 2005 2009 2008 2008
[2577] 2003 2003 2003 2008 2006 2003 2008 2004 2005 2007 2009 2007 2005 2003
[2591] 2009 2003 2009 2009 2006 2004 2005 2007 2009 2005 2006 2005 2008 2005
[2605] 2008 2008 2007 2007 2009 2006 2003 2006 2003 2008 2006 2007 2007 2003
[2619] 2008 2005 2007 2008 2003 2008 2005 2003 2003 2008 2003 2004 2005 2007
[2633] 2008 2009 2008 2005 2009 2006 2008 2005 2008 2009 2005 2004 2005 2009
[2647] 2004 2006 2003 2003 2006 2007 2004 2003 2005 2006 2008 2005 2007 2003
[2661] 2008 2006 2003 2006 2003 2005 2004 2006 2004 2009 2003 2007 2003 2007
[2675] 2003 2005 2009 2004 2006 2006 2006 2004 2006 2009 2006 2005 2006 2009
[2689] 2009 2003 2006 2005 2005 2006 2009 2006 2009 2008 2008 2003 2004 2004
[2703] 2006 2004 2005 2003 2005 2004 2004 2006 2008 2007 2006 2008 2003 2007
[2717] 2009 2007 2006 2004 2009 2003 2005 2008 2003 2007 2007 2008 2003 2003
[2731] 2005 2003 2005 2003 2004 2003 2006 2009 2007 2005 2009 2008 2008 2003
[2745] 2009 2005 2007 2006 2004 2004 2008 2006 2009 2009 2006 2007 2003 2004
[2759] 2006 2004 2003 2005 2003 2003 2005 2006 2003 2007 2003 2009 2007 2007
[2773] 2009 2003 2006 2008 2006 2005 2003 2008 2003 2004 2009 2006 2009 2005
[2787] 2008 2004 2008 2008 2006 2004 2004 2004 2005 2009 2003 2008 2006 2003
[2801] 2005 2007 2003 2005 2003 2004 2003 2005 2008 2004 2005 2005 2005 2004
[2815] 2004 2007 2005 2009 2004 2005 2003 2008 2003 2005 2005 2008 2004 2005
[2829] 2004 2008 2007 2005 2004 2003 2009 2006 2004 2009 2009 2004 2008 2009
[2843] 2006 2009 2007 2005 2005 2005 2004 2003 2005 2009 2003 2006 2009 2005
[2857] 2008 2003 2004 2007 2008 2009 2008 2009 2009 2003 2004 2008 2003 2004
[2871] 2004 2003 2009 2003 2008 2003 2004 2004 2006 2008 2007 2004 2007 2005
[2885] 2003 2006 2007 2004 2006 2007 2007 2009 2005 2003 2004 2006 2003 2005
[2899] 2003 2003 2009 2003 2005 2007 2009 2004 2006 2005 2005 2007 2007 2003
[2913] 2007 2003 2006 2004 2008 2003 2003 2009 2004 2006 2007 2007 2005 2005
[2927] 2009 2007 2003 2006 2003 2005 2005 2007 2007 2003 2007 2008 2008 2009
[2941] 2008 2006 2009 2005 2004 2004 2007 2007 2006 2009 2007 2009 2003 2004
[2955] 2003 2003 2003 2007 2007 2006 2004 2007 2009 2006 2004 2007 2003 2003
[2969] 2008 2009 2007 2003 2009 2005 2003 2008 2006 2007 2003 2009 2007 2003
[2983] 2005 2003 2008 2007 2003 2007 2007 2008 2009 2003 2007 2006 2009 2008
[2997] 2007 2005 2005 2009

Matrices revisited

Using the row and column notation just presented, the matrix \(X\) can be written:

\[\begin{equation} \textbf{X} = \left( \begin{array}{c} x_{1}^T \\ x_{2}^T \\ \vdots \\ x_{n}^T \end{array} \right) = (\textbf{x}_1, \textbf{x}_2, \dots, \textbf{x}_p) \end{equation}\]

The \({}^T\) notation denotes the transpose of a matrix or vector, eg \({x}_{i}^T =({x}_{i1}, {x}_{i2}, \dots {x}_{ip})\).

you can think of a matrix as: - \(n\) row vectors or - \(p\) column vectors

remember, since vectors are represented columnwise, we need to transpose the row vectors before stacking them on one another - so \(x_i^T\) denotes the row matrix in its more natural horizontal representation and - \(x_1^T\) stacked on \(x_2^T\), stacked on \(x_3^T\), etc will give us back our matrix capital bold-face X

Output vs. Input Variables

• We use \(x\) to denote our input variable(s) and \(y\) to denote our output variable¹

• For instance, \(y_i\) may refer to the wage of the \(i\)th observation in the Wage data set, whose observed features are stored in \(x_i\).

• The collection of all \(n\) observed outcomes form the vector \(\textbf{y} = (y_1,y_2, \dots, y_n)^T\)

\[\begin{equation} \textbf{y} = \left( \begin{array}{c} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{array} \right) \end{equation}\]

In terms of choice of letters for denoting random varialbes we typically use: - \(x\) for input or predictor variable (features, independent variables) - \(y\) for output or response variable ( dependent variables) - \(y\) is a column vector of length \(n\), we have one “answer” for each observation in our data set

\(N.B.\) \(y\) will only be available in the Supervised setting

1. \(\textbf{y}\) only available in the Supervised setting!

Paired data

Our observed data consists of \(\{(x_1, y_1)\), \((x_2, y_2)\), \(\dots\), \((x_n, y_n)\}\), where each \(x_i\) is a vector of length \(p\).

Visualization of inputs and outputs in tabular data

Assuming we’re in the supervised setting…

• we can think of our observations as a collection of \((x, y)\) pairs

So back to the wage example, we might choose wage to be our output variable \(\textbf{y}\) and use the remaining 10 variables as our input variables \(\textbf{x}\)

Summary

Note there is a potential for ambiguity in the usage of \(a\) (as a vector) and \(a\) as a scalar

Notation	Description	Representation
\(n\)	lower case “n”	number of samples
\(\textbf{x}\), \(\textbf{y}\), \(\textbf{a}\)	lower-case bold	vectors of length \(n\)
\(x\), \(x_i\), \(a\)	lower-case normal	vectors of length \(\neq n\)
\(\textbf{X}\), \(\textbf{A}_{n\times p}\)	capital bold letters	matrix
\(a\)	lower-case normal (beginning of alphabet)	scalars
\(X\)	capital (end of alphabet)	random variables

Why \(a\)?

• vectors that are not of length \(n\) (such as feature vectors of length \(p\)) will be denoted in lower case normal font. - In the rare cases in which these two uses for lower case normal font lead to ambiguity, we will clarify which use is intended.

Ambiguous cases

In the rare cases the use of lower case normal font leads to ambiguity, clarification will be provided using the following notation:

• \(a\) \(\exists \ \mathbf{R}\) if \(a\) scalar or \(a \in \mathbb{R}\)
• \(a\) \(\exists \ \mathbf{R}^k\) if \(a\) is a vector of length \(k\) or \(a \in \mathbb{R}^k\)

We will indicate an \(r \times s\) matrix using:

• \(\textbf{A} \exists\) \(\mathbf{R}^{r\times s}\) or \(\textbf{A} \in \mathbb{R}^{r \times s}\) or \(\textbf{A}_{r \times s}\)

• For completion we use A in R^{r times s} to denote the dimension of a matrix.
  
• Alternatively we may specify the dimension of a matrix using subscripts
• I will avoided matrix algebra whenever possible.
• I will brush up on some (basic) matrix algebra when we need it

Statistical Learning

• SL and ML i will use as synomyms

• this goes through some of chapter

General Model

\[Y = f(X) + \epsilon\]

We assume that data arises from the above formula where:

• \(X=(X_1, X_2, \ldots, X_p)\) are inputs (also referred to as predictors, features, independent variables, among others)

• \(Y\) is the output (also referred to as response, dependent variable, among others)

• \(\epsilon\) is the error term (independent of \(X\) and with mean 0)

• our general learning task can be summarized by this formula

• In this formulation, \(f\) represents the systematic information that \(X\) provides about \(Y\) and

• \(epsilon\) captures the inherent randomness that our model will never be able to capture

• the idea is that this process has some generative model \(f\) represents the systematic information that \(X\) provides about \(Y\)
  

• epsilon captures the random behaviour that we’ll never be able to account for

• ML wants to learn about this mechanism \(f\) that mapped our input variables \(X\) to an output variable \(Y\)

Use of Capital Letters

\[Y = f(X) + \epsilon\]

• Note that we are using capital letters¹ to denote random variables in this context.

• For ease of notation in this section, we use \(X\) to denote the input variable(s) which we distinguish using subscripts.

• eg. using the Wage data set we might consider the input variables: \(X = (X_1, X_2)\) where \(X_1\)= year and \(X_2\)= age and response variable \(Y\) = wage.

• The model this model could be regression, KNN, SVM, gaussian mixture models etc.. and will tell the computer how it should find patterns in the data.

1. We denote observed or sampled data by lower-case.

Deterministic f

\[Y = f(X) + \epsilon\]

• The function \(f\) models the systematic/deterministic part of the relationship between the predictors \(x\) and the response \(y\)

• so notice the importance of the noise term in this model. Even if we know what \(f\) is, we would still never be able to predict random processes with absolute certainty because there is always that stochastic element in \(\epsilon\)

• its form can very, it could be linear, polynomial, or a more complex function depending on the nature of the relationship

• \(f\) it is in generally unknown.

If we knew this we wouldn’t need ML

• Statistical learning is concerned with estimating \(f\).

• we don’t know this and what to estimate with some model
• This model \(f\) can be linear regression, KNN, random forests, svm, gaussian mixture model, etc

• Estimates of \(f\) are denoted by \(\hat{f}\), and \(\hat{Y}\) will represent the resulting prediction for \(Y\).

We will use these estimates a couple of different ways throughout the course - predictions on “new” data for which we don’t know the true “answer” - predictions on “new” data for which we do know the true “answer” to be used to see how well we’re doing

Stochastic \(\epsilon\)

\[Y = f(X) + \epsilon\]

• \(\epsilon\) represents the error term or noise, which captures the random variation in \(y\) that is not explained by \(f(x)\).
• It accounts for the stochastic or unpredictable elements in the response variable, including measurement errors, omitted variables, or inherent randomness.
• It is commonly assumed that \(\epsilon\) has a mean of zero and is normally distributed with constant variance.

• (i.e., the errors are unbiased) and homoscedastic

Model Visualization

All models are wrong but some are useful.

George E. P. Box

Adapted from ISLR

• a famous quote often attributed to the British statistician George E. P. Box

• ILSR authors argue that underlying distribution is never truly linear, though linear regression is extremely useful both conceptually and practically.

• e.g. if we try to predict the number of tourist in Niagara Falls on a given day, will be wrong more that it is right and it will never represent the exact real behaviour. Having said that, even if a model cannot describe exactly the reality it could be very helpful if it is close enough.

• f is a bit of make believe - it is our “nature” in the black box

Simulated Bacteria Growth

This is simulated data representing the growth of bacterial in a population.

• to mimic this omniscient (aam-ni-shnt) scenario where we know \(f\) we often will first work in simulation land.

• This is simulated data and the function used to generate these points were indeed based on the exp function in R (discussed more in lab).

To mimic the real-world, let’s pretend we don’t know \(f\)

Goal of Statistical Learning

• The main objective is to find a function \(\hat f(x)\) that closely approximates the true underlying function \(f(x)\).

• \(\hat f(x)\) is then used to make predictions about \(y\) given new inputs \(x\).

f is only known in simulation

For real (ie. non-simulated data) data, the function \(f\) is generally unknown and must estimated based on the observed points.

• based on only the points, we want to try and discover that underlying exponential red curve

• Importantly, we won’t generally know the underlying relationship (i.e. exponential)

We don’t know the “shape” of \(f(x)\)

• food for thought: we’ll need to come up with ways to assess how well our estimate is \(\hat f\)

Simulate using seeds

set.seed(12345)
x <- seq(from = 0, to = 3, by = 0.1)
y <- exp(x) + rnorm(length(x), mean = 0, sd = 1)
errors <- y - exp(x)

• When simulating data you should also see a seed (in R using set.seed()) for your results to be reproducible.
• A seed is an initial value used by a random number generator (RNG) to start generating a sequence of random numbers.
• In programming, setting a seed ensures that the same sequence of “random” numbers is produced each time the code is run.

in practice we would only have the data points and we need to try and figure it out just based on the data.

Recall

Reasons for finding \(f\) fall into two primary categories:

• Prediction: with inputs \(X\) available, our concern is predicting the output \(Y\).

• Inference: we want to understand the relationship between \(X\) and \(Y\).

Often, we will be interested in both, perhaps to varying extent. Our goal will dictate what choices for \(f\) will be “better” for us.

Motivated by Prediction

• Primary goal is making accurate forecasts on new or unseen data, rather than understanding the underlying relationships between variables.
• Motivation for prediction problems often stem from the situation where \(X\) is cheap but \(Y\) is “expensive”.

is advantageous for prediction problems because more data typically leads to better model performance.

• “expense” could be monetary
  • running an expensive laser for each data point
• could be time insensitive:
  • collecting data from far-away satellites
• could be labour intensive
  • image categorization needing human sorting

Medical Diagnosis:

• Cheap XXX: Patient demographic information, basic symptoms, and routine test results (e.g., blood pressure, temperature).

• Expensive YYY: Diagnosis of a disease requiring complex and costly tests (e.g., MRI scans, genetic testing, or biopsies).

• Using a complicated black box method for \(f\) may be preferable if our many focus is prediction since we are not particularly concerned with the exact form of \(\hat f\).

• We don’t care what are spam detector is looking for or how it works, we just care to sort out our genuine e-mail from spam.

• Generally speaking, the more complex our algorithm, the harder it will be to interpret.

sometimes its better to sacrifice interpretibility to gain an extra 2% accuracy which might equte to an additional 2 billion in revenue.

Motivated by Inference

When the goal shifts from prediction to inference in medical diagnosis, the focus changes from simply predicting whether a patient has a disease to understanding the underlying relationships between the inputs (XXX) and the outcome (YYY).

Rather than just predicting the presence of a disease, we might want to understand how specific factors (e.g., age, gender, or certain symptoms) contribute to the likelihood of the disease. This requires interpretable models that can identify and quantify these relationships.

• Inference aims to answer how \(Y\) affected by \(X\)

• Since \(\hat f\) is used to model this relationship, we don’t want a black box, we want to understand its nuts and bolts.

• Historically, most methods for estimating f have taken a linear form.
• In some situations, such an assumption is reasonable or even desirable.
• But often the true relationship is more complicated, in which case a linear model may not provide an accurate representation of the relationship between the input and output variables.

Some related question may include:

1. Which input variables (predictors) are associated with output variables (response)?

Variable Selection - do i use all my variables or will a subset of them suffice

1. What is the relationship between the response and each predictor? e.g. positive, negative, linear

• Should I expect an increase in wage if i obtain a higher level of education?
• is bacteria increasing linearly or exponentially? at what rate?

Takeaway Message

When choosing the “best” model for a problem, it is important to keep the underlying task in mind.

i.e. prediction or inference - do we care more about getting the right “answer” - or determining the rules to get at that answer

Generally speaking:

• complicated models are often better at prediction but harder to understand;

• simpler models tend to be easier to interpret but will not necessary make accurate predictions.

• If we are mainly interested in inference, then simpler models are much more interpretable.
• If we are only interested in prediction, and the interpretability of the predictive model is simply not of interest.

In many cases a balancing act between the interpretation and accuracy of prediction is needed.

Interpretability vs. accuracy tradeoff

Inspired by Fig 2.7 of ISLR2 (pg 25)

• be aware of the inherent trade-off between accuracy and interpretability in predictive performance
• there will be times when neural networks are needed (eg classification of pictures, videos)

Other pitfalls and considerations:

Why not fit the most complicated model if we are just concerned with prediction? - KISS only use as complicated a model as needed (otherwise you’ll risk “overfitting”) - in other words your findings wont be generalizable to “new” data

General Workflow

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• old data = data we have on hand

Step 1: pick a model

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

pick a model \(f\). throughout the course we will be looking at a variety of ways to do this.

Step 2: Train the model

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• “learn” algorithm/process from data = “train” the model
• trained model = \(\hat{f}\).
• now i want to test that model …

Step 3: Test the model

Get New Data

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• get some new data and feed it into that trained model \(\hat{f}\)

Step 3: Test the model

Make predictions

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• this will produce predictions \(\hat{y}\)

Step 4: Assess

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• we can compare \(\hat{y}\) with the true \(y\) to see how we did
• in future lectures we will discuss different metrics for answering that

Step 5: Rinse Repeat

Source: Introduction to Machine Learning with the Tidyverse. by Dr. Alison Hill. rstudio::conf2020

• based on the results of that check, we can improve our model
• Iterative process!

Question: Why is it important to have new data?

• don’t want to overfit. E.g. cats/dogs are we learning the pattern or are we just memorizing the answers on the pictures we have on hand

Question: any problems with this workflow

Hint: step 3. Usually we only have one data set on hand.

Training vs. Testing

• If we do not have access to another “new” data set (as we often don’t), we can divide our data (randomly) into two non-overlapping sets:

  1. The training set will be the “old” data used to fit the model
  2. The testing set is strictly used to evaluate performance (i.e. the “new data”)

Comments

• A common split would be 80% of data for fitting and 20% for testing; however, the decision is very dependent on the problem and data available.

A word of warning:

• If training set is small, model fit may be poor
• If testing set is small, performance metrics may be unreliable

Workflow Summary

This is the general workflow we’ll be using throughout this course.

Next class we will talk about how we go about doing step 4.