Lecture 4: Sampling Distributions

STAT 205: Introduction to Mathematical Statistics

Dr. Irene Vrbik

University of British Columbia Okanagan

Outline

\(\newcommand{\Var}[1]{\text{Var}({#1})}\)

In this lecture we will be covering

• Sampling Distribution of the mean

• Central Limit Theorem

• Sampling Distribution for Proportions

• Sampling Distribution of Variance

Introduction

• Based on the probabilistic foundations covered in STAT 203, this lecture will cover a fundamental concept in statistics that describes the behavior of the sampling distribution.

• Statistical inference aims to draw meaningful and reliable conclusions about a population based on a sample of data drawn from that population.

Example: loan* data

• Let’s return to our loan50 example from previous lectures.

• This data set represents 50 loans made through the Lending Club platform, which is a platform that allows individuals to lend to other individuals.

library(openintro)
data("loan50")             # data from last lecture
data("loans_full_schema")  # full data 

loan50 just 50 observations

loans_full_schema has 10,000 obs

Population data

• The loans_full_schema comprise 10,000 loans made through the Lending Club platform.

• Let’s pretend for demonstration purposes that this data set makes up all the loans made through the Lending Club and thus represents the population.

• loan50 is therefore a sample from that population.

Histogram: Population vs Sample

hist(loans_full_schema$loan_amount, xlab = "Loan Amount",
     main = "loans_full_schema Population")

hist(loan50$loan_amount,  xlab = "Loan Amount",
     main = "loan50 Sample", breaks = 15)

For the population we might be interested in some parameter, for example, the average loan amount, i.e. the mean value \(\mu\), and it’s standard deviation \(\sigma\).

Parameters

While the population mean is not usually possible to compute, since we have the data for the entire population we can easily calculate it using:

mu = mean(loans_full_schema$loan_amount)
sigma = sd(loans_full_schema$loan_amount)

There average loan amount is $16,361.92

The standard deviation is $10,301.96

If we did not have access to the population information (as is often the case) and instead only had access to a sample, we would instead compute the sample mean:

xbar = mean(loan50$loan_amount)
sample_sd = sd(loan50$loan_amount)

There sample average loan amount is $17,083.00

The sample standard deviation is $10,455.46

Random Samples in R

Of course, a different random sample of 50 loans will produce a different sample mean:

N = nrow(loans_full_schema) # 10,000 observations
sampled50_loans <- loans_full_schema[sample(N, 50), ]
(mean_loan_amount <- mean(sampled50_loans$loan_amount))

[1] 15745

There sample average loan amount is $15,745.00

If we do it again, we get yet another (different) sample mean:

sampled50_loans <- loans_full_schema[sample(N, 50), ]
(mean_loan_amount <- mean(sampled50_loans$loan_amount))

[1] 16183

There sample average loan amount is $16,183.00

In repeated sampling the value of the sample mean would vary from sample to sample.

Sampling Distribution of the mean

Let’s repeat this process many times and keep track of all of the sample mean calculations (one for each sample).

mean_vec = NULL 
for (i in 1:1000) {
  sampled50_loans <- loans_full_schema[sample(N, 50), ]
  mean_vec[i] <- mean(sampled50_loans$loan_amount)
}
mean_vec[1:20]

 [1] 16227.0 16194.0 16127.5 16297.5 13865.0 16121.0 18582.0 15622.5 16526.5
[10] 16262.0 18015.0 17969.5 17211.0 15508.5 19218.0 17669.5 16795.0 16488.0
[19] 18747.5 16850.5

...

Interesting question: what is the distribution of these values?

The distribution of the sample means is an example of a sampling distribution.

For the first sample, \(\bar x_1\) = 16227

For the second sample, \(\bar x_2\) = 16194

For the third sample, \(\bar x_3\) = 16127.5

For the forth sample, \(\bar x_4\) = 16297.5

For the fifth sample, \(\bar x_5\) = 13865

For the sixth sample, \(\bar x_6\) = 16121

this was created in

Stat205/r/sampling-distribution-dots.R

You can play around with simulations like this at StatKey: Sampling Distribution for a Mean

Empirical Sampling Distribution

To get so-called empirical estimate of that distribution we can plot a histogram of it’s values

• We note that when the sample size is at least moderately large, the distribution of the sample mean is approximately normal

• This phenomenon is known as the CLT, which states that a sampling distribution of the mean will approach a normal distribution as the number of trials increases (as long as the sample size is large enough).

Theoretical Sampling Distribution

Thanks to the Central Limit Theorem we actually know what the theoretical probability distribution of \(\overline{X}\) is …

Let \(X_1, X_2, . . . , X_n\) be nindependently drawn observations from a population with mean μ and standard deviation σ. Let X̄ be the sample mean of these \(n\) independent observations. Then

• \(\mu_\overline{X} = \mu\). (The mean of the sampling distribution ofX̄is equal to the meanof the distribution from which we are sampling.)

• \(\sigma_{\overline{X}} = \sigma/\sqrt{n}\) (The standard deviation of the sampling distribution of X̄ is equal to the standard deviation of the distribution from which we are sampling, divided by the square root of the sample size.)

• If the distribution from which we are sampling is normal, the sampling distribution of X̄ is normal

• for large sample sizes the sampling distribution of the sample mean will be approximately normal, regardless of the distribution from which we are sampling.That is the gist of the central limit theorem, but let’s fill in a few details

Theoretical Sampling Distribution

Thanks to the Central Limit Theorem we actually know what the theoretical probability distribution of \(\overline{X}\) is …

CLT for the Sample Mean

This phenomenon is known as the Central Limit theorem.

It says the probability distribution of \(\overline{X}\) approaches a Normal distribution as long as the sample size is large enough¹.

The mean (expected value) of the sampling distribution of the sample mean is equal to the population mean (\(\mu\)) and the standard deviation (more commonly referred to as the standard error) is \(\sigma_\overline{X} = \frac{\sigma}{n}\). We often write:

\[ \begin{equation} \overline{X} \sim N(\mu_\overline{X} = \mu, \sigma_\overline{X} = \sigma/n) \end{equation} \]

1. General rule of thumb is \(n \geq 30\)

iClicker

Understanding Sampling Distributions

A sampling distribution represents:

1. The distribution of a sample.

2. The distribution of a population.

3. The distribution of a statistic, like the sample mean, across repeated samples.

4. The distribution of a parameter, like the population mean.

iClicker

Central Limit Theorem

According to the Central Limit Theorem, the sampling distribution of the sample mean:

1. Always follows a normal distribution regardless of sample size.

2. Becomes approximately normal as the sample size increases.

3. Becomes approximately normal as the number of samples increase.

4. Has a standard deviation equal to the population standard deviation.

5. Is skewed when the population distribution is skewed.

iClicker

Variability of the Sample Mean

As the sample size increases, the variability of the sample mean:

1. Increases.

2. Decreases.

3. Remains constant.

4. Depends on the population distribution.

Random Sample

Definition: Sample

If \(X_1, X_2, \dots, X_n\) are independent random variables (RVs) having a common distribution \(F\), then we say that they constitute a sample (sometimes called a random sample) from the distribution \(F\).

• Oftentimes, \(F\) is not known and we attempt to make inferences about it based on a sample.
• Parametric inference problems assume \(F\) follows a named distribution (e.g. normal, Poisson, etc) with unknown parameters.

for instance, one might suppose that F was a normal distribution function having an unknown mean and variance

Capital letters, e.g. \(X\), imply RVs, i.e. they have not been observed yet. Lower case letters, e.g. \(x\), imply observed values for the corresponding RV \(X\).

Statistic

Definition: (Sample) Statistic

Let’s denote the sample data as \(X = (X_1, X_2, \dots, X_n)\) , where \(n\) is the sample size. A statistic, denoted \(T(X)\), is a function of a sample of observations.

\[ T(X) = g(X_1, X_2, \dots, X_n) \]

• A prime example of a statistic is the sample mean and sample variance
• Understanding that these are both RV, we can calculate things like their Expected value.

• IOW, a sample statistic is a function of these \(n\) RVs, thus defining a new (single) RV.
• For example, if \(T(X)\) is a sample mean, it might be used to estimate the population mean.
• Two important statistics that we will discuss are the sample mean and the sample variance.

Sampling Distribution

Definition: Sampling Distribution

A sampling distribution is the probability distribution of a given a sample statistic.

In other words, if were were to repeatedly draw samples from the population and compute the value of the statistic (e.g. sample mean, or variance), the sampling distribution is the probability distribution of the values that the statistic takes on.

Important

In many contexts, only one sample is observed, but the sampling distribution can be found theoretically.

• If an arbitrarily large number of samples, each involving multiple observations (data points), were separately used in order to compute one value of a statistic (such as, for example, the sample mean or sample variance) for each sample, then the sampling distribution is the probability distribution of the values that the statistic takes on.

• Sampling distributions are important in statistics because they provide a major simplification en route to statistical inference. More specifically, they allow analytical considerations to be based on the probability distribution of a statistic, rather than on the joint probability distribution of all the individual sample values.

Review of CLT

Central Limit Theorem

Let \(X_1, X_2, \dots, X_n\) be a sequence of independent and identically distributed (i.i.d) RVs each having mean \(\mu\) and variance \(\sigma^2\). Then for \(n\) large, the distribution of the sum of those random variables is approximately normal with mean \(n\mu\) and variance \(n\sigma^2\)

\[ \begin{align*} X_1 + X_2 + \dots + X_n &\sim N(n\mu, n\sigma^2) \end{align*} \]

\[ \begin{align*} \dfrac{X_1 + X_2 + \dots + X_n}{n} &\sim N\left(\mu, \dfrac{\sigma^2}{n}\right) \end{align*} \]

\[ % Z_n = \dfrac{X_1 + \dots + X_n - n\mu}{\sigma/\sqrt{n}} = \dfrac{\sum_{i=1}^n X_i - \mu}{\sigma/\sqrt{n}} \rightarrow N(0,1) Z_n = \dfrac{\overline{X} - \mu}{\sigma/\sqrt{n}} \rightarrow N(0,1) \text{ as } n \rightarrow \infty \]

• The Central Limit Theorem (CLT) is a fundamental concept in statistics that describes the behavior of the sampling distribution of the sample mean as the sample size increases, regardless of the original distribution of the population.
• The numerator shrinks to 0 as \(n \to \infty\) because \(\bar{X}\) is converging to \(\mu\)
• The denominator also shrinks, since \(\frac{\sigma}{\sqrt{n}}\)​ decreases as \(n\) grows.

Key Idea:

• If we just looked at \(\bar{X}\) it would converge to \(\mu\) with a vanishing standard deviation.

• But when we standardize using \(Z_n\)​, the random fluctuations remain on the same scale for all \(n\), which is why the limiting distribution is standard normal.

However, the rate of shrinking is exactly balanced in such a way that their ratio remains a random variable with constant variance.

\(N(O,1)\) is called the standard normal distribution.

Sample Mean

A prime example of a sample statistic is the sample mean:

\[ \overline{X} = \dfrac{X_1 + X_2 + \dots, + X_n}{n} \]

This is simply another RV having it’s own:

• expected value \(\mathbb{E}[\overline{X}]\) (i.e. “mean of the sampling mean”)
• variance \(\Var{\overline{X}}\)
• and distribution (i.e. sampling distribution)

Expected Value of the Sample Mean

Assuming \(X_i \overset{\mathrm{iid}}{\sim} F\) where \(F\) has a mean of \(\mu\) and variance \(\sigma^2\):

\[ \begin{align*} \class{fragment}{\mathbb{E}[\overline{X}]} &\class{fragment}{{} = \mathbb{E}\left[\dfrac{X_1+ \dots X_n}{n}\right]} \\ &\class{fragment}{{} = \dfrac{\mathbb{E}[X_1]+ \dots \mathbb{E}[X_n]}{n}} \\ &\class{fragment}{{} = \dfrac{\sum_{i=1}^n \mathbb{E}[X_i]}{n}} \\ &\class{fragment}{{} = \dfrac{n\mu}{n}} \class{fragment}{{} = \mu} \end{align*} \]

We denote \(\mathbb{E}[\overline{X}] = \mu_{\overline{X}}\)

Variance of the Sample Mean

\[ \begin{align*} \text{Var}\left(\overline{X}\right) &\class{fragment}{{} = \text{Var}\left(\frac{X_1 + \dots + X_n}{n}\right)} \\ & \class{fragment}{{} \text{by independence ...}}\\ & \class{fragment}{{} = \frac{1}{n^2} \left[\text{Var}(X_1) + \dots + \text{Var}(X_n)\right]} \\ & \class{fragment}{{} = \frac{n \sigma^2}{n^2} = \frac{\sigma^2}{n}} \end{align*} \]

We denote \(\text{Var}(\overline{X}) = \sigma^2_{\overline{X}}\), and call \(\sigma_{\overline{X}}\) the standard error of the sample mean.

Rolling a die

Example: Rolling a die

What is the sampling distribution of the mean of 5 rolls of a dice? \(\overline{X} \sim ?\)

We’ll solve this first theoretically using the CLT and then empirically using simulation.

We first need to identify the distribution of \(X_i\)

Mean and standard error

\(\mathbb{E}(X_i)\)

• \(\mathbb{E}(X_i) = \frac{a + b}{2} = \frac{1 + 6}{2} =\) 3.5 and
• \(\text{Var}(X_i) = \frac{(b-a+1)^2-1}{12} = \frac{(6 - 1 + 1)^2 - 1}{12}\) 2.9166667

Theoretical Result

Population Distribution: \(X_i \sim Unif(1,6)\) with mean and var cal on previous page

Let \(X_i\) be the number of dots facing up when rolling a fair die.

When we roll the die 5 times and calculate the sample mean, the sampling distribution of the mean has the following properties:

• The mean of the sampling distribution (\(\mu_{\bar{x}}\)) is equal to the population mean

• The standard error of the mean (\(\text{SE}\)) is equal to the population sd/ sqrt(n)

Let \(\overline{X} = \frac{X_1 + X_2 + \dots X_5}{5}\) be the mean of five rolls.

Then from the CLT we know: \[ \begin{align*} \overline{X} &\sim N(\mu_{\overline{X}} = 3.5, \sigma_{\overline{X}} = \frac{\sqrt{((6-1+1)^2 -1)/12)}}{\sqrt{5}})\\ &\sim N(\mu_{\overline{X}} = 3.5, \sigma_{\overline{X}} = 0.7637626) \end{align*} \]

The mean of the sampling distribution (\(\mu_{\bar{x}}\)) is equal to the population mean: \[ \mu_{\bar{x}} = \mu = 3.5 \]

The standard error of the mean (\(\text{SE}\)) is: \[ \text{SE} = \frac{\sigma}{\sqrt{n}} = \frac{\sqrt{\frac{35}{12}}}{\sqrt{5}} \approx 0.764 \]

Empirical Result

The sampling distribution of the mean of 5 rolls of a fair die:

• Mean: \(3.5\) and Standard Error: \(\approx 0.764\)
• Approximate normal, but slightly influenced by the uniformity of the die’s outcomes.

The shape of the sampling distribution depends on \(n\): demo Baseball players https://www.lock5stat.com/StatKey/sampling_1_quant/sampling_1_quant.html

• For small \(n\), the distribution may not look perfectly normal and instead resemble the uniform distribution of a single roll.
• However, with \(n = 5\), the Central Limit Theorem suggests that the distribution will approximate a normal distribution, though not perfectly, due to the small sample size.

n = 1

with a sample size of one we just get back the original distribution

The sample mean directly reflects the shape of the population distribution.

n = 10

n = 30

Same shape but what about the x-axis?

Difference Sample Sizes

• smaller sample sizes lead to sampling distributions with a higher variance (or standard error)
• a bigger sample tends to provide a more precise point estimate than a smaller sample.
• sample means become more and more reliable as an estimate of μ as the sample size is increased, as we would expect.

Solar Energy Example

Example: Solar Energy

Suppose we are interested in the proportion of American adults who support the expansion of solar energy. Suppose we don’t have access to the population of all American adults, which is a quite likely scenario, and we sample 1000 American adults and 895 approve of solar energy. What is the sampling distribution of the proportion?

Sampling Distribution for Proportions

CLT for proportions

When observations are independent and the sample size is sufficiently large, the sample proportion \(\hat p\) is given by

\[ \hat p = \frac{X_1 + X_2 + \dots X_n}{n} \rightarrow N\left(\mu_{\hat p} = p, \sigma_{\hat p} = \sqrt{\frac{p(1-p)}{n}}\right) \]

Since \(\mathbb{E}(X_i) = p\) and \(\text{Var}(X_i) = p(1-p)\) it follows from the CLT that for large \(n\)

ˆp will tend to follow a normal distribution with the following mean and standard error:

Success-failure condition

In order for the Central Limit Theorem to hold, the sample size is typically considered sufficiently large when \(np \geq 10\) and \(n(1-p) \geq 10\) , which is called the success-failure condition.

PROBLEM: this relies on \(p\) a parameter we don’t know

Standard error of proportion

\[\begin{align*} \sigma_{\hat p} &= \sqrt{\frac{p(1-p)}{n}} \end{align*}\]

To approximate this we sub \(p\) (the unknown population parameter) with the point estimate \(\hat p\): \[\begin{align*} \hat{\sigma_{\hat p}} &\approx \sqrt{\frac{\hat p(1-\hat p)}{n}} \\ \end{align*}\]

This is sometimes called the plug-in principal.

Empirical Sampling distribution of Proportion

Because I have simulated this data, we can compare the empirical sample distribution to the theoretical sampling distribution (red) with the sampling distribution using the estimate of standard error (blue)

compare where our observed sample lied (\(\hat p = 896/1000\))

why do we care?

Now that we know the theoretical sampling distribution, we can get a sense of how “extreme” our sample was.

Understanding the curve we can now get a plus/minus for our estimate… (all to come)

Sampling Distribution of Variance

The sample variance, \(S^2\), is yet another sample statistic which we define as \[S^2 = \frac{\sum_{i=1}^{n} (X_i - \overline{X})^2}{n-1}\]

It can be shown that this is an unbiased estimate for \(\sigma^2\), i.e. \[\mathbb{E}[S^2] = \sigma^2\]

Regardless of the distribution from which we are sampling ..

\(S = \sqrt{\frac{\sum_{i=1}^{n} (X_i - \overline{X})^2}{n-1}}\) is called the sample standard deviation.

Expected value of Variance

Using the following algebric result (see Proof):

\[ \sum_{i=1}^{n} \left( x_i - \overline{x} \right)^2 = \sum_{i=1}^{n} x_i^2 - n\overline{x}^2 \] we can rewrite the statistic as:

\[ (n-1) S^2 = \sum_{i=1}^{n} (X_i - \overline{X})^2 = \sum_{i=1}^{n}X_i^2 - n \overline{X}^2 \]

Proof

\[\begin{align*} \sum_{i=1}^{n} \left( x_i - \overline{x} \right)^2 &= \sum_{i=1}^{n} \left( x_i^2 - 2x_i\overline{x} + \overline{x}^2 \right) \\ &= \sum_{i=1}^{n} x_i^2 - 2\overline{x} \sum_{i=1}^{n} x_i + \sum_{i=1}^{n} \overline{x}^2 \\ &= \sum_{i=1}^{n} x_i^2 - 2\overline{x} (n\cdot \overline{x}) + n\cdot \overline{x}^2 \\ &= \sum_{i=1}^{n} x_i^2 + \overline{x}^2(n - 2n) \\ &= \sum_{i=1}^{n} x_i^2 - n\overline{x}^2 \end{align*}\]

Expected value of Variance (cont’d)

Taking the expectation of both sides … \[ (n-1)\mathbb{E}[S^2] = \mathbb{E}\left[\sum_{i=1}^{n} X_i^2\right] - n\mathbb{E}[\overline{X}^2] \] Recall for any random variable \(W\), \(\mathbb{E}[W^2] = \text{Var}(W) + (\mathbb{E}[W])^2\) \[\begin{align*} &= n\text{Var}(X_i) + \left(\sum_{i=1}^n \mathbb{E}[X_i]\right)^2 - n\left( \text{Var}(\overline{X}) + (\mathbb{E}[\overline{X}])^2 \right)\\ &= n\sigma^2 + n\mu^2 - n \text{Var}(\overline{X}) - n\mu^2 \\ &= n\sigma^2 - n \text{Var}(\overline{X}) \\ &= n\sigma^2 - n \frac{\sigma^2}{n} = (n-1)\sigma^2 \end{align*}\]

Or \(E[S^2] = \frac{(n-1)\sigma^2}{(n-1)} = \sigma^2\)

• That is, the expected value of the sample variance is equal to the population variance
• Notice this does not depend of the population distribution

Sampling Distribution of Variance

Normal Population

Let \(X_1, X_2, \dots, X_n\) be a random sample from a normal distribution with mean \(\mu\) and variance \(\sigma^2\). It can be shown that

\[\begin{align*} \dfrac{(n−1)S^2}{\sigma^2} \sim \chi^2_{(n-1)} \end{align*}\] where \(\chi^2_{(n-1)}\) denotes a chi-squared distribution with \(n−1\) degrees of freedom.

Normality Assumption

Warning

It’s important to note that this result holds under the assumption of normality for the underlying population. If the population is not normal, the distribution of the sample variance may still be approximately chi-squared if the sample size is sufficiently large due to the central limit theorem.

Simulation 1: Normal Population

Example: Chi-square simulation (Normal population)

Suppose we take 10000 samples of size \(n =\) 5 from a normal population with \(\sigma^2 =\) 1. Plot the empirical distribution from 10000 runs alongside the theoretical distribution of \(\frac{(n-1)S^2}{\sigma^2}\).

n = 10

n = 20

n = 100

Simulation 2: Uniform Population

Example: Chi-square simulation (Uniform population)

Suppose we take 10000 samples of size \(n =\) 5 from a uniform population with \(\sigma^2 =\) 1. Plot the empirical distribution of \(\frac{(n-1)S^2}{\sigma^2}\) from 10000 runs alongside the theoretical distribution of \(\frac{(n-1)S^2}{\sigma^2}\) if the population were normal.

hist = emprical distribution of \(\frac{(n-1)S^2}{\sigma^2}\)
curve = theoretical dist of \(\frac{(n-1)S^2}{\sigma^2}\) if pop was normal

• hist \(\neq\) curve
• curve has more skewness
• there aren’t as many extreme values

n = 10

• hist \(\neq\) curve
• curve has more skewness
• there aren’t as many extreme values
• hist more compact (smaller variance than curve)

n = 20

clearly the sampling distribution of this statistic is not as variable as when we are sampling from a normally distributed population

• you can understand why if you look at the pdf for a normal disribution with variance 0 vs a uniform dist with variance 0.

• sampling doesn’t result in extreme values so the variance of the sampling distribution statistic is less than if we were sampling from a normally distributed

n = 100

why? Both of these PDFs have a variance of 1.

Implication: we are not going to get extreme values from a uniform distribution so the sample variance will not take on large values as often as it does when we are sampling from a normal distribution