# Difference between revisions of "TU Wien:Statistik und Wahrscheinlichkeitstheorie UE (Bura)/Übungen 2019W/5.4"

Unbiasedness of the empirical variance

Let n ≥ 2 and $X_1, \dots, X_n$ be i.i.d. (independent and identically distributed) random varia- bles, with $\sigma^2 := \mathbb V ar(X_1) < \infty$. Calculate the expectation of the empirical variance

$S^2 = \frac 1 {n-1} \sum_{i=1}^n(X_i-\bar X)^2$.

What would have been the expectation if in $S^2$ we had scaled with n instead of n − 1?

## Lösungsvorschlag von Ikaly

### Sample variance

Vorlage:Main The sample variance of a random variable demonstrates two aspects of estimator bias: firstly, the naive estimator is biased, which can be corrected by a scale factor; second, the unbiased estimator is not optimal in terms of mean squared error (MSE), which can be minimized by using a different scale factor, resulting in a biased estimator with lower MSE than the unbiased estimator. Concretely, the naive estimator sums the squared deviations and divides by n, which is biased. Dividing instead by n − 1 yields an unbiased estimator. Conversely, MSE can be minimized by dividing by a different number (depending on distribution), but this results in a biased estimator. This number is always larger than n − 1, so this is known as a shrinkage estimator, as it "shrinks" the unbiased estimator towards zero; for the normal distribution the optimal value is n + 1.

Suppose X1, ..., Xn are independent and identically distributed (i.i.d.) random variables with expectation μ and variance σ2. If the sample mean and uncorrected sample variance are defined as

$\overline{X}\,=\frac 1 n \sum_{i=1}^n X_i \qquad S^2=\frac 1 n \sum_{i=1}^n\big(X_i-\overline{X}\,\big)^2 \qquad$

then S2 is a biased estimator of σ2, because

\begin{align} \operatorname{E}[S^2] &= \operatorname{E}\left[ \frac 1 n \sum_{i=1}^n \big(X_i-\overline{X}\big)^2 \right] = \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n \bigg((X_i-\mu)-(\overline{X}-\mu)\bigg)^2 \bigg] \\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n \bigg((X_i-\mu)^2 - 2(\overline{X}-\mu)(X_i-\mu) + (\overline{X}-\mu)^2\bigg) \bigg] \\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2 - \frac 2 n (\overline{X}-\mu) \sum_{i=1}^n (X_i-\mu) + \frac 1 n (\overline{X}-\mu)^2 \sum_{i=1}^n 1 \bigg] \\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2 - \frac 2 n (\overline{X}-\mu)\sum_{i=1}^n (X_i-\mu) + \frac 1 n (\overline{X}-\mu)^2 \cdot n\bigg] \\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2 - \frac 2 n (\overline{X}-\mu)\sum_{i=1}^n (X_i-\mu) + (\overline{X}-\mu)^2 \bigg] \\[8pt] \end{align}

To continue, we note that by subtracting $\mu$ from both sides of $\overline{X}= \frac 1 n \sum_{i=1}^nX_i$, we get

\begin{align} \overline{X}-\mu = \frac 1 n \sum_{i=1}^n X_i - \mu = \frac 1 n \sum_{i=1}^n X_i - \frac 1 n \sum_{i=1}^n\mu\ = \frac 1 n \sum_{i=1}^n (X_i - \mu).\\[8pt] \end{align}

Meaning, (by cross-multiplication) $n \cdot (\overline{X}-\mu)=\sum_{i=1}^n (X_i-\mu)$. Then, the previous becomes:

\begin{align} \operatorname{E}[S^2] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2 - \frac 2 n (\overline{X}-\mu)\sum_{i=1}^n (X_i-\mu) + (\overline{X}-\mu)^2 \bigg]\\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2 - \frac 2 n (\overline{X}-\mu) \cdot n \cdot (\overline{X}-\mu)+ (\overline{X}-\mu)^2 \bigg] \\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2 - 2(\overline{X}-\mu)^2 + (\overline{X}-\mu)^2 \bigg] \\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2 - (\overline{X}-\mu)^2 \bigg] \\[8pt] &= \operatorname{E}\bigg[ \frac 1 n \sum_{i=1}^n (X_i-\mu)^2\bigg] - \operatorname{E}\bigg[(\overline{X}-\mu)^2 \bigg] \\[8pt] &= \sigma^2 - \operatorname{E}\left[ (\overline{X}-\mu)^2 \right] = \left( 1 -\frac{1}{n}\right) \sigma^2 < \sigma^2. \end{align}