Properties of Good Estimators

A good estimator should possess the 3 proprietress:  Unbiased, Consistent, and Efficient.

• Unbiased: What you get is what you expected.
• Consistent: When you get more and more samples, the value of the estimator will converge to the true value.
• Efficient: The variance of your estimator should be small.

Abstract enough? let's take a look on the following example:

Is the formula \(bar{X} = \frac{1}{n}\sum X_i\) a good estimator for the 'mean' for a set of random variables $X_i$?

Yes, it is! let's proof it:

As there is no other estimator to compare, we can just focus on the properties of unbiased and consistent.

1. For Unbiased:
$E[\bar{X}] = E[ \frac{1}{n}\sum X_i]$
$E[\bar{X}] = \frac{1}{n}\sum E[X_i]$
$E[\bar{X}] = \frac{1}{n}\sum \mu$
$E[\bar{X}] = \frac{1}{n}n\mu$
$E[\bar{X}] = \mu$
What you get is what you expected, so it is unbiased!

2. Consistent:
$Var[\bar{X}] = Var[\frac{1}{n}\sum X_i]$

$Var[\bar{X}] = \frac{1}{n}^2 Var[\sum X_i]$

Since the observation of $X_i$ are independent to each other, the covariance between $X_i$ and $X_{i-1}$ is zero, the equation can be further written as :
$Var[\bar{X}] = \frac{1}{n}^2 \sum Var[X_i]$
$Var[\bar{X}] = \frac{1}{n}^2 n\sigma_X$
$Var[\bar{X}] = \frac{sigma_X}{\sqrt{n}}$
As the sample size $n$ goes large, the variance of the estimator is zero, so the estimator converge.

Done!