Theory of Estimation: Unbiased Estimation

I was happy to see that Theory of Estimation was really an easy topic to understand, easier than I always thought.
In this  article, I would teach you in very simple way, the theory of estimation and you would understand it very clearly.

The challenge many have sometimes is caused by lecturers not explaining the concept clear enough, especially from the basics.




1. Basics: What is Estimation?

Estimation is the process involved in systematically inferring the hidden or unobserved variable from a given information set using a mathematical mapping between the unkowns and the knowns as well as a criterion for estimation.

To carry out estimation you need the following:

  • Data: Set of data
  • Estimator: The estimator takes in the data as well as two more items (Objective Function and Model) and then helps us make an estimate.
  • Model: This is a mapping between the knowns(your dataset) and the unknowns (the parameter)
  • Objective Function: This is a mathematical statement the can be mimimized or maximized to find best possible solutions among a set of solutions.


2. Basic Concepts of Estimation

Take note of these three basic concepts

The theory of estimation provides the following to help us in the task of making estimation:

  • Method for estimating the unknowns (eg. model parameters)
  • Means for accessing the ‘goodness’ of the resulting estimates
  • Making confident statements about the true values (how sure we are about the estimate)


3. Unbiased Estimator

A statistic could be defined as an unbiased estimate of a given parameter if the mean of hte sampling distribution of that statistic can be proved to be equal to the parameter being estimated.

Unbiasedness means, that for a large number of observations(samples), the average over all estimations lies close to the true parameter.


4. Sample Mean Example

Given an n-dimensional vector, X1, . . . ,Xn, prove that the extimator for the means μ is unbiased.

To estimate the mean, we use the sample mean as an estimator.

We now prove that the expected value of the estimator is equal to the true mean μ (condition for unbiasedness). This we would do using the linearity of the expected value

From the above, we can conclude that the estimator

is an unbiased estimator of the sample mean. Thank you for your effort in learning. You can reach me if you find anything difficult.

We consider more examples in the following parts.