Bayes estimator-The Daily Ledger

In estimation theory and decision theory, a Bayes estimator or a Bayes action is an estimator or decision rule that minimizes the posterior expected value of a loss function (i.e., the posterior expected loss). Equivalently, it maximizes the posterior expectation of a utility function. An alternative way of formulating an estimator within Bayesian statistics is maximum a posteriori estimation.

Definition

Suppose an unknown parameter is known to have a prior distribution . Let be an estimator of (based on some measurements x), and let be a loss function, such as squared error. The Bayes risk of is defined as , where the expectation is taken over the probability distribution of : this defines the risk function as a function of . An estimator is said to be a Bayes estimator if it minimizes the Bayes risk among all estimators. Equivalently, the estimator which minimizes the posterior expected loss for each also minimizes the Bayes risk and therefore is a Bayes estimator.

Bayes estimator

If the prior is improper then an estimator which minimizes the posterior expected loss for each is called a generalized Bayes estimator.

Bayes estimator

Examples

Minimum mean square error estimation

The most common risk function used for Bayesian estimation is the mean square error (MSE), also called squared error risk. The MSE is defined by

Bayes estimator

where the expectation is taken over the joint distribution of and .

Posterior mean

Using the MSE as risk, the Bayes estimate of the unknown parameter is simply the mean of the posterior distribution,

This is known as the minimum mean square error (MMSE) estimator.

Bayes estimators for conjugate priors

If there is no inherent reason to prefer one prior probability distribution over another, a conjugate prior is sometimes chosen for simplicity. A conjugate prior is defined as a prior distribution belonging to some parametric family, for which the resulting posterior distribution also belongs to the same family. This is an important property, since the Bayes estimator, as well as its statistical properties (variance, confidence interval, etc.), can all be derived from the posterior distribution.

Conjugate priors are especially useful for sequential estimation, where the posterior of the current measurement is used as the prior in the next measurement. In sequential estimation, unless a conjugate prior is used, the posterior distribution typically becomes more complex with each added measurement, and the Bayes estimator cannot usually be calculated without resorting to numerical methods.

Following are some examples of conjugate priors. If is Normal, , and the prior is normal, , then the posterior is also Normal and the Bayes estimator under MSE is given by

If are iid Poisson random variables , and if the prior is Gamma distributed , then the posterior is also Gamma distributed, and the Bayes estimator under MSE is given by

If are iid uniformly distributed , and if the prior is Pareto distributed , then the posterior is also Pareto distributed, and the Bayes estimator under MSE is given by

Alternative risk functions

Risk functions are chosen depending on how one measures the distance between the estimate and the unknown parameter. The MSE is the most common risk function in use, primarily due to its simplicity. However, alternative risk functions are also occasionally used. The following are several examples of such alternatives. We denote the posterior generalized distribution function by .

Posterior median and other quantiles

A "linear" loss function, with , which yields the posterior median as the Bayes' estimate:

Another "linear" loss function, which assigns different "weights" to over or sub estimation. It yields a quantile from the posterior distribution, and is a generalization of the previous loss function:

Posterior mode

The following loss function is trickier: it yields either the posterior mode, or a point close to it depending on the curvature and properties of the posterior distribution. Small values of the parameter are recommended, in order to use the mode as an approximation ():

Other loss functions can be conceived, although the mean squared error is the most widely used and validated. Other loss functions are used in statistics, particularly in robust statistics.