In information theory, the cross-entropy between two probability distributions and over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution , rather than the true distribution .
Definition
The cross-entropy of the distribution relative to a distribution over a given set is defined as follows:
,
where is the expected value operator with respect to the distribution .
The definition may be formulated using the Kullback–Leibler divergence , divergence of from (also known as the relative entropy of with respect to ).
where is the entropy of .
For discrete probability distributions and with the same support this means
The situation for continuous distributions is analogous. We have to assume that and are absolutely continuous with respect to some reference measure (usually is a Lebesgue measure on a Borel σ-algebra). Let and be probability density functions of and with respect to . Then
and therefore
NB: The notation is also used for a different concept, the joint entropy of and .
Motivation
In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value out of a set of possibilities can be seen as representing an implicit probability distribution over , where is the length of the code for in bits. Therefore, cross-entropy can be interpreted as the expected message-length per datum when a wrong distribution is assumed while the data actually follows a distribution . That is why the expectation is taken over the true probability distribution and not . Indeed the expected message-length under the true distribution is
Estimation
There are many situations where cross-entropy needs to be measured but the distribution of is unknown. An example is language modeling, where a model is created based on a training set , and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, is the true distribution of words in any corpus, and is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:
where is the size of the test set, and is the probability of event estimated from the training set. In other words, is the probability estimate of the model that the i-th word of the text is . The sum is averaged over the words of the test. This is a Monte Carlo estimate of the true cross-entropy, where the test set is treated as samples from .
Relation to maximum likelihood
In classification problems we want to estimate the probability of different outcomes. Let the estimated probability of outcome be with to-be-optimized parameters and let the frequency (empirical probability) of outcome in the training set be . Given N conditionally independent samples in the training set, then the likelihood of the parameters of the model on the training set is
where the last expression is due to the definition of the multinomial PMF. Therefore, the log-likelihood, divided by is
so that maximizing the likelihood with respect to the parameters is the same as minimizing the cross-entropy.
Cross-entropy minimization
Cross-entropy minimization is frequently used in optimization and rare-event probability estimation. When comparing a distribution against a fixed reference distribution , cross-entropy and KL divergence are identical up to an additive constant (since is fixed): According to the Gibbs' inequality, both take on their minimal values when , which is for KL divergence, and for cross-entropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.
However, as discussed in the article Kullback–Leibler divergence, sometimes the distribution is the fixed prior reference distribution, and the distribution is optimized to be as close to as possible, subject to some constraint. In this case the two minimizations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by restating cross-entropy to be , rather than . In fact, cross-entropy is another name for relative entropy, see Cover and Thomas and Good. On the other hand, does not agree with the literature and can be misleading.