In probability theory and statistics, the Dirichlet-multinomial distribution is a family of discrete multivariate probability distributions on a finite support of non-negative integers. It is also called the Dirichlet compound multinomial distribution (DCM) or multivariate Pólya distribution (after George Pólya). It is a compound probability distribution, where a probability vector p is drawn from a Dirichlet distribution with parameter vector , and an observation drawn from a multinomial distribution with probability vector p and number of trials n. The Dirichlet parameter vector captures the prior belief about the situation and can be seen as a pseudocount: observations of each outcome that occur before the actual data is collected. The compounding corresponds to a Pólya urn scheme. It is frequently encountered in Bayesian statistics, machine learning, empirical Bayes methods and classical statistics as an overdispersed multinomial distribution.
It reduces to the categorical distribution as a special case when n = 1. It also approximates the multinomial distribution arbitrarily well for large α. The Dirichlet-multinomial is a multivariate extension of the beta-binomial distribution, as the multinomial and Dirichlet distributions are multivariate versions of the binomial distribution and beta distributions, respectively.
Specification
Dirichlet-multinomial as a compound distribution
The Dirichlet distribution is a conjugate distribution to the multinomial distribution. This fact leads to an analytically tractable compound distribution. For a random vector of category counts , distributed according to a multinomial distribution, the marginal distribution is obtained by integrating on the distribution for p which can be thought of as a random vector following a Dirichlet distribution:
which results in the following explicit formula:
where is defined as the sum . Another form for this same compound distribution, written more compactly in terms of the beta function, B, is as follows:
The latter form emphasizes the fact that zero count categories can be ignored in the calculation - a useful fact when the number of categories is very large and sparse (e.g. word counts in documents).
Observe that the pdf is the Beta-binomial distribution when . It can also be shown that it approaches the multinomial distribution as approaches infinity. The parameter governs the degree of overdispersion or burstiness relative to the multinomial. Alternative choices to denote found in the literature are S and A.
Dirichlet-multinomial as an urn model
The Dirichlet-multinomial distribution can also be motivated via an urn model for positive integer values of the vector α, known as the Polya urn model. Specifically, imagine an urn containing balls of K colors numbering for the ith color, where random draws are made. When a ball is randomly drawn and observed, then two balls of the same color are returned to the urn. If this is performed n times, then the probability of observing the random vector of color counts is a Dirichlet-multinomial with parameters n and α. If the random draws are with simple replacement (no balls over and above the observed ball are added to the urn), then the distribution follows a multinomial distribution and if the random draws are made without replacement, the distribution follows a multivariate hypergeometric distribution.
Properties
Moments
Once again, let and let , then the expected number of times the outcome i was observed over n trials is
The covariance matrix is as follows. Each diagonal entry is the variance of a beta-binomially distributed random variable, and is therefore
The off-diagonal entries are the covariances:
for i, j distinct.
All covariances are negative because for fixed n, an increase in one component of a Dirichlet-multinomial vector requires a decrease in another component.
This is a K × K positive-semidefinite matrix of rank K − 1.
The entries of the corresponding correlation matrix are
The sample size drops out of this expression.
Each of the k components separately has a beta-binomial distribution.
The support of the Dirichlet-multinomial distribution is the set
Its number of elements is
Matrix notation
In matrix notation,
and
with = the row vector transpose of the column vector . Letting
, we can write alternatively
The parameter is known as the "intra class" or "intra cluster" correlation. It is this positive correlation which gives rise to overdispersion relative to the multinomial distribution.