What is the reason that a likelihood function is not a pdf?

Question

What is the reason that a likelihood function is not a pdf (probability density function)?

Answer 1

We'll start with two definitions:

• A probability density function (pdf) is a non-negative function that integrates to $1$.

• The likelihood is defined as the joint density of the observed data as a function of the parameter. But, as pointed out by the reference to Lehmann made by @whuber in a comment below, the likelihood function is a function of the parameter only, with the data held as a fixed constant. So the fact that it is a density as a function of the data is irrelevant.

Therefore, the likelihood function is not a pdf because its integral with respect to the parameter does not necessarily equal 1 (and may not be integrable at all, actually, as pointed out by another comment from @whuber).

To see this, we'll use a simple example. Suppose you have a single observation, $x$, from a ${\rm Bernoulli}(\theta)$ distribution. Then the likelihood function is

$$ L(\theta) = \theta^{x} (1 - \theta)^{1-x} $$

It is a fact that $\int_{0}^{1} L(\theta) d \theta = 1/2$. Specifically, if $x = 1$, then $L(\theta) = \theta$, so $$\int_{0}^{1} L(\theta) d \theta = \int_{0}^{1} \theta \ d \theta = 1/2$$

and a similar calculation applies when $x = 0$. Therefore, $L(\theta)$ cannot be a density function.

Perhaps even more important than this technical example showing why the likelihood isn't a probability density is to point out that the likelihood is not the probability of the parameter value being correct or anything like that - it is the probability (density) of the data given the parameter value, which is a completely different thing. Therefore one should not expect the likelihood function to behave like a probability density.

Answer 2

Okay but the likelihood function is the joint probability density for the observed data given the parameter $θ$. As such it can be normalized to form a probability density function. So it is essentially like a pdf.

Answer 3

The likelihood is defined as $\mathcal{L}(\theta; x_1,...,x_n) = f(x_1,...,x_n; \theta)$, where if f(x; θ) is a probability mass function, then the likelihood is always less than one, but if f(x; θ) is a probability density function, then the likelihood can be greater than one, since densities can be greater than one.

Normally observations are treated iid, then:
$\mathcal{L}(\theta; x_1,...,x_n) = f(x_1,...,x_n; \theta) = \prod_{j} f(x_j; \theta)$

Let's see its original form:

According to the Bayesian inference, $f(x_1,...,x_n; \theta) = \frac{f(\theta; x_1,...,x_n) * f(x_1,...,x_n)}{f(\theta)}$ holds, that is $\hat{\mathcal{L}} = \frac{posterior * evidence}{prior}$. Notice that the maximum likelihood estimate treats the ratio of evidence to prior as a constant(see answers of this question), which omits the prior beliefs. The likelihood has a positive correlation with the posterior which is based on the estimated parameters. $\hat{\mathcal{L}}$ may be a pdf but $\mathcal{L}$ is not since $\mathcal{L}$ is just a part of $\hat{\mathcal{L}}$ which is intractable.

For example, I don't know the mean and standard variance of a Gaussian distribution and want to get them by training using a lot of observation from that distribution. I first initialize the mean and standard variance randomly(which defines a Gaussian distribution), and then I take one case and fit into the estimated distribution and I can get a probability from the estimated distribution. Then I continue to put the case in and get many probabilities and then I multiply these probabilities and get a score. This kind of score is the likelihood. Hardly can it be a probability of a certain pdf.

Answer 4

I'm not a statistician, but my understanding is that while the likelihood function itself is not a PDF with respect to the parameter(s), it is directly related to that PDF by Bayes Rule. The likelihood function, P(X|theta), and posterior distribution, f(theta|X), are tightly linked; not "a completely different thing" at all.

Answer 5

yoooo, lets make something clear. Likelihood is completely different from probability!, when we want to calculate the probability of for example getting x=0, when x is coming from a normal distribution with miu=0 and sigma=1, we need to define a bin, like 0.01, and integral the probability function there(pdf, in this case normal distribution). so we calculate the integral of normal distribution and put, for instance, -0.01 and 0.01 as input for outcome of integral.

BUTT, in likelihood, we just calculate the point value of the pdf... , this is totally different from the probability of the x to be 0, we just enter input x=0 to the function and get the outcome. for instance, by definition, the function y=1, for x=(0 to 1), the integral of this function is 1, so this can be a pdf, but the point value of each x in (0 to 1) is equal 1, which is not the probability of this ponits, just the value of pdf in those point.

now we get to why likelihood is used in modeling, when we maximize the likelihood function for a set of observed data, with respect to a assumed model(function with parameters to be found), in action this would maximize the probability of those data with respect to that model(function). we work with the likelihood, but in the end, probability (integral of pdf) become maximum as well.

edit1: for the second comment, youre right, my bad, I meant the integral of pdf over the data that we collected. the whole purpose of modeling is to fit a 'decided' model to a set of data in a way that the probability of those data considering our 'decided' model would be maximized. for this probability to be maximized, we need to calculate the integral of pdf over our data. by maximizing likelihood, we maximize the integral as well.