What do the terms dispersion parameter - deviance - and variance of $y_i$ mean?

Question

I am studying GLMs and I am struggling a bit with some of the concepts. I think mine is more of a theoretical issue.

Basically I am a bit confused by the meaning of these three concepts: the variance of y (the response), the dispersion parameter and the deviance.

I know they probably have different meanings, but I thought the would be estimated in the same way, I thought they measure the same thing, but now i am wondering what actually each of them measures.

I know that the variance of $y_i=a(\phi)* V(\mu)$ $\rightarrow$ the product of the dispersion parameter and the variance function.

But then I get lost when it's about estimating $\phi$ (the dispersion parameter).

I also know that the concept of deviance, $D$, is related to $\phi$, because $D$ converges in distribution to $\phi*\chi^2$ (with $n-p$ degrees of freedom).

Would someone please specify the meaning of the three terms? :)

Answer 1

An exponential family has a density of the form

$$ f(y\vert \mu, \phi) = h(y,\phi)\exp \left(\frac{b(\mu)T(y) - A(\mu)}{d(\phi)}\right) $$

Above, $\phi$ is the dispersion parameter. Typically you look at models of the form

$$ f(y_i\vert \mu_i, \phi) = h(y_i,\phi)\exp \left(\frac{b(\mu_i)T(y_i) - A(\mu_i)}{d(\phi)}\right) $$

where $\mu_i = \vec{x}^\top_i\vec{\beta}$ where $\vec{x}_i$ are your covariates and $\vec{\beta}$ are slopes. Thus, all observations share the dispersion parameter. Now to

Basically I am a bit confused by the meaning of these three concepts: the variance of y (the response), the dispersion parameter and the deviance.

...

Would someone please specify the meaning of the three terms? :)

then the dispersion parameter, $\phi$, is used as the variance is

$$\text{Var}(y_i\vert\mu_i,\phi)=A''(\mu_i)d(\phi) = V(\mu_i)d(\phi), \qquad V(\mu)=A''(\mu_i)$$

so the dispersion parameter is a parameter used to scale the variance function $V$. Notice that it is used to scale the observations', $i = 1,\dots,n$, variances by the same factor $d(\phi)$.

The variance of $y_i$ is $\text{Var}(y_i\vert\mu_i;\phi)$. It is the variance you expect given a $\mu_i$ and dispersion parameter $\phi$.

The deviance, as Carl writes, is a generalization of the idea of using the sum of squares of residuals. The deviance is given by

$$D = 2\phi\sum_{i=1}^n\left(\log f(y_i\vert \tilde\mu_i) - \log f(y_i\vert \mu_i) \right)$$

were $\tilde\mu_i$ are the linear predictors in a saturated model. To see that this is a generalization of the sum of squares of residuals, we focus on the distribution of canonical form where $b(\mu) = \mu$ and $T(y) = y$. In this case

$$D = 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right)$$

A linear model has

$$A(\mu) = \mu^2/2,\quad \tilde\mu_i = y_i$$

so

$$\begin{split} D &= 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right) \\ &= 2\sum_{i=1}^n\left(y_i(y_i - \mu_i) - \frac{y_i^2}{2} + \frac{\mu_i^2}{2}\right) \\ &= \sum_{i=1}^n\left(y_i - \mu_i\right)^2 \end{split}$$

which is the usual residual sum of squares.

Answer 2

Statistical dispersion is a generic term that includes variance, standard deviation, and interquartile range. Statistical deviance could mean a criterion for "abnormal" occurrence or behaviour based on an infrequency of occurrence in members of a population (talking to yourself happens infrequently etc.) However, as pointed out below by @EdM statistical deviance has a different definition as well. In statistics, deviance is a quality-of-fit statistic for a model that is often used for statistical hypothesis testing. It is a generalization of the idea of using the sum of squares of residuals in ordinary least squares to cases where model-fitting is achieved by maximum likelihood.