8

In the context of likelihood-based inference, I've seen some notation concerning the parameter(s) of interest which I've found a little confusing.

For example, notation such as $p_{\theta}(x)$ and ${\mathbb E}_{\theta}\left[S(\theta)\right]$.

What is the significance of the parameter ($\theta$) in subscript notation above? In other words, how should it be read?

My first assumption was that it simply meant "with parameter $\theta$"; for example, for $p_{\theta}(x)$, it would read:

"The probability density of $x$ with parameter $\theta$."

However, this probably isn't correct because $p_{\theta}(x) = L(\theta)$ and, in general, $L(\theta)$ is not a distribution (i.e. it does not integrate to unity); hence it can't be a density, can it?

In addition, in the case of ${\mathbb E}_{\theta}\left[S(\theta)\right]$, I'm not sure what it changes relative to ${\mathbb E}\left[(S(\theta)\right]$ (i.e. with the subscript $\theta$ omitted).

In the above $S(\theta)$ and $L(\theta)$ signify the score function and likelihood function respectively.

Tim
  • 108,699
  • 20
  • 212
  • 390
Hugo
  • 81
  • 3
  • 1
    $p_\theta$ is a probability (or density) for each $\theta$, that does not imply that the likelihood is a density function as a function of $\theta$. – Stéphane Laurent May 02 '13 at 23:03
  • 1
    Thanks for your reply! $$$$ So $p_{\theta}(x) \text{ is equivalent to } p(x;\theta)$? $$$$ From this, can I assume that: $p_{\theta}(x) = L(\theta) \text{ but } \int p_{\theta}(x) dx = 1 \neq \int L(\theta) d\theta$ $$$$ And also that ${\mathbb E}_{\theta}(f(x))$ refers to the expectation of $x$ for each $\theta$ such that: ${\mathbb E}_{\theta}(f(x)) = \int f(x)p_{\theta}(x)dx$ $$$$ – Hugo May 02 '13 at 23:56
  • 1
    *Usually* the notation $\text{E}_X()$ represents an expectation with respect to the random variable $X$; if you're in a situation where it makes sense to regard $\theta$ as a random variable (such as a Bayesian context), that would be the intent. If you're *not* in a situation where $\theta$ could be regarded as a random variable, @Hugo's comment would the be meaning I'd look at next. – Glen_b May 03 '13 at 00:30
  • 1
    @Hugo Yes you understand. Rigorously we should always denote the expectation $\mathbb{E}_{P}$ where $P$ is the underlying probability but this is useless when there is only one $P$. Here $\mathbb{E}_\theta$ is a shortcut for $\mathbb{E}_{p_{\theta}}$. The notation $\mathbb{E}_X$ mentioned by Geln_b is appropriate for other contexts but usually I don't like this notation. – Stéphane Laurent May 03 '13 at 06:03

1 Answers1

2

This is mostly answered in comments which I will summarize here.

$p_\theta(x)$ means the same as $p(x; \theta)$, its a shorthand. This is a density with respect to $x$, not with respect to $\theta$. So while by necessity $\int p(x;\theta)\; dx = 1$ it does not follow that $\int p(x;\theta)\; d\theta=1$, it could be anything, including $\infty$.

So, ${\mathbb E}_{\theta}\left[S(\theta)\right]$ is the expectation of $S(\theta)$ with respect to the distribution $p_\theta(x)$. The subscript $\theta$ is there for clarity, not because it is necessary, so ${\mathbb E}\left[(S(\theta)\right]$ has the same meaning. The distribution with respect to which we calculate the expectation should be clear from context, or indicated somehow (like by a subscript).

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467