Observed Fisher Information and confidence intervals

Question

I'm trying to put confidence intervals on parameters fitted through MLE through the inversion of the observed Fisher information matrix. More specifically, I define the observed FIM as:

$$ J_{n}(\hat{\theta{}_{n}}) = -\sum^{n}_{i=1}\frac{\partial{}^{2}}{\partial{}\hat{\theta{}_{n}^{2}}}\log{f_{\theta{}_{n}}(X_{i})} $$

Where $\log{f_{\theta{}}(X)}$ is the MLE term. I've seen in [some sources][1] that the confidence intervals for specific parameters can be estimated as (Equation 3.15 in source):

$$ \hat{\theta{}}_{nj} \pm{} c\sqrt{J_{n}(\hat{\theta_{n}})^{-1}_{jj}} $$

My confusion comes from accounting for the total number of data points. Sometimes it seems like these values are scaled by the number of data points, while others [seem to cancel out][2] (below equation 2 on page 2). A definitive answer and source would be really helpful.

Edit:

So to put it in more explicit, applied terms (might be conflating jargon throughout, so bear with me), we have an objective function like:

$$ l_{\theta{}}=\sum_{i=1}^{n}(-\frac{1}{2}\ln(2\pi{})-\frac{1}{2}\ln{\sigma_{i}^{2}}-\frac{1}{2}(\frac{\hat{y}_{i}(\theta{})-y_{i}}{\sigma_{i}})^{2}) $$

where $\theta{}$ is the vector of parameters being fitted, $n$ are the total number of data points in the set, $\sigma{}$ is the standard deviation of the noise around the experimental data, $\hat{y}_{i}(\theta{})$ is the model predicted outlet and $y$ is the measured outlet.

My goal is to minimize a function like this and to then have a covariance matrix around the fitted parameters (which should easily be converted into individual confidence intervals). Once I've reached the minimum, I can calculate the Hessian as:

$$ H_{\theta{}} = \frac{\partial{}^{2}l_{\theta{}}}{\partial{}\theta{}^{2}} $$

I've seen the covariance calculated through $H_{\theta{}}$ in two different ways: through the observed FIM and the expected FIM. If I were to choose the observed FIM, would it just be:

$$ V_{\theta} = H_{\theta}^{-1} $$

where $V_{\theta}$ is the covariance matrix. Would I be missing a scaling factor of $n$?

Then if I were to use the expected FIM, would I simply use:

$$ V_{\theta} = (E({H_{\theta}}))^{-1} $$

where $E$ is meant to be the expectation. [1]: https://www.stat.umn.edu/geyer/s06/5102/notes/fish.pdf [2]: https://arxiv.org/pdf/2107.04620.pdf

Answer 1

I find it easiest to view this as the inversion of a Wald test using the model-based standard error estimate. All of the data points must be used. Sometimes the heavy notation with subscripts, or lacking subscripts, can be confusing.

In order to discuss convergence in distribution the estimator needs to be multiplied by $\sqrt{n}$. The limiting distribution has variance equal to the data generative process, i.e. $\sqrt{n}\{\hat{\theta}-\theta_0\}\overset{D}{\rightarrow}N(0,I^{-1}_1(\theta_0))$. This doesn't mean the sample size is irrelevant for the variance of the MLE. Quite the contrary. It is equivalent to writing $\hat{\theta}\overset{approx}{\sim}N(\theta_0,I^{-1}_n(\theta_0))$ or $\frac{\{\hat{\theta}-\theta_0\}}{I^{-1/2}_n(\theta_0)}\overset{D}{\rightarrow}N(0,1)$.

This notation works if the model has a single scalar parameter $\theta$. In models that have two or more parameters, bold faced vector $\boldsymbol{\theta}$, you might see notation like $I_{11}(\boldsymbol{\theta})$ referring to the 1,1 element of the Fisher information matrix $I_1(\boldsymbol{\theta})$ for a single datum. The same $I_{11}(\boldsymbol{\theta})$ notation might also refer to the 1,1 element of the Fisher information matrix $I_n(\boldsymbol{\theta})$ for all $n$ data points. Unfortunately, there is no standard notation.

Just remember that when it comes time to construct a Wald confidence interval for a scalar $\theta$ it should have the form

$$\hat{\theta}\pm z_{1-\alpha/2}\cdot\hat{\text{se}} $$

where $\hat{\text{se}}$ may be a function of $\hat{\theta}$ and most definitely has $\sqrt{n}$ in its denominator. For example, in a $Y_1,...,Y_n\sim \text{Poisson}(\theta)$ model:

\begin{eqnarray} f(y)&=&\frac{\theta^y\cdot e^{-\theta} }{y!}\\ &&\\ L(\theta)&\propto&\prod \theta^{y_i}\cdot e^{-\theta} \\ &&\\ \ell(\theta)&=&\sum \text{log}\Big(\theta^{y_i}\cdot e^{-\theta} \Big)=\sum \{y_i\text{log}(\theta)-\theta\}\\ &&\\ \frac{d\ell(\theta)}{d\theta}&=& \sum \{y_i/\theta-1\} \\ &&\\ \frac{d^2\ell(\theta)}{(d\theta)^2}&=& \sum \{-y_i/\theta^2\} &&\\ I_n(\theta)&=&nI_{11}(\theta)=-E\Big[\frac{d^2\ell(\theta)}{(d\theta)^2}\Big]=n/\theta &&\\ \text{Var}(\hat{\theta})&=&I^{-1}_n(\theta)=\theta/n\\ &&\\ \hat{\text{Var}}(\hat{\theta})&=&I^{-1}_n(\hat{\theta})=\hat{\theta}/n \end{eqnarray}

The Wald CI for the mean $\theta$ using the model-based standard error estimator would be

$$\hat{\theta}\pm z_{1-\alpha/2}\cdot\sqrt{\hat{\theta}/n} $$

where $\hat{\theta}=\bar{y}$. For a $Y_1,...,Y_n\sim N(\theta,\sigma^2)$ model the Wald CI for the mean $\theta$ using the model-based standard error estimator would be

$$\hat{\theta}\pm z_{1-\alpha/2}\cdot\sqrt{\hat{\sigma^2}/n} $$

where $\hat{\theta}=\bar{y}$ and $\hat{\sigma^2}=n^{-1}\sum (y_i-\bar{y})^2$. Let me know if I have made any mistakes.