2

Let's suppose that the considered set of random variable has a covariance matrix which is psd. Therefore the Gaussian pdf must be written in its degenerate form, where the determninat of the covariance matrix is replaced by the pseudodeterminant (i.e. the product of non-zero egeinvalues) and the inverse of the covariance matrix is replaced by the pseudoinverse (Wikipedia link2 to stats.stackexchange).

My question is whether we are still able to make use of the information criteria and tests used to specify the model in the non-degenerate case, like the AIC, BIC, LR Test (Wald test, etc...). More precisely, given a number of observed samples t and a number of parameters k used in the model

$$AIC=-2Loglik+2k$$ $$BIC=-2Loglik+ln(T)2k$$ $$LRstatistic=2(Loglik_{fullmodel}-Loglik_{restrictedmodel})$$

Can these metrics be used in the current form reported above, even when a degenerate multivariate normal distribution used to compute the Loglik? For example, does the LRstatistic in this case preserve its Chi-sqaured distribution as for the standard non-degenerate case?

Clearly, a good hint on the way to solve the problem is to analyze the empirical distribution of LR test statistic using a simulation and the derivation of Akaike's and other ICs to see whether the non-singularity of the varcov matrix is strictly necessary to the derivation of the ICs. As far as the latter point is concerned, I have checked the derivation of Akaike's IC (which is also available here) and, in my opinion, I do not see that the non-singularity of the varcov matrix is strictly necessary. But I would like to hear the forum opinion on the point.

EDIT: notice that here we are talking about a probability model and the support of the varcov matrix is potentially allowed to vary across different specifications to be found with ICs/LR test

Fr1
  • 1,348
  • 3
  • 10
  • 2
    Please provide some context for understanding this question: exactly what do you mean by "the considered set of random variables" and what is your "covariance matrix"? For instance, are you supposing some kind of probability model with a degenerate covariance matrix, or is it perhaps the case that the *estimated* covariance matrix for a set of *estimates* is nondegenerate? And in the first case, is the support of the covariance always the same subspace for every instance of the model or does the support vary? – whuber Aug 02 '19 at 14:42
  • @whuber I am supposing a probability model and allowing the support to vary across different possible model specifications (I upvoted because the question was useful) – Fr1 Aug 02 '19 at 14:45
  • Can you maybe write down the model and it's implied covariance matrix and elaborate a bit on the reason for the degeneracy? – StoryTeller0815 Aug 02 '19 at 16:03
  • @StoryTeller0815 my question was in broad sense, as I am generically working with degenerate cases, but in order to provide you with a real-world example leading to degeneracy for me, let's assume you have a very large MLE model to forecast a $nx1$ process, and you want to proxy the covariance matrix of the $n$ residuals through some PCA, so that you will use just a subset of eigenvectors and eigenvalues to represent it, ending up with a $nxn$ covariance matrix which however has not full rank because it is replicated through a smaller subset of its n eigenvectors and associated eigenvalues. – Fr1 Aug 02 '19 at 16:40
  • @StoryTeller0815 clearly in that case, you will be forced to use a different formula for the distribution for each sample observation because otherwise $det(\sigma)=0$ and $\sigma^{-1}$ would not exist due to the signularity of $\sigma$, where $\sigma$ denotes here the $nxn$ varcov matrix to be used in the MLE formula for each sample observation of the process – Fr1 Aug 02 '19 at 16:44
  • In the context, I work in, we would try to find reasonable constraints such that the model-implied covariance matrix ist positive definite again. For instance, the residual covariance matrix may be constrained by distance constraints (the further away in time and space, the lower the correlations). This way, large covariance matrices of residuals may be reduced to much fewer parameters. Others would go for regularized or Bayesian estimation. So, I guess, my question is why you cannot resolve the degeneracy any other way. [PS: Sorry, I did not mean to distract, I just got interested.] – StoryTeller0815 Aug 02 '19 at 16:52
  • 1
    @StoryTeller0815 yes also in my context usually we use constraints for a pd varcov, but on this occasion we cannot set constraints because, for reasons a bit long to explain here, the residuals must by definition have a psd varcov. I read that going Bayesian is a good way to overcome the issue. However, working both analytically (i.e. assuming my varcov is any varcov and solving with symbols of each scalar in the varcov) and numerically, I solved the problem reaching the expected solution. So I was wondering, why not staying in the degenerate MLE case? Especially if it worked numerically. – Fr1 Aug 02 '19 at 17:11
  • 1
    For the numerical resolution I used Python Scipy: indeed in the docs they state that if the varcov is singular then the formulas used will be automatically updated to the degenerate case in order to allow a numerical resolution https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.multivariate_normal.htm and I must admit that the numerical solution of the optimization converged exactly where we were expecting for all the experiments and simulations – Fr1 Aug 02 '19 at 17:12

1 Answers1

1

At least for AIC/BIC, I am very sure, that these measures do not care at all as long as your likelihood function is valid.

Based on the most extensive resources on information theoretic criteria, I believe that it does not matter at all. Of course, AIC/BIC may be "rescaled" if you change the likelihood function and hence, you should not compare these AICs to AICs from models with other likelihood functions. But apart from that, I don't see why they should become invalid as they are just approximations of the distance between the model and reality.

References: Multimodel Inference: Understanding AIC and BIC in Model Selection Kenneth P. Burnham and David R. Anderson Sociological Methods Research 2004; 33; 261 DOI: 10.1177/0049124104268644

StoryTeller0815
  • 513
  • 2
  • 9
  • thank you great answer, great sources, upvoted. Can you please tell me what do you mean exactly by “rescaled” when you say “AIC/BIC must be rescaled when you change the likelihood function”? – Fr1 Aug 02 '19 at 17:55
  • You're welcome. Per se the absolute value of the likelihood (and therefore AIC/BIC) is meaningless. Only differences matter because these have a scale (this is explained in detail in the second reference). If you use a different likelihood between two models, you may run into trouble. A famous case of this occurs in the context of the linear mixed model. If you use REML estimation, the likelihood is rescaled whenever you change your fixed effects (because these are essentially partialed out) - hence, you may not use AIC/BIC to compare REML models with differing fixed effects. – StoryTeller0815 Aug 02 '19 at 18:05
  • The same *may* occur here, when you try to compare a degenerate normal distribution likelihood model and the traditional normal distribution likelihood model by AIC/BIC. I don't know if this happens. I just wanted to warn you that it might happen. – StoryTeller0815 Aug 02 '19 at 18:06