1

I'm comparing Bayesian (generalized) linear methods vs Frequentist ones in the case when $p>n$ ($p,n$ being respectively number of regressors, number of samples).

In the frequentist context when $p$ > $n$ I cannot invert the design matrix $X$ since it is not full rank. Of course there are methods like Ridge Regression to deal with this problem, but I don't want to consider them here. So in that case there is an identifiability issue.

My question is this:

Is there any case in which the same problem (impossibility of finding regression coefficients when $p>n$) arises in a Bayesian context? For instance, considering the simple linear model, when I specify the priors as Normal-Invgamma (like here) I have no need to invert the design matrix to obtain the posterior parameters, solving in that way the problem which occurs in the frequentist case.

I'm dealing with pretty complex multilevel models and I let STAN do the computations, so matrix algebra (which is the problem in the linear context) is not something I care about, may be wrongly. Thanks to @Tim I read Michael Betancourt answer in stanusersgroup look here and what I understood is that the prior imposition constraints the posterior sampling to a convex set, not always solving the identifiability problem,but at least allowing to spot high correlations between the regressors.

Thank you in advance

Tommaso Guerrini
  • 1,099
  • 9
  • 27
  • Inverting matrices has *nothing* to do with Bayesian vs frequentist methods... Bayesians as well use matrix algebra. Moreover, there are frequentist methods for p > n cases. – Tim Apr 19 '17 at 08:33
  • 1
    Then please make your question more precise: that you are asking if there are problems with n > p when working with generalized linear models when using Bayesian vs frequentist methods. – Tim Apr 19 '17 at 08:40
  • @Tim Do you think the question is clear enough now?Thank you – Tommaso Guerrini Apr 19 '17 at 12:35
  • 1
    Yes, but (1) Bayesian estimation also uses matrix algebra, so this has nothing to do with frequentist methods; (2) it is also about [model identifability](https://stats.stackexchange.com/questions/20608/what-is-model-identifiability#). – Tim Apr 19 '17 at 12:39
  • Sorry @Tim for asking other questions here, as soon as the context will be clearer to me I'll either modify the question again or remove it: where do you use matrix algebra in Bayesian estimation? I mean, I'm working with pretty complex multilevel models and I let STAN do the work. I'm lacking knowledge about matrix algebra in a Bayesian context if not for simple normal-inv.gamma linear models.. Thank you again – Tommaso Guerrini Apr 19 '17 at 12:44
  • 1
    Check e.g. https://en.wikipedia.org/wiki/Bayesian_linear_regression – Tim Apr 19 '17 at 14:29
  • Yep I checked it, but it refers to a case where, if not using uninformative gpriors, the bayesian models don't have the issue of the frequentist ones. Thank you again for the help.. – Tommaso Guerrini Apr 24 '17 at 13:16

0 Answers0