2

A penalized regression provides biased estimates of the regression coefficients (bias-variance trade-off principle). Therefore, standard errors and confidence intervals are regarded as not very meaningful for those biased estimates arising from (frequentist) penalized regression method, see e.g. the discussion Estimating R-squared and statistical significance from penalized regression model . I would assume that the same problems exists in an Bayesian approach but Kyung, Gill, Ghaosh and Casella (2010) say that the Bayesian formulation produces valid standard errors. Does it mean that a 95% credibility intervals includes with 95% probability the true biased estimate and if yes, is this a useful information?

Community
  • 1
Felix Bach
  • 21
  • 2
  • 2
    Well, the interpretation would rather be that a 95% credibility intervals includes with 95% probability the true parameter (given the model assuptions, of course...). I would say that the 95% credibility interval is extremely useful information, and often the reported end result of a Bayesian analysis (If you do not choose to look at the complete posterior distribution, that is). – Rasmus Bååth Oct 17 '13 at 09:07
  • Thanks a lot Rasmus, that's the answer I had hoped for... Do you know an article where it is explained? – Felix Bach Oct 23 '13 at 08:10
  • It is very well explained in John Kruschkes book, Doing Bayesian Data Analysis. http://www.indiana.edu/~kruschke/DoingBayesianDataAnalysis/ – Rasmus Bååth Oct 23 '13 at 12:41
  • Thanks for your reference, Rasmus but sorry but you may misunderstood my question. I wanted to know if the credibility interval around a biased estimate of a penalized regression can be interpreted as the credibility interval for the true estimate or for the biased estiamte. If it is a credibility interval for a biased estiamte is it really a useful information? – Felix Bach Nov 26 '13 at 13:25
  • It can be the case that a "biased" estimate often is a better estimate an "unbiased" one. Take, for example, the case where you flipped a coin three times and all came up heads. The unbiased estimate of the coin's relative frequency is 3/3 = 1.0, while a biased Bayesian estimate could be 0.8 which could be more reasonable (and closer to the truth) given the little data we have. – Rasmus Bååth Nov 26 '13 at 15:57
  • but would the true parameter be with 95% probability within the 95% credibility intervals? That's what I am struggling with – Felix Bach Nov 26 '13 at 16:21
  • 1
    Yes. But still, that is given your model and your assumptions. And I believe you will never get away from the "given your model and your assumptions" part... A nice article regarding the bias/variance tradeoff can be found here: http://demotrends.wordpress.com/2013/09/04/the-bias-variance-tradeoff-what-it-means-for-quantitative-researchers/ – Rasmus Bååth Nov 26 '13 at 20:01

0 Answers0