Bayesian Elastic Lasso

Question

While studying elastic lasso, I have had a thought if I can apply a Bayesian method to the Elastic Lasso. If I want apply Bsyesian way of making a Regression model with Elastic Lasso, what do I need to think of?

Answer 1

In Bayesian models regularization is achieved by the choice of priors. For elastic lasso regression you need Bayesian linear regression with appropriate priors. To achieve $\ell_1$ regularization, you would use the Laplace prior, for $\ell_2$ regularization, the Gaussian prior. Elastic net means combining both regularization terms. As noticed by Lin and Lin (2010), this translates to the following prior

$$ \pi(\boldsymbol\beta) \propto \exp\left\{ -\lambda_1 \| \boldsymbol\beta \|_1 - \lambda_2 \| \boldsymbol\beta \|_2^2 \right\} $$

You don't need to normalize it for optimization, or MCMC sampling, so the only thing you need to do is to use the custom prior with the software you're already using.

However keep in mind that while those priors should technically lead to same results as in non-Bayesian flavor of the model, in practice there may be better choices for priors to achieve sparsity.