1

I am taking a course about ridge, bridge and lasso regression. My teacher showed us that the solution to ridge regression can be written in the form

$$ \beta=(X^TX+\lambda I)^{-1}X^T Y $$

My question is do lasso and bridge regression have a similar solution?

user3741635
  • 111
  • 2
  • At least for LASSO this [has been answered already in the site](https://stats.stackexchange.com/questions/174003/why-is-my-derivation-of-a-closed-form-lasso-solution-incorrect/174008) (albeit it's written in the question). – Firebug Nov 01 '17 at 15:39
  • @Firebug, I have already checked that question before asking my question but I didn't understand what closed form lasso is and how it is related to lasso regression – user3741635 Nov 01 '17 at 15:53
  • It's written right at the beginning: $$\beta^{\text{lasso}}= \operatorname*{argmin}_\beta \| y-X\beta\|^2_2 + \alpha \| \beta\|_1$$ – Firebug Nov 01 '17 at 16:04
  • At the end of the question is a demonstration why LASSO regression has no closed form solution (in fact some exist for special cases, like orthogonal dependent variables). – Firebug Nov 01 '17 at 16:05
  • @Firebug That link doesn't demonstrate that lasso has no closed form solution: it just shows that the derivation attempted by OP is wrong. (I edited the title of the post last month since I saw someone else mistakenly reference it in the same way.) – user795305 Nov 01 '17 at 22:10
  • @user3741635 The ridge regression estimate is defined as $\beta_\mathrm{ridge} = \arg\min_\beta\frac{1}{2}\|y-X \beta\|_2^2+\lambda\|\beta\|_2^2$. Your teacher showed that it isn't necessary to perform a numerical minimization to compute $\beta_\mathrm{ridge}$ by finding a *closed form* $\beta_\mathrm{ridge} = \left(X^TX+\lambda I\right)^{-1}X^Ty$. Bridge and lasso are defined similarly to ridge, except with the "$2$"s in the last term replaced with $\gamma \in [1,2)$. However, for those problems, there does not exist a closed form, so numerical minimization must be performed. – user795305 Nov 01 '17 at 22:16
  • @Ben The question does in fact show LASSO has no closed form: > _"We can see now that our solution for one $\beta_j$ is dependent upon all the other $\beta_{i\neq j}$ so there is no closed form solution."_ – Firebug Nov 01 '17 at 22:20
  • @Firebug $x = 2y$ and $y = 2 + 3x$ has each term depending on the other terms. Getting stuck on one approach doesn't mean no closed form exists. – user795305 Nov 01 '17 at 22:22

0 Answers0