4
  • Gaussian prior on weights of a GLM lead to Ridge / $\ell_2$ squared regularization.
  • Laplace prior leads to $\ell_1$ regularization

Question

What prior would lead to $\ell_\infty$ regularization ?

gunes
  • 49,700
  • 3
  • 39
  • 75
dohmatob
  • 468
  • 2
  • 13

1 Answers1

3

Thinking of similar examples, the reason behind the correspondence between these priors and norms is the exponent part. $$f(\beta)\propto \exp(-\lambda||\beta||_1)\rightarrow \ell_1$$ $$f(\beta)\propto \exp(-\lambda||\beta||^2_2)\rightarrow \ell_2 \text{ squared}$$ So, if we have a prior in the form below, it will correspond to $\ell_\infty$ because the negative log likelihood will directly include the term $\lambda||\beta||_{\infty}$ as a summand. $$f(\beta)\propto \exp(-\lambda||\beta||_\infty)\rightarrow \ell_\infty$$

I don't know if this PDF has a name, but it is a proper one ($n$ is the dimension): $$f(\beta)=\frac{\lambda^n}{2^nn!}\exp(-\lambda||\beta||_\infty)$$

gunes
  • 49,700
  • 3
  • 39
  • 75