Bayesian interpretation of logistic ridge regression

Question

Most textbooks (also this blog) cover the fact that ridge regression,

$$ \hat y = \hat \beta X; \\ \hat \beta = \underset{\beta}{\text{argmin}}\ \ \frac{(y-\beta X)^T(y-\beta X)}{\sigma^2} + \lambda \|\beta\|_2^2; $$

can be interpreted as a MAP estimate of a Bayesian model with $N(0, \tau)$ priors on the $\beta$ parameters, where

$$ \lambda = \frac{\sigma^2}{\tau^2} $$ What is the equivalent Bayesian interpretation of logistic ridge regression?

$$ \hat y = logit^{-1}(\hat \beta X); \\ \hat \beta = \underset{\beta}{\text{argmin}}\ \ -y\ log(\hat y) - (1-y)\ log(1 - \hat y) + \lambda \|\beta\|_2^2; $$

I'm looking for this both out of theoretical interest, and because I would like to use stochastic gradient descent to estimate MAP logistic regression parameters in a context (JavaScript) where I don't have access to any specialised solvers.

Answer 1

As a preliminary note, I see that your equations seem to be dealing with the case where we only have a single explanatory variable and a single data point (and no intercept term). I will generalise this to look at the general case where you observe $n$ data points, so that the log-likelihood function is a sum over these $n$ observations. (I will use only one explanatory variable, as in your question.) For a logistic regression of this kind you have the observable values $Y_i|\mathbf{x}_i \sim \text{Bern}(\mu_i)$ with true mean values:

$$\mu_i \equiv \mathbb{E}(Y_i|\mathbf{x}_i) = \text{logistic}(\boldsymbol{\beta}^\text{T} \mathbf{x}) = \frac{e^{\boldsymbol{\beta}^\text{T} \mathbf{x}}}{1+e^{\boldsymbol{\beta}^\text{T} \mathbf{x}}}.$$

The log-likelihood function is given by:

$$\begin{align} \ell(\mathbf{y}|\mathbf{x},\boldsymbol{\beta}) &= \sum_{i=1}^n \log \text{Bern}(y_i|\mu_i) \\[6pt] &= \sum_{i=1}^n y_i \log (\mu_i) + \sum_{i=1}^n (1-y_i) \log (1-\mu_i) \\[6pt] &= \sum_{i=1}^n y_i \log (\mu_i) + \sum_{i=1}^n (1-y_i) \log (1-\mu_i) \\[6pt] &= \sum_{i=1}^n y_i \log(\boldsymbol{\beta}^\text{T} \mathbf{x}) - \sum_{i=1}^n y_i \log(1+e^{\boldsymbol{\beta}^\text{T} \mathbf{x}}) - (1-y_i) \log(1+e^{\boldsymbol{\beta}^\text{T} \mathbf{x}}) \\[6pt] &= \sum_{i=1}^n y_i \log(\boldsymbol{\beta}^\text{T} \mathbf{x}) - \sum_{i=1}^n \log(1+e^{\boldsymbol{\beta}^\text{T} \mathbf{x}}). \\[6pt] \end{align}$$

Logistic ridge regression operates by using an estimation method that imposes a penalty on the parameter $\boldsymbol{\beta}$ that is proportionate to its squared norm. (Note that you have stated this slightly incorrectly in your question.) It estimates the parameter $\boldsymbol{\beta} = (\beta_1,...,\beta_K)$ via the optimisation problem:

$$\begin{align} \hat{\boldsymbol{\beta}}_\text{Ridge} &= \underset{\boldsymbol{\beta} \in \mathbb{R}^K}{\text{argmax}} \ \ \ \ \ell(\mathbf{y}|\mathbf{x},\boldsymbol{\beta}) - \lambda ||\boldsymbol{\beta}||^2. \\[6pt] \end{align}$$

Since the log-posterior is the sum of the log-likelihood and log-prior, the MAP estimator is:

$$\begin{align} \hat{\boldsymbol{\beta}}_\text{MAP} &= \underset{\boldsymbol{\beta} \in \mathbb{R}^K}{\text{argmax}} \ \ \ \ \ell(\mathbf{y}|\mathbf{x},\boldsymbol{\beta}) + \log \pi(\boldsymbol{\beta}). \\[6pt] \end{align}$$

We obtain the result $\hat{\boldsymbol{\beta}}_\text{Ridge} = \hat{\boldsymbol{\beta}}_\text{MAP}$ by using the prior kernel $\pi(\boldsymbol{\beta}) \propto \exp(- \lambda ||\boldsymbol{\beta}||^2)$ so that $\log \pi (\boldsymbol{\beta}) = - \lambda ||\boldsymbol{\beta}||^2 + \text{const}$ in the above equation. Integrating to find the constant of integration gives the prior distribution:

$$\pi(\boldsymbol{\beta}) = \prod_k \mathcal{N} \bigg( \beta_k \bigg| 0, \frac{1}{2\lambda} \bigg).$$

Thus, we see that ridge logistic regression is equivalent to MAP estimation if a priori the individual $\beta_k$ parameters are IID normal random variables with zero mean. The variance parameter for this normal distribution is a one-to-one mapping of the "penalty" hyperparameter in the ridge logistic regression --- a larger penalty in the ridge regression corresponds to a smaller variance for the prior.

(Note: For a related question showing LASSO and ridge regression framed in Bayesian terms see here.)

Answer 2

To look for equivalence one should compare the form of,

$$\hat{\beta} = \underset{\beta}{\text{argmin}} -y\log(\hat{y}) - (1-y)\log(1-\hat{y}) + \lambda||\beta||_2^2,$$

with the posterior distribution whilst keeping a general expression for the prior. The posterior distribution has form, $$\pi(\beta|x) \propto \pi(\beta)L(\beta;x).$$ Where $\pi(\beta)$ is the prior and $L(\beta;x)$ is the likelihood. Noting that $\beta$ is $p\times1$ and that $x$ represents the data where $x_i$ is one observation and would be $p\times1$. In logistic regression the model for the data is Bernoulli (more generally Binomial). So, $$y_i|\beta,x_i \sim Bernoulli(p_i)$$ where $p_i = \frac{\exp\{\beta^Tx_i\}}{1 + \exp\{\beta^Tx_i\}}.$ Let $f(\cdot)$ be the density function, then the posterior for $\beta$ becomes

\begin{align*} \pi(\beta|x)&\propto\pi(\beta)\prod_{i=1}^{n}f(x_i|\beta) \\ &= \pi(\beta)\prod_{i=1}^{n}p_i^{y_i}(1-p_i)^{1-y_i}. \end{align*} The maximum-a-posterior (MAP) of $\beta$ is the mode of its posterior distribution and since $\log$ is monotone, $$\hat{\beta}_{MAP} = \underset{\beta}{\text{argmax}}\pi(\beta|x) = \underset{\beta}{\text{argmax}}\log\pi(\beta|x).$$ So taking, $$\log\pi(\beta|x) \propto \log\pi(\beta) + \sum_{i=1}^n\big\{y_i\log p_i + (1-y_i)\log(1-p_i)\big\}$$ and noting that $\hat{\beta}_{MAP} = \underset{\beta}{\text{argmax}}\log\pi(\beta|x) = \underset{\beta}{\text{argmin}}\big\{-\log\pi(\beta|x)\big\}$ we can see that, \begin{align*} \log\pi(\beta) &\propto - \lambda||\beta||_2^2 \\ \Rightarrow \pi(\beta) &\propto \exp\{-\lambda||\beta||_2^2\}. \end{align*} This can be seen as taking independent normal priors with mean zero and variance $\frac{1}{2\lambda}$, $$\beta_j \sim N\left(0,\frac{1}{2\lambda}\right) \ \ j=1,\dots,p.$$