2

I am looking for the correct equation for a ridge logistic regression for multiple variables. I thought it simply was:

$$y=\frac1{1+e^{-(\beta_0+\beta_1X_1+\beta_2X_2+\cdots+\beta_nX_n)}}$$

with an additional penalty parameter added, in this case

$$\sum_i^n (y_i-\widehat{y_i})^2 + \lambda \sum_j^p \beta_j^2$$

Eventually

$$y=\frac1{1+e^{-(\beta_0+\beta_1X_1+\beta_2X_2+\cdots+\beta_nX_n)}}+\sum_i^n (y_i-\widehat{y_i})^2 + \lambda \sum_j^p \beta_j^2$$

Is this true for a ridge regression?

StubbornAtom
  • 8,662
  • 1
  • 21
  • 67
Thomas
  • 332
  • 1
  • 13
  • 3
    Where does the sum of $(y_i-\hat y_i)^2$ come from? That has nothing to do with logistic regression or any regularization of it. Your initial formulation does not describe logistic regression, either: for it to be correct, you must equate the *expectation* of $y$ with the right hand side. Your final formulation makes no sense at all, because it completely changes the model. Please see https://stats.stackexchange.com/questions/29325. Perhaps https://stats.stackexchange.com/questions/228763 answers your question? – whuber Jan 31 '22 at 18:53
  • Hi @whuber. Would this be the correct equation for a multiple logistic regression then: ln[Y/(1−Y)]=a+b1X1+b2X2+b3X3...? – Thomas Feb 01 '22 at 08:07
  • That's stated explicitly in an answer in the first link I provided. – whuber Feb 01 '22 at 14:23
  • I am sorry, I only looked at the second link for my question. I thank you a lot! – Thomas Feb 01 '22 at 15:22

1 Answers1

5

The method of estimation does not affect the regression equation. Your logistic regression model is always the equation you posted in the first equation (save some technicalities).

If you estimate the parameters using the usual maximum likelihood estimation, you will get different estimated values of the parameters than if you use regularization, but those estimates are just guesses at the true values of the $\beta_i$.

As a heads up, the traditional loss function for logistic regression is log loss, not square loss.

$$ L(y,\hat y)=-\sum\bigg[ y_i\log(\hat y_i)+ (1-y_i)\log(1-\hat y_i)\bigg] $$

Dave
  • 28,473
  • 4
  • 52
  • 104
  • Thanks @Dave. Nice and clear. I used the ridge regression with an optimization of the maximum likelihood estimation. Does that influence the second equation (penalty parameter)? If so, what would the equation for maximum likelihood be? – Thomas Jan 31 '22 at 12:33
  • @Thomas The $L(y,\hat y)$ equation in my answer is the loss function corresponding to maximum likelihood estimation. I do not follow what you mean when you ask about the "second equation:. Could you please type out exactly what you think the correct regression equation is? – Dave Jan 31 '22 at 18:49
  • 3
    NB: The first equation is not a correct representation of logistic regression, and therein may lie the source of the question. – whuber Jan 31 '22 at 19:00
  • @Dave, sorry. I meant the loss function I added in my question, but you already answered my question though. Thanks. – Thomas Feb 01 '22 at 08:09