How to choose the link function when performing a logistic regression?

Question

I am running a logistic model. In SAS Entreprise Miner, I noticed there's a link function that has three possible options: logit, probit and cll (complementary log-log).

Can you please shed light on the following questions:

1. Can we use any of these link function to carry out a logistic regression?
2. Are there situations where one would be better than others?
3. Is it intuitively possible to get some insight about which kind of function could be useful in which situation? (By just looking at the formula, complementary log-log function might be good for normalization of data when data does not depart (too much) from a normal distribution.)

Any additional pointers would be greatly appreciated.

Answer 1

I don't know of SAS, so i'll just answer based on the statistics side of the question. About the software you mays ask at the sister site, stackoverflow.

1. If the link function is different (logistic, probit or Clog-log), than you will get different results. For logistic, use logistic.

2. About the real differences of these link functions.

Logistic and probit are pretty much the same. To see why they are pretty much the same, remember that in linear regression the link function is the identity. In logistic regression, the link function is the logistic and in the probit, the normal. Formally, you can see this by noting that, in case your dependent variable is binary, you can think of it as following a Bernoulli distribution with a given probability of success. $Y \sim Bernoulli(p_{i})$

$p_{i} = f(\mu)$

$\mu = XB$

Here, thew probabitliy $p_{i}$ is a function of predictor, just like in linear regression. The real difference is the link function. In linear regression, the link function is just the identity, i.e., $f(\mu) = \mu$, so you can just plug-in the linear predictors.In the logistic regression, the link function is the cumulative logistic distribution, given by $1/(1+exp(-x)). In the probit regression, the link function is the (inverse) cumulative Normal distribution function. And in the Clog-log regression, the link function is the complementary log log distribution.

I never used the Cloglog, so i'll abstein of coments about it here.

You can see that Normal and Logist are very similar in this blog post by John Cook, of Endeavour http://www.johndcook.com/blog/2010/05/18/normal-approximation-to-logistic/.

In general I use the logistic because the coefficients are easier to interpret than in a probit regression. In some specific context I use probit (ideal point estimation or when I have to code my own Gibbs Sampler), but I guess they are not relevant to you. So, my advice is, whenever in doubt about probit or logistic, use logistic!

Answer 2

I have a question/comment. I thought that by definition, logistic regression uses the logit link. If you are using the probit or complementary log-log link, then I do not think that is logistic regression.

What you are doing is fitting generalized linear models on a binary outcome, which is assumed to follow a Bernoulli. The 3 usual choices of link functions are the logit, probit, and complementary log-log. If you are using the logit link, that is logistic regression.

Answer 3

All 3 link functions are s-shaped and are not going to be that different. Li and Duan showed that if the predictor variables are well behaved (elliptically symmetric predictors are a subset of the well behaved group) then changing the link function will change the coefficients by a multiplicitive constant. Even if the predictors are not perfectly well behaved the differences between similar link functions are unlikely to change the overall inference (the exact coefficients will change, but what is important or significant will still be under a different link function).

The logit allows you to interpret individual coefficients as log-odds, so it tends to be the most popular these days.

Answer 4

This is an excellent question that sits at the nexus of mathematics and science. As someone who teaches a linear models course that touches on "logistic regression" and its several possible link functions, I feel compelled to answer.

First, I believe that SAS is fitting a generalized linear model (GLM) and estimating the parameters using MLE (or qMLE) in its "logistic" function. As such, any appropriate link function that transforms (0, 1) into (-\inf, \inf) is appropriate. Of that infinite class of functions, the logit, the probit, and the complementary log-log are members... so are all quantile functions.

Second, there is little appreciable difference between the logit and the probit link functions. While the coefficient estimates will tend to differ by a factor of about 3.8, the predictions will be very similar.

Third, the logit and probit functions are symmetric about (0, 0.5), while the complementary log-log function is not symmetric. This constitutes the primary difference between the logit/probit functions and the complementary log-log function.

Recall that the dependent variable is the probability of a success and the independent variable is the linear predictor. For the logit/probit links, the function value approaches 0 at the same rate as it does 1. For the complementary log-log function, however, that is not true. The cloglog function approaches 1 more sharply than it approaches 0. [Side note: the log-log function is the complement of the cloglog. It approaches 0 more sharply than 1.]

Fourth... I'm not sure what that actually means in terms of your last question. My experience is that the science has not advanced enough to suggest a "correct" link function. As a result, I instruct my students to fit their model using several link functions. If the coefficient results differ by "a lot," then there is something wrong with their model. Otherwise, the model is robust to the selection of the link function.

While this is an answer to ayush biyani, I think #4 could drive an interesting discussion about link functions.