1

Hi i am trying to use AIC value for comparing logit and probit model where in each model the data and the number of covariates are same (say, covariate= 3 for each model)

Does AIC value increases while increasing sample size?

Which one better to use as a tool for comparing logit and probit, AIC or MSPE?

Gala
  • 8,323
  • 2
  • 28
  • 42
Mazumder
  • 11
  • 2
  • Why are you comparing the models? AIC is a penalization of the log likelihood. I don't think it will be useful here. But what are you trying to do? – Peter Flom Jul 19 '13 at 10:35
  • Thank Peter, for your reply . should i use prediction error rate for comparing logit and probit models? – Mazumder Jul 19 '13 at 10:37
  • I am trying to check is there any priory that which models perform better logit or probit , for simulated data of different size – Mazumder Jul 19 '13 at 10:41
  • 1
    You might be interested in http://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models, especially the second half of @gung answer. In short, you might choose based on substantive theory/interpretation/tradition but you should not expect dramatic differences in model fit or an obvious way to decide between them purely empirically. – Gala Jul 19 '13 at 10:47
  • 1
    The number of parameters for each model is the same, so comparing AICs reduces to comparing log-likelihoods; a reasonable enough thing to do. But don't expect much difference. – Scortchi - Reinstate Monica Jul 19 '13 at 10:55

1 Answers1

3

What I would do here is compare the predictions to each other and to the correct values, graphically. That is, first run both models and output the predicted values from each. Then create a) A scatter plot of 1) probit predicted vs. logistic predicted 2) each model vs. actual values. b) A density plot (or maybe a box plot) of the errors of each model (to find any outliers).

Then make a decision.

However, in my experience, the two models often make similar predictions; however, different substantive fields have a tradition of using one or the other (e.g. psychology uses the logistic much more than the probit; I think the situation is the reverse in economics, but I am less versed in that literature).

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • 3
    One other way to consider is that the better of the two models should have the lower residual explained variation with respect to model complexities you hope you don't need. You might do a pooled many-degree-of-freedom likelihood ratio test of a set of nonlinear and interaction terms and see if one model needs these less than the other. I would choose the model needing them less. – Frank Harrell Jul 19 '13 at 11:21