1

I have fit a logistic regression model with four features and now I am able to use it for prediction.

When I plot the data points from my learning sample (i.e. I select two features, plot them in two axis), I see that there is more density of points in an area than in other areas. What I would like my model to return is not only the probabilities, but also a number that tells me how good these probabilities are (e.g. a p-value). I imagine that depending on the region the prediction is made, and the density of the sample points in that concrete region, the better the prediction, and conversely.

Is there a way of obtaining a value that tells me how good my prediction is?

In concrete, I am using python scikit logistic regression package, although the answer can be more broad.

Escachator
  • 240
  • 2
  • 13
  • 2
    The Brier score is appropriate for evaluating predictive accuracy... https://en.wikipedia.org/wiki/Brier_score – Mike Hunter Nov 01 '15 at 19:28
  • 1
    It's not clear to me what's actually asked for. If you want a uncertainty measure for individual predictions, then standard errors or confidence intervals for the predicted probabilities (mean prediction) would provide that information. That's different from an overall goodness of fit measure, and Logit is fully parametric so it doesn't give local fit information. – Josef Nov 02 '15 at 14:34
  • @user333700 thanks for your comment. I want uncertainty measure for individual prediction on this question. I have asked for goodness of fit in another question. – Escachator Nov 04 '15 at 11:46
  • @DJohnson that won't help, because what I need is the potential error when I do the prediction... – Escachator Nov 09 '15 at 18:16

3 Answers3

4

The key thing to check first is the model's calibration, either using the bootstrap to correct for overfitting or using a huge independent sample not used for model development or fitting. The best way to assess calibration is using a loess smooth nonparametric regression. Once you establish calibration you can go on to predictive discrimination using the pseudo $R^2$ and Somers' $D_{xy}$ rank correlation coefficient, or a simple translation of it to the $c$-index AKA concordance probability or AUROC. The Brier score is an excellent addition to all this.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
  • Hi @Frank. Thanks for your answer! Re calibration, I have already worked with the parameters, with an out of sample and test sets, if that is what you mean. The results are quite bad for any set of parameters, that is why I want to check how relevant the featurea are. What do you mean by a less smooth nonparametric regression? – Escachator Nov 02 '15 at 00:03
  • 1
    That is _loess_, which is a flexible way to estimate a smooth relationship (here between continuous predicted probabilities $\hat{P}$ and observed binary $Y$). – Frank Harrell Nov 02 '15 at 12:10
  • I think your method is more for goodness of fit than to know if a probability returned by my fitted model is good. I guess what I need is the confidence interval of that obtained probability, isn't it? Thanks! – Escachator Nov 09 '15 at 18:50
  • The calibration curve is the way to check calibration-in-the-small, i.e., is a high-resolution estimate of the absolute accuracy of a set of probability forecasts. In that sense the calibration curve is both a measure of goodness-of-fit and a great way to check the accuracy of probabilities estimated by the model. – Frank Harrell Nov 09 '15 at 19:08
1

I may be wrong with exactly what it is that you are looking for, but if you are worried about the overall distribution of your prediction quality, then I would borrow from machine learning validation tools since this is exactly what they are interested in as well.

Here, you could do a 10-fold cross-validation (or a single hold out sample but you couldn't get information about the distribution of your metric over multiple samples of your data) of an area under the ROC curve metric (AUROC), for instance, if you would like to see how your predictions behave with different class cut-off probabilities.

Another metric could be the mean squared error if you prefer that loss function.

Noah Hammarlund
  • 143
  • 9
-1

You can return r squared, p value and chi squared for a logistic regression. Try using stats models,instead of Sci kit. Stats models returns all of this data automatically

Rupert
  • 125
  • 2