15

I have a dataset in which the event rate is very low ( 40,000 out of $12\cdot10^5$). I am applying logistic regression on this. I have had a discussion with someone where it came out that logistic regression would not give good confusion matrix on such low event rate data. But because of the business problem and the way it has been defined, I can't increase the number of events from 40,000 to any larger number though I agree that I can delete some nonevent population.

Please tell me your views on this, specifically:

  1. Does accuracy of logistic regression depend on event rate or is there any minimum event rate which is recommended ?
  2. Is there any special technique for low event rate data ?
  3. Would deleting my nonevent population would be good for the accuracy of my model ?

I am new to statistical modeling so forgive my ignorance and please address any associated issues that I could think about.

Thanks,

GaBorgulya
  • 3,253
  • 15
  • 19
ayush biyani
  • 1,397
  • 5
  • 17
  • 27

2 Answers2

11

I'm going to answer your questions out of order:

3 Would deleting my nonevent population would be good for the accuracy of my model ?

Each observation will provide some additional information about the parameter (through the likelihood function). Therefore there is no point in deleting data, as you would just be losing information.

1 Does accuracy of logistic regression depend on event rate or is there any minimum event rate which is recommended ?

Technically, yes: a rare observation is much more informative (that is, the likelihood function will be steeper). If your event ratio was 50:50, then you would get much tighter confidence bands (or credible intervals if you're being Bayesian) for the same amount of data. However you don't get to choose your event rate (unless you're doing a case-control study), so you'll have to make do with what you have.

2 Is there any special technique for low event rate data ?

The biggest problem that might arise is perfect separation: this happens when some combination of variables gives all non-events (or all events): in this case, the maximum likelihood parameter estimates (and their standard errors), will approach infinity (although usually the algorithm will stop beforehand). There are two possible solutions:

a) removing predictors from the model: though this will make your algorithm converge, you will be removing the variable with the most explanatory power, so this only makes sense if your model was overfitting to begin with (such as fitting too many complicated interactions).

b) use some sort of penalisation, such as a prior distribution, which will shrink the estimates back to more reasonable values.

Simon Byrne
  • 3,336
  • 15
  • 29
  • +1 I'd just also add that I've seen contexts where people have reweighted their data to 50:50. The tradeoff seems to be an improvement in the model's ability to classify (assuming a good threshold is chosen) versus some loss of information about overall prevalence and some additional difficulty in interpreting the coefficients. – David J. Harris Jun 21 '11 at 15:35
  • 1
    @David: I have also heard of people reweighting, and using complicated pseudo-bootstrap schemes where they only resample the high-frequency class. For all these techniques, you're ultimately throwing away (or making up) data. I'd argue that if this improves your model, then you're probably fitting the wrong model. See also my comments here: http://stats.stackexchange.com/questions/10356/is-a-logistic-regression-biased-when-the-outcome-variable-is-split-5-95/10397#10397 – Simon Byrne Jun 21 '11 at 16:15
  • 1) Sorry if I wasn't clear: I was talking about changing the relative influence of the events and nonevents, as with the "weights" argument in R's `glm` function. At worst, this is like throwing *part* of each downweighted data point away, I suppose, but it's not really the same thing. 2) As I said, there are tradeoffs associated with this decision. It probably makes the most sense in contexts where the population being sampled from isn't well-defined and the true event rate isn't meaningful to begin with. I certainly wouldn't recommend it across the board. – David J. Harris Jun 21 '11 at 17:18
2

There is a better alternative to deleting nonevents for temporal or spatial data: you can aggregate your data across time/space, and model the counts as Poisson. For example, if your event is "volcanic eruption happens on day X", then not many days will have a volcanic eruption. However, if you group together the days into weeks or months, e.g. "number of volcanic eruptions on month X", then you will have reduced the number of events, and more of the events will have nonzero values.

charles.y.zheng
  • 7,346
  • 2
  • 28
  • 32
  • 6
    I have to say this advice does not answer the question at all. 1) Their is nothing in the question that suggests the OP is dealing with spatial or temporal data. 2) How would aggregating the data help to identify any meaningful relationships (it uses less information than the original units!) – Andy W May 03 '11 at 00:21
  • 2
    Also as a note, for any observed relationship to occur at an aggregated level it has to be present in the level of the original units, although a relationship at the aggregated level does not necessarily reflect what the relationship between the two variables is at the disaggregated level. See http://qmrg.org.uk/files/2008/11/38-maup-openshaw.pdf – Andy W May 03 '11 at 00:22
  • agree with andy. – ayush biyani May 03 '11 at 05:55