3

I am modeling ridership data for specific routes by month over a number of years. Some routes have as little as about 1000 riders per month while other routes may have over 20,000 riders per month. I have been looking at different approaches to model this data including a panel data model and a generalized linear data model (poisson family). However, I have found some information that says you should only use the poisson family when you have a small range in data for the y variable.

Is there a better approach to modeling count data with a large range of y values than Poisson?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
CooperBuckeye05
  • 423
  • 1
  • 6
  • 16

2 Answers2

3

For count data it is indicated (for reasons of interpretability of estimated parameters) to use a generalized linear model (GLM) with logarithmic link function, see my answer to Goodness of fit and which model to choose linear regression or Poisson

But the distribution family can be choosen in different ways. The reason for the advice you refer, to use Poisson regression when the counts are not to large, is that usually with large counts there is overdispersion, that is, the variance is larger than the variance of the Poisson.

That can be solved in various ways, like using (in R terminology) a quasipoisson family, which can be good enough. Or you can use a negative binomial family. Or, especially if you only wants predictions from the model and not need interpretable parameters, use the traditional way of a usual linear model (that is, identity link function) for the response $\sqrt{Y}$. The square-root transformation is (approximately) variance stabilizing for the Poisson (and quasi-Poisson) families. Look at Why is the square root transformation recommended for count data? for an explanation of this!

For more about overdispersion, see Modelling a Poisson distribution with overdispersion and Comparing overdispersion distributions

Community
  • 1
kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • Thanks for you input. Just to clarify, When you mean using the square root, do you mean transforming the y by taking it sqrt and then running the model correct? Running the model with an in transformed y did show extreme over dispersion, but running it with a neg binomial got it just under 1 but real close. – CooperBuckeye05 Aug 15 '15 at 01:39
  • yes, I mean using as the response in a usual (least squares) linear model $\sqrt{Y}$ – kjetil b halvorsen Aug 15 '15 at 12:55
1

I am a little bit confused by the question (and the answer) as Box and Jenkins introduced ARIMA modelling by analyzing count data i.e. the number of people that flew in a particular month with their airline series. If the time series has small count numbers e.g. 0.0,1,1,0,1,0,2,1,0,0,1,2,1,2,0,1,..... there are definite problems/limitations applying ARIMA modelling procedures. I have been reasonably happy with count data that can average 5 or more as the assumption of continuity is less strained. Truncating/rounding the forecast is a way of providing an answer that must be an integer as integers only exist in the history.

IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • The airline data did'nt have covariables (?), yes? So the interpretation problems do not arise. Still, a square root transformation to stabilize variance could be useful, maybe. But, of course Box knew very well about variance stabilizing transformations, if he didnt use one he must have had reasons. – kjetil b halvorsen Aug 15 '15 at 14:20
  • 1
    See my most favored/loved response as to when and why http://stats.stackexchange.com/questions/18844/when-and-why-to-take-the-log-of-a-distribution-of-numbers/18852#18852 . He inadvertently thought that a log transformation was appropriate/needed where http://www.autobox.com/cms/index.php/afs-university/intro-to-forecasting/doc_download/53-capabilities-presentation suggests that a few outliers were simkply needed . Th issue of co-variables is not relevant in my oipnion. – IrishStat Aug 15 '15 at 14:54
  • My idea with mentioning covariables is that without covariables, the parmeters just represent means, or differences of means. Hardly any interpretation problems. With covariables the interpretation problem with transformation grows. – kjetil b halvorsen Aug 18 '15 at 08:40
  • 1
    You are correct in that without covariables we are just using the history of the observed series which may entail means/differences/lagged values/level shifts/time trends/seasonal pulses etc. . With covariables we can possibly incorporate supporting/helping/user-specified predictors. In both cases variance stabilizing transformations be they power transforms/weighted least squares/garch transforms may be needed to generate model errors that have constant variance and a zero constant mean. – IrishStat Aug 18 '15 at 10:10
  • Yes. My point is/was is that with covariables, we might be better off transforming fitted values than observations. In practice that is done via a GLM/link function. – kjetil b halvorsen Aug 18 '15 at 12:11
  • The whole idea is to select the transform based upon the errors NOT the fitted values ALTHOUGH by chance (or arithmetic !) you might end up with the same answer. The reason for this is that you wish to generate a noise process that is a central ch-square distributed variable so that F tests are meaningful/valid/correct/useful in saluting/criticizing a tentative model . This has nothing whatsoever to do with whether or not a co-variable is in the model – IrishStat Aug 18 '15 at 12:18
  • I think we are misunderstanding each other. My concerns are different than your (not saying that yours are not valid, they are, but I was thinking about something else). My concern was about interpretability of parameter estimates, see my answer here: http://stats.stackexchange.com/questions/142338/goodness-of-fit-and-which-model-to-choose-linear-regression-or-poisson/142353#142353 for an elucidation of that point. – kjetil b halvorsen Aug 18 '15 at 12:24
  • If interpretability of the parameter estimates is the objective then what you are proposing could be useful but I genuflect to the statistical requirements not the "window dressing". But I do see your point that one often has to sell the solution in terms of the client's needs to "understand' . Perhaps that is why I have had some non-successful encounters with the real world. As Martin L once said "Here I stand I can do no other" – IrishStat Aug 18 '15 at 12:35