3

Is there a way to get a prediction from an gam model (from package mgcv) that contains random effect smooths, where the new data contains a level of the random effect that didn't exist in the training set? Something comparable to the "re.form=NA" parameter in lme4:predict.merMod ?

library(hflights) # get a big data set
hflights$UniqueCarrier <- factor(hflights$UniqueCarrier) # turn carrier into a factor
newhflights <- head(hflights) # create "new" d.f. for prediction
newhflights$UniqueCarrier <- "ZZ" # change carrier to a previously unseen value 


library(mgcv)
m_gam <- gam(ActualElapsedTime ~ s(Distance) + s(TaxiOut, UniqueCarrier, bs="re"), data=hflights)
predict(m_gam, newdata=newhflights) # fails with "ZZ not in original fit"

library(lme4)
m_lmer <- lmer(ActualElapsedTime ~ Distance + (TaxiOut | UniqueCarrier), data=hflights)
predict(m_lmer, newdata=newhflights) # fails with "new levels detected in newdata"
predict(m_lmer, newdata=newhflights, re.form=NA) # works

I have searched for an answer and found this question, but it didn't appear to be directly related to my problem. Predicting with random effects in mgcv gam But I did try adding a dummy variable to the model based on it.

hflights$dummy <- 1
newhflights$dummy <- 0
m_gam <- gam(ActualElapsedTime ~ s(Distance) + s(TaxiOut, UniqueCarrier, bs="re", by=dummy), data=hflights)
predict(m_gam, newdata=newhflights) # still doesn't work, not in original fit

I'm out of ideas. Am I overlooking something obvious?

Community
  • 1
colonel.triq
  • 659
  • 1
  • 5
  • 16
  • 1
    I suppose refitting the model with `ZZ` added as a level to the original data is out of the question? (you'd need to refit with `drop.unused.levels = FALSE`). Alternatively, just set that new level to be one of the existing levels; as long as you've turned off the `re` term it will make no difference to the predictions (assuming the models are as per the ones shown). – Gavin Simpson Jun 12 '15 at 20:16
  • The new previously unseen levels are not necessarily known until prediction time, and refitting the model is not a desirable option. Overwriting the random effect level with a known one (then changing it back post-prediction) seems to work, and I guess that's the only real option. It seemed awkward to do it that way, so I wondered if there was a more elegant solution. Thanks! – colonel.triq Jun 15 '15 at 21:42
  • No, even Simon Wood freely admits that handling of random effects is simple at best in `mgcv::gam`. Perhaps `mgcv::gamm` or `gamm4::gamm` might work better for you seeing as the spline fixed-effects part of your model appears relatively simple. – Gavin Simpson Jun 16 '15 at 02:17

0 Answers0