1

I have a dataset with 200,000 entries with four columns (time_of_day, order_size, time_taken, shop_number). I need to build a model and predict time_taken using the other three variables. There are more than 30,000 shop ids.

My approach has been to: 1. Try to use one_hot_encoding to encode each shop no. However, this leads to very large number of columns in my dataset. 2. Build a separate model for each shop assuming all shops have different efficiencies independent of one another.

Any other approach will be appreciated? Also, what type of model should I use in my data? I have tried simple linear regression and bayesian regression till now.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
John Samuel
  • 11
  • 1
  • 4
    On average you have less than seven observations per shop, which is nowhere near enough to begin modeling. Something has to give: you need to make assumptions about similarities among shops so that you can reduce the number of parameters to estimate. Would it be reasonable, for instance, to suppose that all or most shops exhibit the same kind of relationships between the independent and response variables? Do you have other information about these shops, such as their sizes, locations, ages, numbers of employees, etc.? – whuber Dec 15 '18 at 20:08
  • I only have shop ids which dont provide much information about them. I am not sure if I can assume a similar relation for each shop's independent variables and dependent variable because I think a shop's efficiency to process an order will vary. Moreover, I think the answer is expected on per shop basis because time_taken will be displayed for each shop. – John Samuel Dec 15 '18 at 20:10
  • 1
    That's good to know. But sometimes even ids can give surprising information. For instance, if they were assigned by the company sequentially over time (without recycling old ids), then the numerical value of an id can be a proxy for the age of the shop--and therefore can provide valuable input in a predictive model. Some companies (such as the US Postal Service) assign identifiers that have detailed geographic information, which can be used with similar good effect. – whuber Dec 15 '18 at 20:14
  • I think the ids were just created in database while onboarding a shop and has no relation with the actual age of the shop.(there might be some relation as you pointed out but since I have very less data per shop, it cannot be concluded.) Also, this is an online platform which lists these shops on its page and delivers the orders. So, they need an estimate of time_taken(time taken by the shop to prepare the order), to send delivery boys to the shops to optimize their delivery. Is a regression model suitable according to you? Or I am thinking very simply. – John Samuel Dec 15 '18 at 20:29
  • There are some extra columns with data like : time_at_which_ordered_placed and time_at_which_customer_receives_the order. which I dont think is useful in predicting the time_taken – John Samuel Dec 15 '18 at 20:33
  • Similar questions have been asked before. Search this site for the tag [many-categories] – kjetil b halvorsen Dec 17 '18 at 09:49

1 Answers1

2

This should be manageable, if a little slow, with a linear mixed model, e.g. in R with lme4:

lmer(time_taken ~ time_of_day*order_size + (time_of_day*order_size|shop_number),
     data=your_data)

A couple of advantages of LMMs:

  • in most LMM computational frameworks the random-effects model matrix $Z$ is automatically constructed using a sparse model matrix, so 30K shop IDs is not a big problem
  • the model is fitted with an empirical Bayes shrinkage estimator, so shops with less information are automatically adjusted closer to the population mean

If something like this works (you could start with a simpler model that uses (1|shop_number), i.e. only the intercept and not the effects of time of day and order size vary among shops) you might want to consider a more complex, additive model (e.g. the gamm4 package) that allows for more complicated patterns of variation with time of day and order size.

If you want to do this and R's lme4 is too slow you can try MixedModels.jl in Julia (or perhaps the commercial AS-REML, which is said to be very fast) (I'm not sure how SAS's speed compares).

Ben Bolker
  • 34,308
  • 2
  • 93
  • 126
  • 1
    It would be interesting to see how this approach compares with fused lasso ... – kjetil b halvorsen Dec 17 '18 at 13:44
  • I'm not super-familiar with fused lasso, but a quick read says it's for structured data, which this doesn't seem to be. Instead, this would be a candidate for a *grouped* lasso, where the shop values are grouped/shrunk toward a group mean, while the main effects are unshrunk or shrunk separately ... – Ben Bolker Dec 17 '18 at 14:04
  • :I don't think it is correct that fused lasso is for pre-grouped data. It is for data that one thinks ought to be grouped, but do not know the groups. So in a way, it is simultaneously finding clusters and shrinking to cluster means. – kjetil b halvorsen Dec 17 '18 at 14:05