35

I was watching a presentation by an ML specialist from a major retailer, where they had developed a model to predict out of stock events.

Let's assume for a moment that over time, their model becomes very accurate, wouldn't that somehow be "self-defeating"? That is, if the model truly works well, then they will be able to anticipate out of stock events and avoid them, eventually getting to a point where they have little or no out of stock events at all. But then if that is the case, there won't be enough historical data to run their model on, or their model gets derailed, because the same causal factors that used to indicate a stock out event no longer do so.

What are the strategies for dealing with such a scenario?

Additionally, one could envision the opposite situation: For example a recommender system might become a "self-fulfilling prophecy" with an increase in sales of item pairs driven by the output of the recommender system, even if the two items aren't really that related.

It seems to me that both are results of a sort of feedback loop that occurs between the output of the predictor and the actions that are taken based on it. How can one deal with situations like this?

Skander H.
  • 10,602
  • 2
  • 33
  • 81
  • 7
    (+1) In some analogous situations involving higher education, people talk about a model "cannabalizing itself." College officials, using models, award financial aid to achieve certain enrollment- and financial-aid-related goals, only to find that, as a result, eventually prospective students' enrollment decisions are less and less determined by or predictable from the financial aid award. – rolando2 Oct 22 '18 at 20:02
  • 1
    This question is hard to answer in general, as it depends quite a bit on the situation. In case of stockouts there are good solutions, but in case of recommenders there may simply not be a big problem if your model becomes a bit prescriptive. – Dennis Jaheruddin Oct 23 '18 at 10:46

5 Answers5

27

If you are using a model to support decisions about intervening in a system, then logically, the model should seek to predict the outcome conditioned on a given intervention. Then separately, you should optimize to choose the intervention with the best expected outcome. You are not trying to predict your own intervention.

In this case, the model could predict demand (the variable you don't directly control) and this, in combination with the stocking choice, would result in having an out-of-stock event or not. The model should continue to be "rewarded" for predicting demand correctly since this is its job. Out-of-stock events will depend on this variable along with your stocking choice.

nanoman
  • 481
  • 3
  • 6
  • 2
    This is exactly how it is solved in practice. A black box model that would not provide understanding of the driving factors but just predict that a stockout would happen on wednesday would have very limited use if people cannot understand the assumptions. (With a key assumption being that the model is not in place). – Dennis Jaheruddin Oct 23 '18 at 10:43
  • @DennisJaheruddin: On the other hand, make a model that can predict when the product will be out of stock despite a timely re-order and you can make a killing. – Joshua Oct 23 '18 at 15:16
  • 1
    It is a little bit more complex than that, because in the model they used, demand signal was only one factor. But they also used other factors, based on in store conditions, to account for situations where the stock was in the store but not on the shelf (For example in the backroom, or at one of the cash registers or service desks because the customer changed their mind at the last minute). For that, they were not using just demand but other causal drivers as well. – Skander H. Oct 23 '18 at 17:46
  • @Alex that complexity just amounts to a network of simple supply/demand mechanisms governing different locations, whether or not the model represents them explicitly. The model's objective is more accurately described as predicting stock levels, not demand, but that only becomes relevant if you're specifically considering there to be uncontrolled factors in both the supply and demand at the point in the network (the shelf) where the stock levels matter. Interventions like ordering more stock or having staff replenishing shelves more regularly still need to be factors in the model. – Will Oct 24 '18 at 11:05
  • You might want to mention the possibility that demand varies with stock. – Yakk Oct 24 '18 at 20:09
15

There are two possibilities by which an out-of-stock (OOS) detection model might self-derail:

  1. The relationship between inputs and OOS might change over time. For instance, promotions might lead to higher OOS (promotional sales are harder to predict than regular sales, in part because not only average sales increase, but also the variance of sales, and "harder-to-predict" translates often into OOS), but the system and its users might learn this and lay in additional stock for promotions. After a while, the original relationship between promotions and OOS does not hold any more.

    This is often called "model shift" or similar. You can overcome it by adapting your model. The most common way is to weight inputs differently, giving lower weight to older observations.

  2. Even if the relationship between a predictor and OOS does not change, the predictor's distribution might. For instance, multiple days with zero sales of a particular stock keeping unit (SKU) might signal an OOS - but if the model performs well, then OOS might be reduced across the board, and there might simply not be as many sequences of zero sales.

    Changes in the distribution of a predictor should not be a problem. Your model will simply output a lower probability of OOS.

In the end, you probably don't need to worry overmuch. There will never be zero OOS. Feedback mechanisms like the ones above do occur, but they will not work until OOS are completely eradicated.

  • Some pending OOS may simply not be avertable. "I have one unit on the shelf and will probably face a demand for five over the coming week, but the next delivery is only due a week from today."
  • Some OOS will be very hard to predict, even if they are avertable, if they had been known in time. "If we had known we would drop the pallet off the forklift and destroy all the product, we would have ordered another one."
  • Retailers do understand that they need to aim for a high service level, but that 100% are not achievable. People do come in and buy up your entire stock on certain products. This is hard to forecast (see above) and sufficiently rare that you do not want to fill up your shelves on the off chance this might happen. Compare Pareto's law: a service level of 80% (or even 90%) is pretty easy to achieve, but 99.9% is much harder. Some OOS are consciously allowed.
  • Something similar to Moore's law holds: the better ML becomes, the more expectations will increase, and the harder people will make life for the model. While OOS detection (and forecasting) algorithms improve, retailers are busy making our life more difficult.
    • For instance through variant proliferation. It's easier to detect OOS on four flavors of yoghurt than on twenty different flavors. Why? Because people don't eat five times as much yoghurt. Instead, pretty much unchanged total demand is now distributed across five times as many SKUs, and each SKU's stock is one fifth as high as before. The Long Tail is expanding, and signals are getting weaker.
    • Or by allowing mobile checkout using your own device. This may well lower psychological barriers to shoplifting, so system inventories will be even worse than they already are, and of course, system inventories are probably the best predictor for OOS, so if they are off, the model will deteriorate.

I happen to have been working in forecasting retail sales for over twelve years now, so I do have a bit of an idea about developments like this.

I may be pessimistic, but I think very similar effects are at work for other ML use cases than OOS detection. Or maybe this is not pessimism: it means that problems will likely never be "solved", so there will still be work for us even decades from now.

Community
  • 1
Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • I especially agree with your last comment. The worst-case version of this scenario seems like the starting point for a full employment / no free lunch theorem. Which is what makes this an interesting question IMO! – senderle Oct 23 '18 at 20:56
9

Presumably you can track when restock events happen. Then it's just a matter of arithmetic to work out when the stock would be depleted had the model not been used to restock inventory.

This assumes that any positive stock level is independent of the level of sales. A commenter says that this assumption doesn't hold in reality. I don't know either way -- I don't work on retail data sets. But as a simplification, my proposed approach permits one to make inferences using counterfactual reasoning; whether or not this simplification is too unrealistic to give meaningful insight is up to you.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
  • I guess I don't see how this answers the question. The problems as I understand them are: (a) After implementing the model in production, the optimal prediction function for stockouts is now completely different than it was before, because we changed the data distribution; (b) The better our model is, the more rare stockout events will become, and therefore the more difficult it will become to accurately predict them going forward. Knowing "when the stock would be depleted had the model not been used to restock inventory" is neither here nor there because the model is in production from now on – Jake Westfall Oct 23 '18 at 00:15
  • @JakeWestfall This type of analysis is called counterfactual reasoning. If you know the inventory at all times, and you know when it's restocked, then you can create a counterfactual that supposes restocking didn't occur: just subtract the restock from the inventory after the restock occurred. Now you have a time series that reflects the supposition you never restocked. Carry this time-series forward until stockout. Now you know when a stockout would have occurred without restocking. How does this counterfactual have a different data distribution? – Sycorax Oct 23 '18 at 00:35
  • I understand all that. What I don't understand is how this solves the issues raised in the OP. For example, suppose a strong predictor of stockout is whether it's the first day of a month (when many people get paid). Using our new model, we can now avoid these stockout events by preemptively ordering more units near the end of each month. So now "first day of the month" will no longer be a useful predictor of stockouts going forward. We could indeed compute the counterfactual probability of a stockout on the first of the month had we not preemptively ordered, but how exactly does this help us? – Jake Westfall Oct 23 '18 at 00:54
  • 2
    This helps us because it gives you a counterfactual probability of a stockout had the restock not occurred. OP is asking "how to deal with" the fact that a model that reduces occurrences of stock outs will not have as many occurrences of stockouts available in the raw data. My point is that you can make inferences about the counterfactual occurrence of stockouts, and use that as a surrogate. What kind of help did you want? – Sycorax Oct 23 '18 at 01:11
  • Wait-- "use those as a surrogate"? Are you suggesting we'd _fit the model to the simulated counterfactual_?? As for the kind of help the OP wants: in the stockout example this is rather a "first world problem" and a response of "you should be so lucky to have this problem" may be good enough. But the recommender system example in the OP seems more serious. Or consider a model to find the price that maximizes profit. We estimate the best price and then "set it and forget it." But now we can't detect if changing market conditions alter the optimal price, because we're no longer varying the price – Jake Westfall Oct 23 '18 at 01:44
  • These all have in common that we used the model predictions to make changes to the data generating process, thus changing the distribution of what's being modeled. I interpret this to be the core issue the OP is asking about. The specific examples highlight different potential problems that may arise from this, and it's unlikely that they can all have the same solution. But it's not clear (at least to me) how _any_ of these kind of problems are addressed by the sort of counterfactual calculation you described in your answer. – Jake Westfall Oct 23 '18 at 01:49
  • I take the question to be "how can you make predictions about the new distribution," not "how can you emulate the old distribution." I suppose in some cases, continuing to train the model on the old distribution will help with the new distribution. Are you saying that it's *guaranteed* to? – senderle Oct 23 '18 at 02:16
  • @senderle I'm just saying that stock outs can be easily modeled using counterfactuals. Not all problems are like stock outs, but this does address a question that OP asked. – Sycorax Oct 23 '18 at 02:24
  • 1
    @Sycorax: You assume that purchaser behavior is not influenced by the number of items visible on the shelf. That's an invalid assumption. The influence may be weak, but it's not absent. – Ben Voigt Oct 24 '18 at 04:20
  • @BenVoigt Ah! Thank you! At last, someone was able to **explain** why this model might not be realistic. I didn't know that people were influenced by stock levels on shelves. I suppose in an e-commerce setting where stock levels are not as visible, this effect would be smaller than a physical commerce setting. – Sycorax Oct 24 '18 at 12:47
  • @Sycorax: E-commerce settings sometimes point out stock levels. I know when searching for plane tickets, quite a few sites will include a statement along the lines of "One 2 seats left at this price!" Hard to confirm that's true, but there's reasoning behind that statement. A very obvious case where stock levels affect a purchase decision is when there's a buy-1-get-1-free offer, but only one item remaining. – Ben Voigt Oct 24 '18 at 13:38
  • @BenVoigt Selling out seats on aircraft seems like a fundamentally different problem, since the airplane can't be "restocked" with additional seating. – Sycorax Oct 24 '18 at 14:03
8

Your scenario bears a lot of resemblance to the Lucas Critique in economics. In machine learning, this is called "dataset shift".

You can overcome it, as @Sycorax says, by explicitly modeling it.

generic_user
  • 11,981
  • 8
  • 40
  • 63
2

One thing to remember is that ML is an instrumental goal. Ultimately, we don't want to predict out of stock events, we want to prevent out of stock events. Predicting out of stock events is simply a means to that end. So as far as Type II errors are concerned, this isn't an issue. Either we continue to have OOSE, in which case we have data to train our model, or we don't, in which the problem that the model was created to address has been solved. What can be a problem is Type I errors. It's easy to fall into a Bear Patrol fallacy, where you have a system X that is built to prevent Y, you don't see Y, so you conclude that X prevents Y, and any attempts to shut X down are dismissed on the basis "But it's doing such a good job preventing Y!" Organizations can be locked into expensive programs because no one wants to risk that Y will come back, and it's difficult to find out whether X is really necessary without allowing that possibility.

It then becomes a trade-off of how much you're willing to occasionally engage in (according to your model) suboptimal behavior to get a control group. That's part of any active exploration: if you have a drug that you think is effective, you have to have a control group that isn't getting the drug to confirm that it is in fact effective.

Acccumulation
  • 3,688
  • 5
  • 11