I'm looking for help understanding how to re-train a fraud detection classifier that's been deployed to production (where it successfully blocked much, but not all fraud coming into the system). I haven't seen this subtlety addressed in papers on model updates - when you deploy a system to production it filters the fraudulent examples and false positives out of all data that arrive after the deploy. What strategy should I use to improve the model's performance on the new fraud without losing performance on the old?
Example: Let's say the classifier was trained two years ago on one year of data. The recall is 50%, so given a 2% "actual" fraud rate and decent precision, the fraud rate in the data that showed up after the model was deployed is about 1%.
In the past year, the fraudsters have figured out new ways to defraud customers and committed new fraud that doesn't look like the old fraudulent examples. What's the best way to incorporate these new examples into training without diluting the signal of the old fraud?
Option 1 - sliding window
Say we just slide the time window forward one year. Now the only examples of fraud in the training set are examples that the old classifier missed. All the true positives and false positives get filtered out.
Option 2 - roll the end training date forward Keep the starting date, and train regularly (say, weekly) on new data that rolls in. As the model gets better, the percentage of fraud will get smaller and smaller.
Option 2 feels closer to the right approach, but I don't have intuition for how to pre-process the data. Do we downsample the negative examples to balance classes for each new batch of training data?
I'd love links to examples of systems, blog posts or papers that deal with this problem of a deployed classifier filtering out future data. Thanks so much!
