Is it in general helpful to add "external" datasets to the training dataset?

Question

Several people have already asked "is more data helpful?":

• What impact does increasing the training data have on the overall system accuracy?
• Can increasing the amount of training data make overfitting worse?
• Will a model always score better on the training dataset than the test dataset?

I would like to ask "is more external data helpful"? By external I mean data from a similar though not equal domain. For example, if we want to detect cars in Japan, I would consider a U.S. dataset as external since the average car (and street) look different there. Or a dataset taken with the same kind of objects but a different camera.

The reason I'm asking is that many papers seem to use external datasets with great success. For example, depth estimation methods additionally train on the CityScape dataset to perform prediction on the KITTI dataset, see paper. Similarly, external datasets are often used in kaggle competitions. Last, a 2014 paper reports the "surprising effect" that pretraining on the first half of ImageNet classes and then finetuning on the other half yields better results than training only on the second half of classes. On the other hand, this paper reports in Fig.2 that adding new datasets worsens the error. Thus, what is your experience? Are there any guidelines or interesting review articles? Or do you simply have to "try it out" always?

EDIT: To clarify, with "more data" I mean more rows (not more columns / features). More specifically, I am assuming a computer vision problem where more data corresponds to more images.

Answer 1

At some point, adding more data will result in overfitting and worse out-of-sample prediction performance. Always.

That papers report improved accuracy by leveraging additional data is not surprising at all. After all, people (both in academia and in industry) are heavily incentivized to report precisely this. Here is the relevant algorithm:

1. Pick an external dataset D.
2. Can you tell a story about how D *might* improve accuracy?
    If no: GOTO 1
3. Fit your model using D. Does it improve accuracy?
    If no: GOTO 1
4. Publish your accuracy improvement using D. Bonus points if you can get a press release.

Note how a publication only happens if accuracy improves. You don't see all the loops where accuracy didn't improve. This is called a "file drawer effect" (everything that is not successful ends up in a file drawer). The end result is a strong publication bias.

Note also that step 2 is crucial. An ability to tell a story about how the accuracy improvement might have come about is indispensable, because if you don't have such a story, it's too blatant that you went on a wild goose chase.

So: in order to know whether your external data actually did improve matters, you always need to keep from "overfitting on the test set", as the algorithm above does. If you follow this algorithm, don't be surprised if the "winner" does not perform as well in production as after this selection process (which in itself is an example of regression to the mean).

Answer 2

I think the examples you bring are mostly from computer vision/image recognition and that case external datasets are very likely to include similar signal/dynamics as the prior data at hand. A "car" is a "car" irrespective of its surroundings. A "good customer" or "abnormal shopping activity" is different in Luxembourg than it is in Moldova. Unless we actively account for "covariate shift" (input distribution changes) and/or "concept drift" (ie. correct output for a given input changes over time/space/etc.) then "more data is helpful" only if we are lucky. We should note that this includes computer vision too; for example if our additional data is biased in a way we are unaware and/or unable to control (e.g. the photos are always at night-time or are subjected to over-exposure) that will not necessarily help the generalisability of our model.

Answer 3

It depends. One way to think about this problem is as follows. The data in your training and test/out-of-sample sets can be modeled as h(x) + noise. Here, the noise is the variability in your data that is not explained by some common (theoretically optimal) model h(x). The important thing here is that if your training and test data are sampled from entirely different/unrelated distributions, then ALL of your training data is noise, even if on their own, both training and test set data are very well structured.

What that means is that the more different the external dataset is to your test data, the greater the amount of noise in it. The greater the amount of noise, the easier it is to overfit (i.e. fit your model to noise - as defined above). For your car example, that would mean that a complex model might fit to the specifics of US number plates, which is not part of h(x) when it comes to detecting cars in Japan.

Having said that, if your goal is to make your model more robust (i.e. you want your car-in-Japan model to still work if the numberplate design is changed, or in some other way the distribution of your OOS data changes), then introducing the US dataset might help - in this case, the Japanese idiosyncrasies also become a part of 'noise' and in, e.g., cross-validation, you will be forced to come up with perhaps simpler models that pick up features that work both in the US and in Japan, making your model more general and therefore more robust.

So the answer is that it really depends on your data, on what the external data is, and on what you are trying to achieve.