Balance classes when sampling

Question

I have a large dataset describing numerous customers' behaviour and I am trying to solve a binary classification problem with a null accuracy on 90% (90/10 distribution amongst the two classes).

Given that I have computational limitations and thus are forced to take a subset of the sample, would it make sense for me to manipulate the balance to, let's say; 60/40 or 50/50 in my sample, now that I am limited to a fixed amount of total observations due to my hardware, just to "expose the machine learning algorithm to more of both classes" (from an marginal utility point of view)?

I have found multiple discussions about this online but not about this exact situation. I am very much aware of the fact that it would be optimal to just use ALL observations, and that it will mess up the true disitribution, but my rationale is that the problem is nothing like a poll sample but rather the idea of feeding the algorithm with more examples of observations that it haven't seen that many times.

Following guide states: "Consider testing under-sampling when you have a lot data (tens- or hundreds of thousands of instances or more)"

Would this impact the performance of the machine learning algorithm negatively and thus my prediction model so that I will get worse classifications on a 90/10 test set? And would someone be able to explain me why?

Answer 1

In general, more data is preferred, but not necessarily the raw full data, especially when it is imbalanced.

The robustness of algorithms to imbalanced data varies from one to another, take naive Bayes as an example, the core formula of this algorithm is:

$$\mathbb{P}(Y|X) = \mathbb{P}(Y)\mathbb{P}(X|Y)/\mathbb{P}(X)$$

We can see that the prediction probability $\mathbb{P}(Y|X)$ for any $X$ is proportional to $\mathbb{P}(Y)$, which in your case, is either $0.1$ or $0.9$, that is to say the original distribution of $Y$ has a huge impact on the result of prediction. In other words, naive Bayes can't handle imbalanced data very well.

So what to do?
a) You can choose another algorithm that does better job in this situation such as Random Forest
b) Resampling your data
c) Reweight your data by $1/\mathbb{P}(Y)$
... and so on

In conclusion, more data is better provided that it is well formed, and when it's not the case, keeping the integrity of it doesn't really make any sense.

Answer 2

I think one needs to take into account that changing the class proportions in the training sample will substantially change the final model that is going to be learned through training. This might not be something we necessarily want. The cost of misclassifying majority class samples might be not negligible (invasive medical treatments to rare diseases being a standard example). When down-sampling a dataset we intrinsically hypothesis that the cost of misclassification of their class is similar but this might not be the case; cost-sensitive learning (Elkan,2001) is potentially important concept in this setting.

In addition, before randomly under-sampling a dataset it could be beneficial to look at a number of resampling algorithms (eg. NearMiss - (Zhang & Mani (2003)) or One-sided selection - (Kubat & Matwin (1997)) which can help us do an informed downsampling of a dataset. Majority downsampling algorithms while far less famous than their minority oversampling cousins (eg. SMOTE) still exist! In that way when downsampling the data we are able to discard points that (in principle at least) do not greatly benefit our training process while at the same time retaining useful exemplars.

Finally, examining carefully the "large dataset" it might be possible to construct a better training set through it. By that I mean to recognise latent features or patterns in the data that can assist our training while at the same time lower the size of our training dataset in absolute terms. Feature engineering is an extremely crucial part of any real life Machine Learning application. It allows domain expertise to enter the modelling task; in addition it effectively changes the representation of our original problem (hopefully to a smaller but more informative set in this case). To that extend we might want to use a dimensionality reduction technique like PCA or ICA to reduce the size of our feature space in absolute terms and possibly avoid hardware limitations altogether.