10

For image and tabular data, lots of people transform the skewed data into normally distributed data during preprocessing.

What does the normal distribution mean in machine learning? Is it an essential assumption of machine learning algorithms?

Even the image data, I've seen quantile transformation, which transforms the whole pixels of an image to follow normal or uniform distribution.

I can think of one reason: to avoid the influence of outliers. But these transformation distort the original distribution of data.

Why is the normal distribution so important to machine learning that lots of preprocessing includes this step?

mkt
  • 11,770
  • 9
  • 51
  • 125
林彥良
  • 103
  • 7
  • 2
    It's not, this comes from a wrong belief that models will perform better on normal data, but this is simply not true (except for models which actually require normality). Uniform data on $[0, 1]$ do sometimes help though, especially with NN, because of the way they work. – user2974951 Aug 01 '19 at 10:17
  • 4
    This superstition could be summarized as "Normality isn't a requirement for any model, unless it is." Novices often mistakenly *believe* normality is **always** a requirement, even though the cases where that is true are few and far between. – Sycorax Aug 01 '19 at 14:01
  • 4
    There are other, often more important, assumptions. **But those other assumptions are more difficult to understand**. – kjetil b halvorsen Aug 01 '19 at 14:19
  • So for neural networks, sometimes normal distribution data do help improve performance, but in other models it may depends on the data? And to check it I may have to try by hand or explore the mathematics behind the algorithms? – 林彥良 Aug 02 '19 at 00:50

2 Answers2

11

As @user2974951 says in a comment, it may be superstition that a Normal distribution is somehow better. Perhaps they have the mistaken idea that since Normal data is the result of many additive errors, if they force their data to be Normal, they can then treat the resulting numbers as having additive error. Or the first stats technique they learned was OLS regression and something about Normal was an assumption...

Normality is in general not a requirement. But whether it’s helpful depends on what the model does with the data.

For example, financial data is often lognormal -- i.e. has a multiplicative (percentage) error. Variational Autoencoders use a Normal distribution at the bottleneck to force smoothness and simplicity. Sigmoid functions work most naturally with Normal data. Mixture models often use a mixture of Normals. (If you can assume it’s Normal, you only need two parameters to completely define it, and those parameters are fairly intuitive in their meaning.)

It could also be that we want a unimodal, symmetric distribution for our modeling and the Normal is that. (And transformations to “Normal” are often not strictly Normal, just more symmetrical.)

Normality may simplify some math for you, and it may align with your conception of the process generating your data: most of your data is in the middle with relatively rarer low or high values, which are of interest.

But my impression is that it’s Cargo Cult in nature

Wayne
  • 19,981
  • 4
  • 50
  • 99
5

The answer above really nails it. I'd just like to add that it is worth separating the idea of wanting "normality" vs. wanting to scale all features to be on the similar range (even if they have different distributions). Both of these transformations have their pros and cons, and sometimes are necessary to avoid numerical quirks in the optimization step or avoid systemic biases in these algorithms.

Also, it depends what type of "machine learning" you're referring to (i.e., SVMs, tree-based models, neural nets, etc..), as these all behave differently and may have different numerical issues. As mentioned above, there are benefits in certain situations, but the idea that normalizing skewed data will lead to better performance is not a bullet-proof strategy. In general, justifying any "pre-processing" or "data manipulation/transformation" steps tends to be a more robust alternative.

Samir Rachid Zaim
  • 746
  • 3
  • 15