Let’s set up a supervised learning problem with $p$ predictors and $n$ observations. The response variable is univariate. The problem can be regression or classification, though I think a classification problem introduces additional complexity if there are more than two categories.
Euclidean distance between two vectors, which is quite related to square loss where the two vectors are the $n$-dimensional prediction vector and the $n$-dimensional vector of observed responses, has problems when the dimension is high (whatever “high” means).
Why is Euclidean distance not a good metric in high dimensions?
Nonetheless, square loss is popular, even when there are hundreds of observations and therefore dimensions (which should be enough to trigger some of the bizarre behavior of the Euclidean norm).
Yes, if we use square loss for OLS regression under the Gauss-Markov conditions, we get the BLUE. Yes, we can do all sorts of inference on the parameters. However, I am thinking of a pure prediction problem, perhaps a complicated neural network. In that kind of prediction problem, the inference and interpretation are much less important than the predictive accuracy.
So why use square loss when there are many observations?
