Why are the "Loss Functions" being Optimized in most Statistical/Machine Learning Problems usually "Quadratic"?

Question

Why are the "Loss Functions" being Optimized in most Statistical/Machine Learning Problems usually "Quadratic"?

Using very basic logic, in statistics/machine learning we are trying to minimize the "error" between a model and some "hypothetical ideal function" that perfectly models the data. This "error" is typically described as "Mean SQUARED Error" (MSE) - and the function characterizing this error is what we are trying to minimize. "Squared" is said to be the main characteristic of "quadratic functions" (i.e. the highest power term in a quadratic function is "power 2", i.e. squared). I have heard the reason that loss functions are typically quadratic is because quadratic functions have certain "attractive and desirable theoretical properties" that facilitate convergence for such tasks as root finding and minimization - but I am not sure about this and what these properties are:

Thus - no matter how many parameters in the loss function being optimized (e.g. neural networks with many weights/layers), and no matter how complex the behavior of the loss function being optimized : Is it safe to assume that the loss functions being optimized in most statistical/machine learning problems are usually "quadratic"? And does anyone know why this is?

Can someone please comment on this?

Thanks!

Note : I have heard similar arguments being made in reference to describing the "non-convexity" of loss functions being optimized in most statistical/machine learning problems. Although there are standard definitions in mathematics used to determine whether a function is convex or non-convex (e.g. https://en.wikipedia.org/wiki/Convex_function, "Definitions"), high dimensional loss functions containing "random variables" (opposed to non-random deterministic functions in classical analysis, i.e. "noisy") are almost always said to be non-convex. This has always made me wonder that even though optimization algorithms like Gradient Descent were designed for convex and non-noisy functions - they are still somehow able to display remarkable success when optimizing non-convex and noisy functions.

Answer 1

No, it is not safe to assume that all loss functions are quadratics. One of the most common cost functions is the binomial cross-entropy $$ L(p) = y\log p +(1-y)\log(1-p) $$ where $0 \le p \le 1$ and $y \in \{0,1\}$. The function $L$ is not a quadratic because it is not a polynomial of degree 2.

Answer 2

Five reasons come to mind quickly.

1. Square loss brutally punishes bad misses. If you miss by $1$, your square loss is $1$, but if you miss by $2$, your square loss is $4$. This helps keep a model from making gigantic errors.

2. Square loss is related to the variance of an error term, if you’re willing to assume that variance to be constant.

3. Minimizing square loss seeks out the conditional expected value.

4. Minimizing square loss corresponds to maximum likelihood estimation if the conditional distribution is Gaussian. Maximum likelihood estimation is a technique with which statisticians are quite comfortable.

5. Tradition! The first type of machine learning one learns is “trendline” in Excel. Then one learns how to extend that by learning simple and then multiple linear regression via ordinary least squares, which minimizes square loss.

That being said, many other loss functions are popular. “Crossentropy” is popular for classification problems, for instance. The neural network libraries like Keras list many popular loss functions that are built-in.