2

The most basic machine learning model called OLS uses the RSS (squared loss) or its average, mean squared error (MSE), for its loss function, which is aligned with Euclidean geometry.

What is the analogue of the MSE loss in the Riemann (non-Euclidean) world?

And is non-Euclidean based learning somehow better or empirically accurate than Euclidean learning algorithms?

develarist
  • 3,009
  • 8
  • 31
  • The following question is related to your question, discussing other loss functions and scenarios under which minimizing MSE loss is "best": https://stats.stackexchange.com/questions/146077/why-not-robust-regression-everytime/146100#146100 – Christoph Hanck Oct 14 '20 at 14:04
  • wasn't really helpful – develarist Oct 14 '20 at 14:05
  • 3
    OLS is not a model, it is an estimation technique. – Richard Hardy Oct 14 '20 at 18:11
  • isn't that just a matter jargon. we would then have to call SVM, random forests and other machine learning models "estimation techniques". but no one does. for all they care, they more often call them algorithms – develarist Oct 14 '20 at 18:13
  • 2
    There are some clear, conventional, well-understood distinctions between models estimators, and algorithms. Whether you consider the terminology jargon or not, it has a point. – whuber Oct 14 '20 at 18:32

1 Answers1

3

If you have a metric space $(M, d)$ with distance function $d$, calculate the distance between the vector of true values $y$ and the predicted values $\hat{y}$. For example:

$$d_{L2}(y, \hat{y}) \propto RMSE$$

$$d_{L1}(y, \hat{y}) \propto MAE$$

Scaling by the sample size (dimension of $y$ and $\hat{y}$) does not influence the argmin of a loss function (that is, $RMSE$, $SSE$, and $MSE$ have the same argmin).

Dave
  • 28,473
  • 4
  • 52
  • 104
  • I don't see how this has anything to do with Riemann (non-Euclidean) geometry – develarist Oct 14 '20 at 17:17
  • @develarist Euclidean space is a Riemannian manifold, so what do you mean by Riemann geometry? – Dave Oct 14 '20 at 17:24
  • non-Euclidean geometry as opposed to Euclidean space – develarist Oct 14 '20 at 18:16
  • 1
    $L_1$ is not Euclidean, so $MAE$ is a non-Euclidean loss function. – Dave Oct 14 '20 at 18:21
  • so MAE is a non-Euclidean analogue to MSE? are there any others – develarist Oct 14 '20 at 18:23
  • All of the $p$-norms would be analogous in $(\mathbb{R}, d_{Lp})$, but there are other non-Euclidean norms. I gave you a general answer in an arbitrary metric space. If you don't know what a metric space is, it is a space where "distance" makes sense and follows the usual nice properties that distance should have (e.g. distance from $A$ to $B$ is the same as the distance from $B$ to $A$). If you have further questions about metric spaces or Riemannian manifolds, those would warrant separate questions, most likely on Math.SE instead of here. – Dave Oct 14 '20 at 18:27
  • in your answer, you seem to be distinguishing $d_{L1}$ from $MAE$ by saying they are inversely proportional but not the same? how so – develarist Oct 14 '20 at 18:30
  • $L1$ metric is exactly equal to $MAE$, while the $L2$ metric is off of $RMSE$ by a factor of $\sqrt{n}$ (I forget if you multiply or divide, so I will leave this as an exercise ;)). The symbol means proportional, though, not inversely proportional. – Dave Oct 14 '20 at 18:32
  • 1
    The $L_p$ metrics are not Riemannian, either. By definition, a Riemannian manifold is locally Euclidean. – whuber Oct 14 '20 at 18:33