3

What do the red ellipses mean to us in the chart below? Does it have a relationship with RMSE values?

enter image description here

Figure 3.11 from Elements of Statistical Learning by Hastie, Tibshirani, and Friedman

Community
  • 1
mezzocannone
  • 31
  • 2
  • 1
    This is copied from a book or paper apparently, and without knowing the context of the book it's impossible to say. – Christian Hennig May 15 '20 at 12:03
  • 1
    See also: https://stats.stackexchange.com/questions/414491/understanding-lasso-regressions-sparsity-geometrically – Sycorax May 16 '20 at 18:58
  • 2
    Does this answer your question? [Why L1 norm for sparse models](https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models) in particular, the answer which includes the exact same diagram – Sycorax May 16 '20 at 18:59

1 Answers1

2

The ellipses are the level sets of the loss function, which would be square loss (the loss function in OLS), which looks like an up-opening parabola that is spun around. Here is the equation for the simple linear regression depicted in your image.

$$L(y,\hat{\beta})=\sum_{i=1}^n \bigg( y_i-\big(\hat{\beta}_0+\hat{\beta}_1x_i\big)\bigg)^2$$

Yikes! But it’s actually not that bad because:

1) A computer is going to do the heavy lifting.

2) It just means the sum of the squared residuals. (Do you see why?)

The level sets are the shapes formed by intersecting horizontal planes with the function. Think about slicing a bowl this way; it would be a bunch of circles. Now compress the bowl so it’s rim is elliptical instead of circular; the slices are ellipses.

Note that L1 and L2 regularization apply in more generality than a regression that uses square loss. You can regularization in GLMs like logistic regression, for instance.

Dave
  • 28,473
  • 4
  • 52
  • 104
  • If there are n points in the validation data, should I consider these ellipses as graphic representation of the residues of these points? I may have asked with mixed and incomplete information, sorry. – mezzocannone May 15 '20 at 14:28
  • The $n$ refers to the points in the training set you then find the $\beta$ giving the smallest value of the loss function. (In OLS, that is the familiar $\beta=(X^TX)^{-1}X^T$.) So no, the ellipses have nothing to do with the points in the validation set, and the number of ellipses has nothing to do with the number of points used for training. Are you familiar with either loss functions or multivariable calculus? (Univariate calculus?) – Dave May 15 '20 at 15:00
  • When I say residues, I actually meant the loss function value. thanks for the explanation and help. Yes, I'm familiar with the topics, but sometimes it can get confused. – mezzocannone May 16 '20 at 09:55