0

I was trying to show that in polynomial regression, the model overfits the data when the degree of polynomial $k$ increases. To demonstrate this, I had 30 2D datapoints and $k = 1,\dots,18$. I plotted the MSE training error against $k$.

However the plot shows that the training error kept decreasing until $k=10$, after which point the error rate goes up and down randomly.

From this answer I realise the problem might be that my design matrix is ill-conditioned so the solution becomes numerically unstable.

How can I get around this problem?

Thanks.

FrankieYin
  • 1
  • 1
    Have you used orthogonal polynomials? – Dave Nov 06 '19 at 16:43
  • @Dave Hi, I haven't used them. In fact I don't think I learned that in class. – FrankieYin Nov 06 '19 at 16:49
  • 1
    I found two links on CV: https://stats.stackexchange.com/questions/241703/orthogonal-polynomials-for-regression and https://stats.stackexchange.com/questions/258307/raw-or-orthogonal-polynomial-regression. My suspicion is that the issue is numerical instability. – Dave Nov 06 '19 at 17:09

0 Answers0