0

I know that if the X matrix when using the normal equation is dependent then the transpose(X)*X will be non invertible and as a result there will be an infinite number of solutions to the problem.

I don't know why this is an issue. Why can't the normal equation still execute and then just return the first solution that it finds, even if there are many other equally good solutions?

tbrick
  • 77
  • 5
  • It can and it does. Are you using any computational technique, algorithm, or software that actually produces multiple solutions? Could you give an example of its input and output? – whuber Aug 25 '18 at 13:34
  • I am currently studying Linear Algebra https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/index.htm which talks about how the transpose(X)*X matrix must be invertible for the normal equation to work. This is also reinforced here: https://stats.stackexchange.com/questions/69442/linear-regression-and-non-invertibility where it is implied that if the matrix is not invertible then the normal equation won't work. – tbrick Aug 25 '18 at 18:14
  • I doubt that's what that course claims, because MIT professors generally know what they're talking about. In the other link I don't see that implication at all. Indeed, the highest-voted answer states early that "This equation has a single solution if $X^\prime X$ is invertible (non-singular). If it's not, you have more solutions." The other answer agrees. In either case, the Normal equations "work" and give the desired results. – whuber Aug 25 '18 at 18:54

1 Answers1

1

The solutions to the normal equations will typically be used as estimators of coefficients in some prediction model, yes? So, when you have infinitely many solutions, you need some criterion to choose between them. That can be done, and one possibility is to choose a minimum-norm solution, have a look at

Just return the first solution that it finds is very imprecise, and might not work well in a prediction problem!

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467