3

I am a total beginner in Machine Learning but I would say that I have kind of strong mathematical background. I have started learning Machine Learning algorithims and I have the following question. Thank you for your help in advance. As we know, our basic linear regression algorithm is $\hat{\beta} = (X^T X)^{-1} X^T Y.$ I have tried this algorithim for $y = 2x + 5$ For example, (the first column is ones vector and imagine that we give $1, 2, 3, 4$ numbers each for $x$ and get $7, 9, 11, 13$ for $y$ values)

x = [[1, 1]; [1, 2]; [1, 3]; [1, 4]] y = [[7]; [9]; [11]; [13]]

When I solve the algorithm, it gives $2$ and $5$ which are the parameters of $y=2x + 5$. There is no problem till here. My question starts when I use this algorithm for, lets say $y=X_1 + X_2 + 5$. When I use this algorithm for this equation, I cannot get $1, 1, 5 $ parameters as a solution. The inputs I use as following: (I give $1, 2, 3, 4$ for $X_1$ and $2, 3, 4, 5$ for $X_2$ and get $8, 10, 12, 14$ for $Y$) (Again, the first column is ones vector)

X = [[1, 1, 2]; [1, 2 , 3]; [1, 3 , 4]; [1, 4 , 5]]
y = [[8]; [10]; [12]; [14]]
George
  • 156
  • 9
Jonpromie1
  • 35
  • 3
  • Are you sure if the inverses are computed correctly? Because the rank of X*X' of your example is not 4, it is 2. This will lead to inaccuracies while inverting the matrix. And therefore the results may not be exactly what you expect. This also means that the problem has to be defined differently if you want to get a unique solution. – lekshmi dharmarajan Sep 01 '16 at 11:34
  • The answers to https://stats.stackexchange.com/questions/69442/linear-regression-and-non-invertibility are relevant – Adrian Sep 01 '16 at 12:26

1 Answers1

1

Your third column is a linear combination of the first two. Your design is very ill-conditioned. Try fudging with the independent parameters and try again to see if a well-conditioned problem won't behave nicely.

Beyer
  • 1,072
  • 1
  • 7
  • 10
  • I tried with a different input and it worked! Thank you! – Jonpromie1 Sep 02 '16 at 05:45
  • You're very welcome. As far as I'm concerned: The first corollary of toy problems is that you always build a toy problem that breaks something fundamental :) If this answer solved your problem, feel free to accept the answer formally. – Beyer Sep 02 '16 at 09:41