1

I am very new to statistical learning (I'm a graduate student in experimental biology with very little exposure to math or statistics) and I'm working my way through Introduction to Statistical Learning to acquire an intuitive understanding of machine learning while slowly breaking down each equation in Elements of Statistical Learning.

I'm currently going equation (2.2) where the predicted Y is the inner product between X^T and estimated coefficient: Y\hat=X^Tb\hat

I am attempting to prove that the gradient of this equation is equal to: f\prime(X)=b

Since I'm still learning linear algebra, I decided to explicitly find the inner product of the transposed matrix and vector of coefficients:

equation1 equation2 equation3 equation3 finalequation

I ended up with a Beta^Nxp matrix where each row is the vector Beta. I realize that my fundamentals are very weak and I am going through linear algebra and multivariate calculus courses but I would appreciate any help with this calculation!

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
ojipanda
  • 11
  • 1
  • use that $\frac{\partial }{\partial \beta_p} x_i^\top\beta = x_{ip}$. As I see it you are on right track but when you diff with respect to $\beta$ in last step you get $x$'s not $\beta$'s. Also in general matrix partioning techniques are very useful in general - you may want to chek that out. – Jesper for President Dec 09 '20 at 03:21
  • @JesperforPresident Thank you for your response! Ah, I foolishly computed the derivative incorrectly. Differentiating it properly, I'm then left with the vector of transposed vectors of all p predictors for observation x_i. So I realize I was confused from the beginning, if I was trying to compute the gradient, I should have differentiated with respect to the matrix X and not \beta. Would that be the correct logic here? – ojipanda Dec 09 '20 at 04:52
  • 1
    The derivative with respect to beta is x an with x is beta. Rest is a matter arrangenent, is the derivative written with or without transpose. Which dervative you are interested in depends on context. Derivative with x says something about modelinterpretion giving marginal effects derivative with respect to beta is important when solving for estimator. – Jesper for President Dec 09 '20 at 09:39
  • The derivative of $X^\prime \beta$ with respect to $X$ is $\beta$ because $X^\prime \beta$ is *explicitly* a linear function of $X.$ No calculation is necessary (or even desirable); all you need is to interpret these symbols correctly. For definitions and computational approaches see https://stats.stackexchange.com/a/257616/919. – whuber Dec 09 '20 at 15:10
  • @whuper Thanks for clarifying guys! I had been attempting to calculate it to practice matrix calculus and understand what is going on mathematically. I think this question shows my lack of understanding of the mathematical background/fundamentals so that's what I'll focus on for now. – ojipanda Dec 10 '20 at 05:07

0 Answers0