Linear algebra use case

Question

I learning some machine learning course, and I would like to know in which case we use linea algebra and Matrix Algebra?

Thank you

Kind regards

Answer 1

I agree with Mark L. Stone's comment: too many examples, linear algebra is the foundation of statistics and machine learning, because your data is organized in columns and rows, and most widely used operations are multiplication and addition.

I will use linear regression as an example.

Suppose you are doing multiple linear regression, where you want to build a model for data $x^{(i)}$ (the $i$th data in your data set, and assume your data as $m$ features.)

$$ f(x^{(i)})=\beta_0+\beta_1x_1^{(i)}+\beta_2x_2^{(i)}+\beta_3x_3^{(i)}+\cdots+\beta_mx_m^{(i)} $$

And you want to minimize the squared error

$$ \sum_i (y^{(i)}-f(x^{(i)}))^2 $$

Such problem can be written in a very concise notation

$$ \min \|X\beta-y\|^2 $$

Where $X$ is data matrix, every row is a data instance, and every column is a "feature". $\beta$ is a vector and $y$ is the response vector. $\|\cdot\|$ is $L_2$ norm of a vector.

Here is where the matrix algebra comes in: take the derivative respect to $\beta$ of $\|X\beta-y\|^2$, you get $2X^T(X^T-y)$. You can set it to $0$ and solve it, which is the "normal equation" for linear regression, where

$$ \beta=(X^TX)^{-1}X^Ty $$

Note that, using matrix inverse is not a good solution from numerical analysis point of view. In real world, such as in R, QR decomposition is used. And that is more matrix algebra.

Check this post, you will see matrix operations are everywhere

What algorithm is used in linear regression?