How to differentiate with respect to a matrix?

Question

How can I differentiate the following by $\mathbf{W}$ ?

\begin{equation} \mathbf{Y} = (\mathbf{W}^T\mathbf{x} + b)^2 \end{equation}

Where $\mathbf{W} \in \mathcal{R}^{d\times D}$ and $\mathbf(x)\in \mathcal{R}^{d\times 1}$

How to calculate $\partial \mathbf{Y}/\partial \mathbf{W}$ ?

Can you explain what is the context? In this kind of equations you usually differentiate the vector, and the matrix is constant. — Aksakal, Jan 08 '15 at 15:08
$\mathbf{W}^T\mathbf{x} + b$ does not make any sense. You cannot add a column vector to a scalar. — Rodrigo de Azevedo, Oct 14 '16 at 15:56
@RodrigodeAzevedo from context it is clear that b is a column vector that they forgot to bold. — John D, Jan 17 '22 at 08:59

Aksakal · Accepted Answer · 2015-01-08T16:36:41.683

Matrix calculus is used in such cases. Your equation looks like it's from OLS (least squares) theory. In those you differentiate by vector $x$ some quadratic forms like $\frac{\partial (x'A'Ax)}{\partial x}$. Look up relevant formulae in my link above.

If you really are up to differentiating by matrices not vectors, you'll end up with tensors. Tensors are fun, but so far I haven't seem them used a lot in statistics. They're ubiquitous in physics, btw. Again, follow the link I gave.

How to differentiate with respect to a matrix?

1 Answers1