Matrix calculus deals with the problems of differentiating (possibly matrix-valued) functions of matrices
Matrix calculus deals with the problems of differentiating (possibly matrix-valued) functions of matrices, extending further the well-known calculus formulae.
From "regular" calculus, the section on derivatives of powers, we know that $$ \frac{\rm d}{{\rm d}x} x^{-1} = -x^{-2} $$ How does this look for matrices? What is the derivative of the inverse of a matrix... if any? Well it turns out that that the concept of a derivative of a matrix with respect to a matrix is a fairly complicated one: if you take a derivative of an $m\times n$ matrix by a $p\times q$ matrix, you end up with an object that must have $m\cdot n\cdot p\cdot q$ entries. This could be a tensor with four dimensions, but few statisticians have training in tensors (most theoretical physicists do, though). It is, however, reasonably easy to talk about differentials of matrices.
If a matrix $A$ is given by $$ A = \begin{pmatrix} a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & a_{22} & ... & a_{2n} \\ ... & ... & ... & ... \\ a_{m1} & a_{m2} & ... & a_{mn} \end{pmatrix} $$ with elements treated akin to the elements of a multivariate vector in multivariable calculus, then its differential is $$ {\rm d}A = \begin{pmatrix} {\rm d}a_{11} & {\rm d}a_{12} & \ldots & {\rm d}a_{1n} \\ {\rm d}a_{21} & {\rm d}a_{22} & ... & {\rm d}a_{2n} \\ ... & ... & ... & ... \\ {\rm d}a_{m1} & {\rm d}a_{m2} & ... & {\rm d}a_{mn} \end{pmatrix} $$
Let us re-write the derivative expression for the inverse in differentials: $$ {\rm d}\bigl( x^{-1} \bigr) = -1/x^2 \, {\rm d}x $$
The matrix differential of the inverse simply takes care of lack of commutativity of matrix operations: $${\rm d}(A^{-1}) = -A^{-1} \, {\rm d} A \, A^{-1}$$ Of course, this simplifies to the "standard" expression with $m=n=1$, and matrix $A$ is simply a scalar.
Most of the operations have the differentials you'd expect them to have: $$ {\rm d}(A+B) = {\rm d}A + {\rm d}B $$ $$ {\rm d}(AB) = {\rm d}A \, B + A {\rm d}B $$ $$ {\rm d}(A \otimes B) = {\rm d}A \otimes B + A \otimes {\rm d}B $$ where the latter is the Kronecker product. Some matrix operations are, however, unique to matrices, and differentials for them require extra work, such as the determinant $$ {\rm d} \, {\rm det} A = {\rm det}A \cdot {\rm tr} (A^{-1} \, {\rm d}A ) $$ where $A$ is full rank, or the eigenproblem: $$ A=A^T, Au = \lambda u, \| u \| =1 \Rightarrow {\rm d}\lambda = u^T ({\rm d}A) u, {\rm d}u = (\lambda I_n - A)^+ ( {\rm d}A) u $$ where $X^+$ is the Moore-Penrose inverse of $X$. If $A$ is not symmetric, the eigenvalues and eigenvectors are, in general, complex, and the expressions for differentials are more complicated.
If $x \in \mathbb{R}^n$ and $f: \mathbb{R}^n \to \mathbb{R}^m$ is an $m$-dimensional vector function of $x$, then the Jacobian matrix of this transformation is the $m \times n$ matrix of partial derivatives: $$ {\rm D} f(x) = \frac{\partial f(x)}{\partial x'} = \begin{pmatrix} \frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} & \ldots & \frac{\partial f_1}{\partial x_n} \\ \frac{\partial f_2}{\partial x_1} & \frac{\partial f_2}{\partial x_2} & \ldots & \frac{\partial f_2}{\partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \frac{\partial f_m}{\partial x_2} & \ldots & \frac{\partial f_m}{\partial x_n} \end{pmatrix} $$
Having introduced the differentials, Magnus and Neudecker (2007) argue in favor of the following definition of the derivative of a matrix function $F(X)$, if you really want to define one, based on the Jacobians: $$ {\rm D} F(X) = \frac{\partial {\rm vec} F(X)}{\partial ({\rm vec} X)'} $$ They discuss the advantage of this definition in terms of mathematical consistency it provides (such as that the derivative of an identity function should probably be some sort of an identity matrix, that one-to-one transformations have non-degenerate Jacobians with non-zero determinatns, and that the chain rule could be defined in the traditional manner for matrix functions, etc.) over some other definitions. However in many problems, working directly with differentials is more convenient than working with derivatives.
References:
Magnus, J. R., and H. Neudecker (2007). Matrix Calculus with Applications in Statistics and Econometrics, 3rd ed
