9

$MAE=|y_{pred} - y_{true}|$

$\dfrac{dMAE}{dy_{pred}} = ?$

I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some APIs - e.g. Keras - however I see tensorflow doesn't allow it (although you can manually declare it tf.abs(tf.minus(y_pred,y_true))).

My question is: How is the derivative of MAE generally calculated (I only found this which uses an apparently complex approximation) and specifically how is it computed in tensorflow (when manually declaring) and keras?

tea_pea
  • 536
  • 1
  • 6
  • 11

2 Answers2

14

The , as a function of $y_{\text{pred}}$, is not differentiable at $y_{\text{pred}}=y_{\text{true}}$. Elsewhere, the derivative is $\pm 1$ by a straightforward application of the chain rule:

$$\dfrac{d\text{MAE}}{dy_{\text{pred}}} = \begin{cases} +1,\quad y_{\text{pred}}>y_{\text{true}}\\ -1,\quad y_{\text{pred}}<y_{\text{true}} \end{cases}$$

The interpretation is straightforward: if you are predicting too high ($y_{\text{pred}}>y_{\text{true}}$), then increasing $y_{\text{pred}}$ yet more by one unit will increase the MAE by an equal amount of one unit, so the gradient encourages you to reduce $y_{\text{pred}}$. And vice versa if $y_{\text{pred}}<y_{\text{true}}$.

Skimming the paper you link, it seems like they approximate the MAE by a differentiable function to avoid the "kink" at $y_{\text{pred}}=y_{\text{true}}$.

As to what specifically is implemented in TensorFlow and keras, that is off-topic here. Best to consult the documentation, the source code or any specific help community.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Thanks, I'm guessing implementations are programmed to default to either +1 or -1 if y_pred=y_true, I'm sure I've read somewhere that's how the Relu is normally handled. – tea_pea Nov 09 '17 at 11:17
  • 1
    But doesn't this mean that the weighted increase or decrease is independent of the magnitude of the error? – Pranit Bankar Mar 10 '20 at 04:54
  • 2
    @PranitBankar: yes indeed. If you are at any point that is different from the minimum, moving towards the minimum will reduce the MAE value - and the amount of the reduction does not depend on where you started. – Stephan Kolassa Mar 10 '20 at 06:08
1

As for Tensorflow's Keras here is the good answer: https://stackoverflow.com/a/35344963

And if we will dig into the source code we will see that it uses math_ops.sign wich is return 0 for 0 https://github.com/tensorflow/tensorflow/blob/c91e944d626b517781af6a63c0aee302ab2457e3/tensorflow/python/ops/math_grad.py#L577

Divien
  • 21
  • 2