0

I have been studying AD for these days and I think I understand how it works, but all functions for which AD has been applied in the lectures I've studied are elementary in the mathematical sense, I mean, with a formula with elementary functions.

That's why I want to ask you an example of an aplication of AD to a function like a black box or if you can explain to me the way experts in the field use AD in real problems (using machine learning, for example).

Thank you very much.

Mads C
  • 3
  • 1
  • 1
    If you don't know the function, you cannot apply automatic differentiation to it. In real world machine learning scenarios, though, we always know the functions that are being automatically differentiated. – Jan Kukacka Feb 10 '20 at 10:39
  • @JanKukacka: Your comment seems like a reasonable answer for me. (and I agree with it.) – usεr11852 Feb 10 '20 at 13:20

2 Answers2

1

You can't do AD without a functional form because you can't take the derivative of something that you don't know. You can approximate derivatives numerically by sampling the function after making small changes to the inputs, which is a technique used in curve fitting.

People have managed to apply AD to some non-differentiable functions (like max pooling). In max pooling, the derivative ends up being just the derivative of the max position. In reality, changing the inputs might effect which position max pooling would select, and so the function is not differentiable.

SciGuy
  • 26
  • 2
0

I'm not familiar with automatic differentiation for blackbox functions. In machine learning, the loss is typically elementary, or close to it. For example, a simple feedforward neural network consists of matrix multiplications (multiplication is certainly elementary), sigmoid (which is elementary), and perhaps you have some cross-entropy loss, which is also elementary.

shimao
  • 22,706
  • 2
  • 42
  • 81