How well should I expect Adam to work?

Question

I've been coding up a neural network package for my own amusement, and it seems to work. I've been reading about Adam and from what I've seen it's very difficult to beat.

Well, when I implement the Adam algorithm in my code it does terribly - converging very slowly or even diverging for some of the problems I've tested. It seems like I must have made an error, but the algorithm is pretty straightforward.

To cut out the possibility of some programming error, I decided to create a very simple function in Excel and compare Adam to standard gradient descent. From what I can see, standard gradient descent works better pretty consistently for lots of parameters (at least for relatively simple, deterministic functions). Adam seems to converge much more reliably regardless of what you feed it, but is consistently slower.

However - what I've read pretty consistently paints Adam as a panacea that converges significantly faster than any other algorithm in pretty much all situations. So what gives?

Does it only outperform other algorithms on sufficiently complex problems? Do the hyperparameters need to be tuned more carefully? Do I need to look at my network architecture more carefully if I'm not getting convergence? Are there certain activation functions that make it perform especially poorly? Or maybe I've just straight up implemented the algorithm incorrectly?

Here's an example where I compared standard gradient descent to Adam for x^2 + x^4, using a learning rate of 0.1 (and using 0.9, 0.999 and 1e-8 for the other Adam parameters). I've just plotted the gradient at each iteration, starting both off at x=1. Adam is slower to converge for this simple function for small learning rates, but it will converge for every learning rate I've tested (whereas standard gradient descent struggles to converge for learning rates over about 0.3). Does this look right or does it look like I've got something wrong?

Here's the intermediate variables for a few iterations of Adam:

I (perhaps naively) expected that I would just plug the Adam algorithm into my code with a stock set of parameters, and everything would just speed up. What am I missing here?

Thanks for any help!

Answer 1

There might be several factors at play here:

1. The optimal learning rate for momentum based algorithms is usually lower than that for plain GD, because momentum increases the effective step size. I don't know what rate you tried, but between $10^{-5}$ and $10^{-3}$ usually works for me.

2. Adam and many other SGD w/ momentum variants were designed for optimizing noisy, very high dimensional non-convex functions, with many saddle points and other pathologies. Your test on $x^2+x^4$ is pretty much the opposite of this, so it might not reflect the strengths of Adam.

3. iirc with a proper learning rate and decay schedule, SGD will converge to results which are just as good or possibly better than Adam. As you noted, reduced sensitivity to hyperparameter settings is an advantage of Adam

Answer 2

Actually, one of ADAM's key features is that it is slower and thus more careful. See section 2.1 of the paper.

In particular, there are pretty tight upper bounds on the step size. The paper lists 3 upper bounds, the simplest being that no individual parameter steps larger than $\alpha$ during any update, which is recommended to be 0.001.

With stochastic gradients, especially those with the potential for very large variations sample to sample, this is a very important feature. Your model may currently have near optimal parameter values at some point during optimization, but by bad luck, it hits an outlier shortly before the algorithm terminates, leading to an enormous jump to a very suboptimal set of parameter values. By using an extremely small trust region, as ADAM does, you can greatly reduce the probability of this occurring, as you would need to hit a very large number of outliers in a row to move a far distance away from your current solution.

This trust region aspect is important in the cases when you have a potentially very noisy approximation of the gradient (especially if there are rare cases of extremely inaccurate approximations) and if the second derivative is potentially very unstable. If these conditions do not exist, then the trust region aspect of ADAM are most likely to greatly slow down convergence without much benefit.