ADAM Gradient descent oscillates close to minimum

Question

I am using ADAM as an optimization algorithm to minimize some black box function $f(x,y)$. I know this function is convex and has a minimum $f(5,5) = 0$.

Initially, the algorithm proceeds as expected:

where the $x$ and $y$ axis are the respective variables, and the red dot indicates the target of $x=5, y=5$

However at small values of $f(x,y)$ I get,

This is a problem if I want convergence to zero within some precision $\epsilon \ \sim 10^{-12}$.

What is the cause of oscillations like this?

What is a solution to avoid/compensate for them?

Answer 1

What is the cause of oscillations like this?

SGD (with and without momentum) can naturally have oscillations. This is largely due to high curvature areas of the objective landscape. In particular, especially with momentum present, gradient descent behaves like a damped harmonic oscillator.

Some references:

• Goh, Why Momentum Really Works

• Lucas et al, Aggregated Momentum: Stability Through Passive Damping

• Yang et al, The Physical Systems Behind Optimization Algorithms

What is a solution to avoid/compensate for them?

One answer is to correct for this curvature, which is presumably high near the minimum. The first reference above suggests tuning the gradient descent hyper-parameters. Another option, depending on your situation (mostly the size of the model and the inherent noise in the gradient estimates), it may be possible to use second-order optimization methods instead, such as the natural gradient (see also KFAC) or Newton's method (e.g. see this, but also see here).