Why is gradient descent and it's variants used instead of BFGS and L-BFGS for training neural nets?

Question

My understanding is that BFGS and L-BFGS solve the same type of optimization problems as GD and it's variants.

Why is GD the go to algorithm for training neural networks?

Answer 1

You can use L-BFGS for optimization, it is included in some libraries as an optimizer, however it is very memory expensive algorithm, so many times it is more reasonable to use the gradient descent family.