Interpreting cost change plot in a neural network for learning XOR

Question

I tried to build a neural net for learning XOR. The design is as follows:

1st layer: compute linear function of input 4:2 with 2:2 weights and adding 1:2 bias.
2nd layer: apply sigmoid to all elements in 4:2 matrix from layer 1.
3rd layer: compute linear function of the 2nd layer's output with 2:1 weights
final layer: apply sigmoid to 4:1 vector from the previous layer

The model converges well after as much as 1,000,000 of iterations (I don't know if it's too big of a number for such a simple problem.

I get the following plot (X - iterations, Y - cost). It seems a little bit strange. Why it stays on the same cost value for so long? Could anyone explain is it a normal graph for such a problem and is anything wrong with my setup that causes such cost behavior? Thanks

Answer 1

Gradient descent isn't a great optimizer. It's not unusual for GD to reach a plateau of very slow progress before finding a good descent direction again.

See this post for a comparison of GD and Levenberg-Marquardt for a simple linear regression network with poor conditioning.

• Can we apply analyticity of a neural network to improve upon gradient descent?

See this post for an explanation of why poor conditioning makes gradient descent challenging.

• For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

In the case of this particular task, it's true that a small, single-layer network with sigmoid neurons can solve the XOR task. However, there's a difference between the theoretical possibility of solving a task and whether or not it's practical to train a network to do so. One reason that networks are wider and deeper and use tricks like residual layers, batch norm, sophisticated optimizers and ReLUs is to make these networks easier to train using gradient descent.