Why is my neural network not learning?

Question

I am experimenting with neural networks in python with PyBrain.

Here is the problem I'm trying to solve: Given a sorted list of integers of size 5, where each integer is in the range 0-12, determine whether at least two of the numbers are identical.

For example, [1,3,5,7,11] has 5 distinct numbers whereas [1,1,3,4,5] does not.

I'm using PyBrain's backpropagation trainer with 2 hidden layers -- 20 neurons on each. I have experimented with a lot of different topologies, so this shouldn't be the source of the problem.

I divide each number by 12 so that each value is in the range 0-1.

When training the network, the trainer reports a constant error and the network never identifies an input as having duplicate entries.

Here is my code:

from pybrain.datasets            import ClassificationDataSet
from pybrain.utilities           import percentError
from pybrain.tools.shortcuts     import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.structure.modules   import SoftmaxLayer

alldata = ClassificationDataSet(5,1, nb_classes=2)

#Build our training data
for a in range(0,13):
    for b in range(a, 13):
        for c in range(b, 13):
            for d in range(c, 13):
                for e in range(d, 13):
                    inputList = [a,b,c,d,e]
                    for j in range(len(inputList)):
                        inputList[j] = float(inputList[j])/12
                    inputSet = set(inputList)
                    kclass = 0
                    if (len(inputSet) == len(inputList)):
                        kclass = 1

                    alldata.addSample(inputList, [kclass])

tstdata, trndata = alldata.splitWithProportion(0.25)

trndata._convertToOneOfMany()
tstdata._convertToOneOfMany()

#Build network with 20 neurons on each of 2 hidden layers
fnn = buildNetwork(trndata.indim, 20,20, trndata.outdim, outclass=SoftmaxLayer)
trainer = BackpropTrainer(fnn, dataset=trndata, momentum=0.1, verbose=True, weightdecay=0.01)

#Train network for 5 epochs
trainer.trainEpochs(5)

trnresult = percentError( trainer.testOnClassData(),
                              trndata['class'] )

tstresult = percentError( trainer.testOnClassData(
           dataset=tstdata ), tstdata['class'] )

print "epoch: %4d" % trainer.totalepochs, \
          "  train error: %5.2f%%" % trnresult, \
          "  test error: %5.2f%%" % tstresul

For experimentation purposes... To gain an intuition as to what kind of mapping neural networks can and cannot learn... — Uwat, Jun 29 '13 at 20:13
I don't know pybrain but did you test your method first on smaller example, like XOR that should be not far from what you are looking for. Second, increase your number of epoch, I don't think 5 iterations is enough to go anywhere. — ThiS, Jun 29 '13 at 20:23
I build a network in the same way to determine which of 2 numbers is the largest (1st or 2nd)... It worked fine... 5 iterations might be too small, but how the error changes (it does not improve) after each iteration, it does not seem like more iterations would help... — Uwat, Jun 29 '13 at 20:40
@ThierrySilbermann, it turns out your suggestion was good. I used a single hidden layer with 30 neurons, and 200 epochs. truePos = 1242, falsePos = 61, trueNeg = 4840, falseNeg = 45... Those are quite good results. Is 200 or more epochs typical for a problem of this complexity/size? — Uwat, Jun 29 '13 at 21:54
I have seen networks run tens of thousands of epochs before convergence, but convergence time depends on data size and complexity, network size and complexity, and training method. — EngrStudent, May 30 '14 at 15:38
See https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn — kjetil b halvorsen, Nov 04 '21 at 11:48

score 1 · Answer 1 · answered May 29 '14 at 18:03

You are trying to get order statistics in a non-order statistic framework.

You could change your input vector from being a 5x1 vector of integers to being a 10x1 vector comprised of the first 5x1 vector that is then concatenated to the rank of each value. Easy idea to think, hard to write in words. Code might help, see below.

I use the following MATLAB code to make a sample vector:

x1=randint(5,1,[0,12])
[~,~,x2]=unique(x1)
X=[x1;x2]

It looks like you are trying to wrap sorting and order-based comparison in a classic one-off algebraic expression instead of a loop. The problem is that sorting is an NP-easy (non-polynomial easy) task. the P vs. NP part of the problem means it is unlikely that there exists an analytic expression for the NN to converge to. The NN learns from data and approaches an analytic expression, but if the expression does not exist then you have problems. By preprocessing the order statistic you then return the problem to be within what the NN (assuming it is well behaved and properly related in topology to the problem) is capable of approximating.

Best of luck.

Why is my neural network not learning?

1 Answers1