Training a neural network (R)

Question

I'm working on a neural network with one hidden layer. So far I've implemented the algorithm, and I've been able to numerically verify the partial derivatives I get from back propagation. My problem is when I try to train the network by updating the weights. No matter what I do it doesn't seem like I can get my estimates to come close to the actual values.

The data I'm using is from the function:

$$ f(x) = \frac{\sin(x)}{x} $$

On a side note, I haven't added bias parameters because I couldn't get the dimensions to fit in back propagation once I added the bias in forward prop. Not sure if this is an issue since I understood the bias parameters as a safeguard against inputs equal to 0?

theta <- function(a){
  a / (1+abs(a))      # Here we apply the alternative sigmoid function as our
                      # non-linearity.
}
theta.prime <- function(a){
  1 / (1+abs(a))^2
}

x <- c( 5.949110, -1.036600,  3.256780,  7.824520, -3.606010,  3.115640, -7.786960,
        -7.598090,  2.083880,  3.983000,  8.060120,  7.879760, -2.456670,
         -2.152720,  3.471950,  3.567960, -4.232630,  6.831610, -9.486860,  8.692330,
        -1.551860,  0.917305,  4.669480, -7.760430,  2.835410)
y <- c(-0.10804400,  0.78264000, -0.05313330,  0.13484700, -0.05522470, -0.05758530,
         0.19566100,  0.13846000,  0.43534100, -0.16861400,  0.10625000,
          0.08427310,  0.27012900,  0.44004800, -0.00880575, -0.10711400, -0.18671100,
         0.01158470,  0.02767190,  0.06319830,  0.61802000,  0.87124300,
         -0.25668100,  0.06160800, 0.10575700)


neuralnet <- function(x,y,theta,theta.prime,neurons){

w1 <- t(matrix(rnorm(neurons,0,.01),neurons,1)) 
w2 <- matrix(rnorm(neurons,0,.01),neurons,1)
E <- 1
i <- 0
  for(i in 1:1000){
  ### Forwardpropagation ###
  s2 <- x%*%w1
  a2 <- apply(s2,c(1,2),theta)
  s3 <- a2%*%w2
  yhat <- apply(s3,c(1,2),theta)
  ### Error function ###
  E <- sum((yhat-y)^2)/length(x)

  ### Back Propagation ###
  delta3 <- (2*(yhat-y)) * apply(s3,c(1,2),theta.prime)
  djdw2 <- t(a2) %*% delta3
  delta2 <- delta3 %*% t(w2) * apply(s2,c(1,2),theta.prime)
  djdw1 <- t(x)%*%delta2

  ### Update the weights ###
  w1 <- w1 - 0.01*djdw1
  w2 <- w2 - -0.01*djdw2
  i <- i+1
  }
print(i)
print(E)
return(yhat)
}

yhat <- neuralnet(x,y,theta,theta.prime,2)

Please register &/or merge your accounts (you can find information on how to do this in the **My Account** section of our [help]), then you will be able to edit & comment on your own question. — gung - Reinstate Monica, Jan 01 '17 at 18:38

score 0 · Answer 1 · answered Jan 01 '17 at 18:28

I haven't re-run your code but I have quickly gone through the code and have a few pointers which may be helpful:

You should add the bias inputs for each neuron. If the bias is not required, the optimization process will adapt the weights to minimize the error.
The theta function you're using is not a sigmoid. The abs() function returns the absolute value of a, whereas a sigmoid would use 1/exp(a) in its place.
The function you're trying to approximate sin(x)/x does not have a simple decision boundary. I don't think you can use two neurons to approximate it.

I'll be able to go through this in detail after I get a chance to step through your code. Meanwhile, please check if adding more neurons helps.

Thanks for the response! 1) I've been having a hard time adding the bias parameters. I just can't seem to figure it out with the dimensions of the weight vectors for each layer. I end up with non-conforming vectors when I try to do back propagation. 2) I corrected the tag in the post. It should have said "alternative sigmoid". 3) My assignment just states to run the network with 2 and 20 neurons. Both situations give me horrible predictions. — CPerkins, Jan 01 '17 at 19:01

Training a neural network (R)

1 Answers1