1

I am a bit confused about deep belief networks.

Should the RBM output be the input to the feed forward neural network for the fine tuning step or just the weights of the neural network have to be initialized which we get from the RBM?

Frans Rodenburg
  • 10,376
  • 2
  • 25
  • 58
jpj
  • 29
  • 2
  • Welcome to the site. You mention a fine tuning step, do you mean to ask how to use a deep belief network in transfer learning? Currently, it isn't clear to me what the question is. – Frans Rodenburg Apr 14 '21 at 12:30
  • Yes. I am a bit confused. Do I have to use the output from the RBM as input to my neural network for the fine tuning step or use the weights and bias from the RBM to initialise the weights and bias of the neural network? – jpj Apr 28 '21 at 12:49

1 Answers1

1

The point of transfer learning in general is to restrict the parameter space and act as a kind of regularization. You can think of the network as having some 'transferrable skill' from being trained on a tangentially related problem, for which a lot of data is available. The pretrained network then only has to learn how the target task is different from the task it was pretrained on, which allows you to use complex networks even when only limited data is available for the target task. To take image classification as an example: The idea is that the pretrained network is already able to discern shapes and features from the pixel intensities of an image.

What is then fine tuning? There are different ways in which you can use a pretrained network. Fine tuning refers to the case where you use the pretrained network as a starting point, and then continue training all parameters with the target task. It is the latter of the two options listed in your question: The weights of the network are initialized by the first task.

As shown in the linked answer, fine tuning is not the only option: You could also freeze the pretrained layers and add a dense layer at the end that connects to the output layer. Only these new connections to the output at the end are trained by the data you have on the target task.

Frans Rodenburg
  • 10,376
  • 2
  • 25
  • 58
  • Let's say I have a 3 RBM's stacked together with hidden layer structure [50,50,50]. My understanding is that the feed forward neural network will have the same hidden layer structure and weights between the layers will be the same as those between the corresponding RBM'S. And the input to the feed forward NN will also be the RBM output not the raw input that was the input to the 1st RBM in pre training. That's what I understood from page 3 of https://iopscience.iop.org/article/10.1088/1742-6596/1176/3/032046/pdf. But it's not giving me better generalization results and getting a very high loss – jpj May 02 '21 at 13:20
  • 1
    You're almost there: The *first layers* of the neural network indeed have the same [50, 50, 50] structure as the RBM. What the paper describes is that if you add more layers to your final neural network, *those layers* receive the output of what was the RBM. But if we consider the neural network to be [50, 50, 50, something], then the input to the first layer of the neural network is the same input as the RBM. Fine tuning in this case simply means that backpropagation of the final neural network also updates the first three layers that have been pretrained by an RBM. – Frans Rodenburg May 04 '21 at 07:21