Unsupervised pre-training initializes a discriminative neural network from one which was trained using an unsupervised criterion, such as a deep belief network or a deep autoencoder.
Questions tagged [pre-training]
47 questions
38
votes
3 answers
What is pre training a neural network?
Well the question says it all.
What is meant by "pre training a neural network"? Can someone explain in pure simple English?
I can't seem to find any resources related to it. It would be great if someone can point me to them.
Machina333
- 863
- 2
- 9
- 10
11
votes
3 answers
What is pretraining and how do you pretrain a neural network?
I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because backpropagation is slow, and also that I can get stuck…
Michael Yousef
- 251
- 2
- 3
9
votes
1 answer
Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?
Is it possible to achieve state of the art results by using back-propagation only (without pre-training) ?
Or is it so that all record breaking approaches use some form of pre-training ?
Is back-propagation alone good enough ?
user70990
8
votes
1 answer
Is initializing the weights of autoencoders still a difficult problem?
I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it.
I have been reading different articles. In one of Hinton's papers (2006), it says:
With large initial weights,…
Shannon
- 211
- 1
- 3
7
votes
3 answers
Feature Selection in unbalanced data
I was always taught 3 things:
Training algorithms (rf, trees, etc) don't perform well with unbalanced data.
I should balance data only after performing feature selection (mainly to keep variables independent)
Feature selection algorithms usually…
Riddle-Master
- 433
- 3
- 14
7
votes
0 answers
Change image input size of a pre-trained convnet
maybe this question will sound a bit as a newbie one but I'd like to have some clarification.
I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my classification problem; specifically I removed the…
matteodv
- 171
- 2
- 5
7
votes
1 answer
How to pretrain Convolution filter
I was implementing convolutional neural network, For classification of natural images like face, car, flower etc of about 10 categories. I read(from Andrew NG notes) that pre trained convolutional filter are much efficient and less resource…
madan ram
- 173
- 5
5
votes
2 answers
Why does pre-training help avoid the vanishing gradient problem?
I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down through the network becomes ever more small.
In…
volperossa
- 625
- 5
- 9
3
votes
1 answer
Rationale for different activation function neural network pretraining vs. supervised training?
I was reading a paper that used neural networks to predict protein conformation, Improving prediction of secondary
structure, local backbone angles,
and solvent accessible surface
area of proteins by iterative deep
learning:
Linear activation…
user49404
- 407
- 3
- 10
3
votes
0 answers
State of the Art Status of Deep Boltzmann Machine and Pretraining
I have been reading some old papers by Hinton on deep Boltzmann machine and deep belief networks, but I wonder what the current status is regarding these models:
Are DBM and DBN totally outdated? I can understand they do not work as well as CNN and…
DiveIntoML
- 1,583
- 1
- 11
- 21
3
votes
1 answer
Visualizing model trajectories for Neural Networks using function approximator
Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15
In there, they compare different neural network models by visualizing the function representation…
The Wanderer
- 647
- 4
- 16
3
votes
1 answer
Using pretrained segmantation network for unseen motives
For a research project, I need to do a segmentation on images. Since the motivation is nothing any of the big networks was ever trained on, I would ask if it still makes sense to use pretrained segmentation networks like SegNet to do the…
Luca Thiede
- 657
- 6
- 13
2
votes
1 answer
Transfer learning for regression problems
I have trained a regression model with 7 features for a given problem.
Now, I have another regression problem (quite similar to the previous one) where I have only 6 samples in hand, but with 3 more features than the first model (7+3). The…
jojo
- 153
- 4
2
votes
1 answer
How to resume training in neural networks properly?
I'm working on training a network to identify different kinds of cells. For each experimental batch, I would take my previous model weight, and then train a few new pictures on it. Since the model is for the same kind of data, thus this is not a…
kikyo91
- 23
- 3
2
votes
0 answers
Pre-training without seeing data
Is there a solid reference on pre-training methods in deep neural networks which never see the actual inputs? Any such known thing in literature?
I guess a more correct term is "initialization using gradient methods" instead of "pre-training".
I see…
Daniel Paleka
- 173
- 4