Same neural weights in the same layer

Question

What will happen if we set the same weights to two neurons in the same layer?

What will happen if weights are 0 (zero)?

Explain why.

Please, also give some references for further study.

For instance, If we use a single layer, 2-neuron network,

the following picture is with random weights,

The following with all 1s,

The following with all zeros,

What neural network architecture do you use? Feedforward, fully connected? — Franck Dernoncourt, Jan 02 '17 at 14:45
@FranckDernoncourt, I am using Feedforward. But, I want to know both of the cases. — user366312, Jan 02 '17 at 16:17
Is it the same question as [What happens when many neurons have same weights?](http://stats.stackexchange.com/q/47592/12359)? — Franck Dernoncourt, Jan 10 '17 at 03:31

score 0 · Answer 1 · answered Jan 10 '17 at 17:53

Not clear what your exact scenario is. When you say "set the same weights", it can be interpreted as either a) initialising weights before training or b) constrain weights during training.

weights are typically initialised to small random values (not 0 or 1) and thus I am not sure if this is your scenario.

assuming that weights are constrained using a well known technique like weight decay or using a CNN/pooling technique.

What will happen if we set the same weights to two neurons in the same layer?

This kind of weight "sharing" will cause the neurons to learn the same features. In the context of a CNN, the effective number of parameters that have to be learnt reduces as a result. This improves the runtime efficiency of the network.

What will happen if weights are 0 (zero)?

It has the effect of reducing the capacity, this is equivalent to L1 weight decay regularisation.

Same neural weights in the same layer

1 Answers1