Questions tagged [weight-initialization]

In the original Batch Norm paper (Ioffe and Szegedy 2015), the autors define Internal Covariate Shift as the "the change in the distributions of internal nodes of a deep network, in the course of training". They then present Batch Norm as a solution…

Background I have a collection of functions with trainable parameters that I am implementing as Keras model classes, which enables immediate use of a variety of objective functions, optimizers, and training-related methods (e.g. early stopping…

In class we discussed that if the weights of an ANN (standard feed forward NN in binary classification setting [0,1]) are initialized all at zero, the ANN fails to break symmetrie and therefore, the units in each layer develop equivalently. My…

My CNN architecture use pre-initialization, i.e. BatchNorm -> ReLU -> Conv. Which weight initialization shall I use for the convolutions? I'm under the impression that the standard ReLU initialization scheme of HeNormal is designed for Conv -> ReLU…