What is the definition of layer in neural network?

Question

What is the precise definition of layer in neural network? Are things like concatenate functions, activations, batch normalizations, skip connections considered as layers?

Answer 1

Here is my attempt at a definition.

I think the common usage of "layer" refers to a parameterized linear transform optionally followed by some parameterless nonlinear function.

For example, $\sigma(Wx+b)$ contains a linear transform parameterized by $W$ and $b$ followed by the nonlinear sigmoid activation.

I don't think most people would consider an activation function by itself to be a layer, so that's one point for this definition.

The term "softmax layer" usually means $\text{softmax}(Wx+b)$ where $W \in \mathbb{R}^{n \times m}$ and $b \in \mathbb{R}^n$ where $n$ is the desired number of categories. So it satisfies the definition.

I don't think many consider batch norm by itself to be a layer, although it may be referred to as a layer by certain programming frameworks. According to this definition, a convolution followed by batch norm followed by relu is a single layer. This agrees with how layers are counted in the ResNet-152 architecture.

Skip-connections and concatenations aren't counted as layers, and this definition accounts for that by considering them as part of the linear transform of the next layer they feed into.

This definition correctly rules out ResNet "residual-blocks" as single layers, since they have two separate parametric linear transforms separated by the nonlinear relu in between.

This definition also rules out a single-layer LSTM as a layer, which I think is fair, given its complexity. However many people probably do think of an LSTM as a single layer.

This definition also unfairly rules out quadratic neural networks which often compute a layer like $\vec y = \{x^T W_ix\}$.