0

Two-part question: Neural Networks(NN) can be looked at as stacked units of logistic regression classifiers (LRC). A basic requirement of an activation function is to be non-linear. When LRC is a neuron, sigmoid function is the activation function and is said to bring non-linearity to NN.

Q1:) If this is true, why is LRC still a linear classifier even when it uses non-linear sigmoid function?

Further, if the activation function is a linear function(e.g. an identity function), the NN can no longer learn non-linear decision boundaries.

Q2:) Does this mean that the depth of a neural network plays no role in making it non-linear?

I have scanned similar threads for this question. But not convinced yet!

Abhinav Mishra
  • 1
  • 1
  • 1
    What's the definition of a linear function? How can you apply that definition to the particular case of the neural networks you're considering? – Sycorax Dec 08 '20 at 17:42
  • Hi, thanks, I edited my answer a bit. A linear function's graph would have a straight line, it's output can be expressed as a sum of it's inputs (could be scaled). I say if we use the identity function as an activation function, the final output for an input 'x' can be expressed as Wx + B, where W encompasses matrix products of all Ws of all layers of NN. Same goes for B. In my opinion, that does make an NN linear when a linear activation function is used. – Abhinav Mishra Dec 08 '20 at 17:58
  • Yes, the composition of several linear operations is itself linear. Some more detail can be found here https://stats.stackexchange.com/questions/228296/what-is-the-purpose-of-a-neural-network-activation-function/236386#236386 – Sycorax Dec 08 '20 at 18:02
  • Thanks. Is it okay to conclude that the non-linearity of a NN arises due to the combination of its depth and non-linear activation functions? Hence, either of the two alone will not suffice. – Abhinav Mishra Dec 08 '20 at 18:16
  • "Linear" and "nonlinear" are overloaded terms. From one perspective, logistic regression is "linear" in the sense that is (generalized) linear in the parameters to be estimated. But often the adjective “generalized” is omitted. From a second perspective, logistic regression is "nonlinear" because the sigmoid function is not a linear function. So, the answer to your question turns on what exactly you mean. – Sycorax Dec 08 '20 at 18:19
  • @Sycorax How do you figure that logistic regression is linear in the parameters? – Dave Dec 08 '20 at 18:24
  • @Dave That's addressed in one of the duplicate threads. – Sycorax Dec 08 '20 at 18:26
  • 1
    @Sycorax Ah! I see. Such overloadings have been a constant source of paradoxical intuitions for me. Thanks a lot for clarifying that they exist outside my mind too. – Abhinav Mishra Dec 08 '20 at 18:27
  • Yes, I had a similar breakthrough in my understanding when I realized that different people were meaning different things but using the same words. – Sycorax Dec 08 '20 at 18:29

0 Answers0