2

Is the following hypothesis true ?

If a simple neural network cannot overfit a single training sample, there is something wrong with its architecture or its implementation.

To give you more background on why I am asking this question, I am working on a single convolution layer that aims at segmenting the input image (classify every pixel of the image from either class 0 or class 1). The network does not manage to overfit a single training sample, so I suppose that there is something wrong with what I have done.

Edit: This is not a duplicate of What should I do when my neural network doesn't learn?. The post (which is very informative) suggest, among other things, to unit test the network to see if it is error-proof. Basically, I am asking a question on how to unit test my network. The hypothesis I stated is the one which I hold to run the unit test. If the hypothesis is wrong, the unit test I am making does not make any sense, thus the question.

Xema
  • 195
  • 7
  • In a way, I am unit testing my former whole FCN32 (close to a VGG16) with this simple network. Your link is good, though. Thanks for answering! – Xema Sep 07 '18 at 09:18
  • The goal of a unit test is to check something which is known to work. Since you're not checking your actual architecture, what are you checking exactly? The way you read & feed data? Then it would be much better to use an existing, pre-trained architecture for image segmentation, for the unit test, rather than a single convolution layer (?!?). – DeltaIV Sep 07 '18 at 09:50

1 Answers1

2

In this case I don't think it's so much about unit tests as @DeltaIV suggests in the comments. Rather, a single convolution layer has simply too few parameters to memorize the output for the single input image that you are trying to segment.

Let's say the image has size 100$\times$100. To memorize its segmentation you would need 104 parameters. Now, a single convolution layer for binary segmentation would have probably 2 filters of shape 3$\times$3, including the bias that makes 20 parameters. So, unless the sample image segmentation is super trivial, overfitting cannot happen.

That said, I very much recommend reading the linked thread "What should I do when my neural network doesn't learn?" for some great tips for making your own neural network.

Jan Kukacka
  • 10,121
  • 1
  • 36
  • 62
  • This is a very good suggestion. However, if the kernel size is 1, the channels 2 and the strides 1, every pixel of the input is mapped to the pixel of the output. There should be enough parameters, right ? At the moment, the input image is 720x1280x3 wide. – Xema Sep 07 '18 at 08:51
  • Every pixel of the input is mapped to the pixel of the output - true. But they are all mapped using **the same two kernels**. Their parameters are shared between all locations. – Jan Kukacka Sep 07 '18 at 09:35
  • True! I realised that my conception of a convolutional layer was a bit off. So far I tried to add another conv layers of 512 kernels before the other (tried 1024 but got OOM). It seems to have improved the loss, but not enough to overfit (which seems quite normal given there are not enough filters). I think you put me of the right track, thank you ! – Xema Sep 07 '18 at 09:46
  • You are right, I didn't notice the OP specifically mentioned a **single** convolution layer (why on Earth would someone build a neural network with a single convolution layer?) – DeltaIV Sep 07 '18 at 09:51
  • @DeltaIV Perhaps taking out of context sentence "First, build a small network with a single hidden layer and verify that it works correctly." appearing in the linked debugging thread :) – Jan Kukacka Sep 07 '18 at 09:53
  • that's not in my answer in that thread :-) – DeltaIV Sep 07 '18 at 10:02
  • 1
    @DeltaIV I know, but it's even higher rated than your answer! (Which, btw, has the best last bullet point :D ) – Jan Kukacka Sep 07 '18 at 10:04