5

Suppose a model M classifies apples and oranges. Can M be extended to classify a third class of objects, e.g., pears, such that the new images for 'retraining' only have pears annotated and apples and oranges ignored? That is, since M already classifies apples and oranges, can the old weights be somehow preserved and let the retraining focus specifically on learning about pears?

Methods such as fine-tuning and learning without forgetting seem to require all objects in the new images annotated though.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
John M.
  • 161
  • 1
  • In this case, do you only have examples of the new object? – arinarmo Apr 04 '19 at 18:08
  • @arinarmo No. The new images may contain all 3 objects. – John M. Apr 04 '19 at 18:13
  • I believe you can use transfer learning if that's the case. – arinarmo Apr 04 '19 at 18:15
  • @arinarmo But like you suggested, transfer learning requires every class in the new task annotated. I'm trying to not do that, but just have the new class annotated. – John M. Apr 04 '19 at 18:33
  • I don't understand your situation, why wouldn't you get the new set of images labeled? Can you re-write your question to make it more clear what your trying to do? Maybe add some equations? – Charlie Parker May 26 '19 at 02:58
  • crossposted: https://www.reddit.com/r/deeplearning/comments/btng4t/how_does_one_extend_a_neural_network_to_classify/ also: https://www.quora.com/unanswered/How-does-one-extend-a-neural-network-to-classify-new-objects-not-in-the-training-data?__filter__=&__nsrc__=2&__snid3__=4512097721 – Charlie Parker May 27 '19 at 15:41
  • @CharlieParker The issue is that the new set of images may not necessarily contain the same labels as the old. You may have only oranges labelled in one and only pears labelled in another. – John M. May 27 '19 at 15:56
  • @JohnM. so you are not worried about a case were an entirely new class is introduced to the data set and the old one remains in the data set...that's NOT your scenario? – Charlie Parker May 28 '19 at 02:49
  • @CharlieParker That is actually the scenario. The old dataset may contain pears but not labelled whereas the new dataset may contain oranges but not labelled. Hope that explains it better. – John M. May 28 '19 at 08:07
  • @JohnM. why would the new data set not contain labels for oranges? :/ I am thinking of a case where an intelligent system starts to learn more and more concepts. – Charlie Parker May 28 '19 at 22:42
  • @CharlieParker Why must it? It's often the case that a pretrained model is fine-tuned with a completely different dataset. Say, a model pretrained with Imagenet is fine-tuned with some arbitrary dataset. – John M. May 29 '19 at 12:10

3 Answers3

3

I don't know of any method to do exactly what you're asking, but in general you may want to look at Transfer Learning. In deep learning. this technique consists of loading a network trained to predict images on a large dataset such as ImageNet and replace the last, fully connected layer with your own, then train on your images. Examples of such networks are AlexNet or VGGNet

In this way, the features extracted by the convolutional layers will be preserved, but you can use those features to predict a different task.

Here's a practical tutorial that explains the specifics for pytorch, and you can read more about it here

arinarmo
  • 356
  • 2
  • 5
2

This general area is called "continual", "incremental" or "life-long" learning, and it's quite an active area of research.

There are many approaches to continual learning, including different forms of regularization which penalize forgetting, dynamically expanding architectures, and explicit model memories. For more detail, I would refer you to this survey paper: Continual Lifelong Learning with Neural Networks: A Review

Methods such as fine-tuning and learning without forgetting seem to require all objects in the new images annotated though.

Well, that is not always the case. Let me give an explicit continual learning approach to make this clear: you can train a classifier with a binary output for whether the image has pear or not, and apple or not. You can add a new binary output for whether it has orange or not. Then when you start training on your orange dataset, impose a regularization loss which penalizes the pear and apple predictions from changing much (freeze a copy of your model from before you started training on oranges, and penalize the MSE or cross-entropy between your old and new pear and apple predictions).

shimao
  • 22,706
  • 2
  • 42
  • 81
1

EDIT: now that I understand your question a little better. I don't have enough rep to comment so using responses.

So the labels on the dataset you have are binary, pear = 1/0. And the no class contains oranges and bananas which you already have a model for.

What if you did a two-step approach where you classify an object as being a pear or not, and then you apply the frozen pretrained orange/banana model to images not classified as a pear? This would get you a prediction on each object as pear/banana/orange and you would be preserving the original model weights because it's frozen.

Leo Brueggeman
  • 171
  • 4