Linking generative, discriminative models to supervised and unsupervised learning

Question

Definitions that I am considering: A generative model learns p(x,y) whereas a discriminative model learns p(y|x=x).

I would like to verify if my understanding is correct by sharing the following claims. Please let me know if my claims are right or wrong (do provide counter-examples).

Claim 1: All generative models are learnt using unsupervised learning.

Claim 2: Not every unsupervised learning algorithm is a generative model. For example, clustering algorithm is not a generative model.

Claim 3: All discriminative models are learnt using supervised learning.

Claim 4: Every supervised learning algorithm is a discriminative model.

Answer 1

A note about the generative-discriminative dichotomy

Although your formal generative-vs-discriminative definition is widely used and very useful [1], I agree with @shimao that it's somewhat of a false dichotomy [2] which starts falling apart when you consider that it's in fact a spectrum which you can even smoothly interpolate between [3], and many popular models do not neatly fall into this dichotomy [4]. Furthermore, it assumes the supervised learning setting and is less useful for classifying unsupervised or semi-supervised methods.

A more general (but more informal) definition [5, 6] to complement it might be:

• Discriminative models learn the boundary between classes
• Generative models learn the distribution of data

With that in mind...

Addressing the claims

TL;DR There is no blanket claim that can be made linking generative/discriminative models and unsupervised/supervised methods, since every combination of the two exists.

|              | Generative             | Discriminative                                |
|--------------+------------------------+-----------------------------------------------|
| Supervised   | GANs, Naive Bayes, [7] | SVM, logistic regression,deep neural networks |
| Unsupervised | LDA, normalizing flows | monocular depth and optical flow models, [9]  |

Claim 1: False. Many generative models---from the basic like Naive Bayes to complex like a GAN or this generative up-convolutional network [7]---are learnt in a supervised or semi-supervised fashion.

Claim 2.1: True. Unsupervised learning algorithms fall all across the generative-discriminative spectrum, e.g. k-nearest neighbours (kNN) is somewhere in between generative and discriminative [8] (formally generative, informally discriminative).

Claim 2.2: False. Many clustering algorithms are generative models, e.g. latent Dirichlet allocation (LDA).

Claim 3: False. There are discriminative models trained on unsupervised datasets, e.g. [9], and as @shimao pointed out, monocular depth and optical flow models.

Claim 4: False. In disproving Claim 1 we showed some examples of supervised learning algorithms with generative models (Naive Bayes, [7]). Also your phrasing is a bit strange since algorithm != model. "GANs are generative models that use supervised learning to approximate an intractable cost function" [10]. GANs, like VAEs, have both generative and discriminative components, but are commonly referred to as generative.

---

Sources

1. Possibly originating from "Machine Learning - Discriminative and Generative" (Tony Jebara, 2004).
2. The Generative-Discriminative Fallacy
3. "Principled Hybrids of Generative and Discriminative Models" (Lasserre et al., 2006)
4. @shimao's question
5. benhammer's answer
6. raegtin's answer
7. "Learning to Generate Chairs, Tables and Cars with Convolutional Networks" (Dosovitskiy et al., 2017) trains a generative upsampling CNN in a supervised fashion.
8. Binu Jasim's answer
9. "Discriminative Unsupervised Feature Learning with Convolutional Neural Networks" (Dosovitskiy et al., 2014)
10. Goodfellow, Ian (2016). NIPS 2016 GAN tutorial.

Answer 2

Well first I would nitpick your definition. Consider a conditional GAN which given a category $x$ generates a realistic image $y$ in category $x$ -- that is, it models $P(y|x)$. This is arguably not a discriminative model, but it is according to your definition.

One counterexample to claim 1 would be a model like Neural Scene Derendering. It is a generative model with a highly structured latent space, which can only be learned with supervision (because learning structured latent representations is in general a very hard problem).

A counterexample to 3 is that monocular depth and optical flow models can be learned in an unsupervised manner by exploiting geometric and temporal consistencies. (Yet these are discriminative regression models)

Assuming all models can be dichotomized into generative and discriminative and supervised/unsupervised, Claim 4 is the contrapositive of Claim 1. For an explicit counterexample, consider a GAN conditioned on image category -- this would require labeled data to train, yet is generative.