1

I've read here that generative models have less degrees of freedom than discriminant ones, so they are more robust and less prone to overfitting. I would like to understand this statement with a simple example.

Suppose I want to predict a binary Y with a binary X.

With a discriminant model, I predict P(Y|X) directly and the model has 2 degrees of freedom: once P(Y=0|X=0) and P(Y=0|X=1) are estimated, the other two probabilities P(Y=1|X=0) and P(Y=1|X=1) are determined.

With a generative model, I have to estimate P(X|Y) and P(Y). It appears to me that the model has three degrees of freedom though: for example once I have estimated P(X=0|Y=0),P(X=1|Y=0) and P(Y=0), all the other parameters (P(X=0|Y=1),P(X=1|Y=1) and P(Y=1)) are determined.

Am I wrong? Or is the statement about generative models having less degrees of freedom than discriminant ones false?

Patrick
  • 393
  • 1
  • 9

1 Answers1

1

I think you have to compare models with the same capacity. If we continue with your example of a model where the parameters just describe probabilities, suppose we have 3 parameters in each case. Given a certain set of inputs, for example, suppose you have (X=1,Y=1) and (X=0,Y=0), then you can model P(X=0|Y=0) = 1 and P(X=1|Y=0) = 0 and perfectly fit the data. In the generative model case, you also need P(Y=0) = $\frac 12$ (suppose that this is actually the true parameter). However, the discriminator doesn't need to model P(Y=0), so you could imagine that any number for P(Y=0) works.

The analogy maybe becomes more useful as you think about a neural network with a fixed number of units/layers/weights. The generative task constrains the weights of the network more, as not only does it have to classify properly, it also has to model the probability of the data. In the space of all possible weights, there are fewer sets of weights that will properly accomplish the generative task - thus there are fewer degrees of freedom. And this tighter set of constraints is kind of like regularization, so perhaps will also help with generalizing/avoiding overfitting.

That's my interpretation anyway. I'm still learning, so could be wrong somewhere, but hopefully this helps.

Silent Zebra
  • 26
  • 2