Evaluating a discriminative model is relatively easy: compare the predictions with ground truth, using cross-validation.
Unfortunately this strategy can't be used for generative models. Surely this problem has been tackled already?
Asked
Active
Viewed 2,439 times
6
-
1You can look at the likelihood of a test sample under the generative model. – Arthur B. Nov 13 '14 at 15:31
-
@ArthurB.Could you detail a little bit? How do you go from the likelihood of a sample to a measure of goodness for the model? – static_rtti Nov 14 '14 at 09:01
-
1The likelihood of a sample is a measure of goodness of the model, it's the best measure there is. – Arthur B. Nov 14 '14 at 14:58
1 Answers
8
Discriminative algorithms model P(Class|variables), whereas generative algorithms model P(Class,variables) = P(Class|variables)* P(variables). Hence, by modelling the joint distribution of the variable space, generative algorithms model the underlying process that 'created' your data.
My point in starting with this first paragraph is to note that generative algorithms have discriminative properties. Therefore, the same method of evaluating the predictive performance:
"compare the predictions with ground truth, using cross-validation."
applies to generative models, as well as discriminative ones.
However, as you imply, we can additionally asses the ability of the generative algorithms in modelling the underlying process that generates data. A commonly used group of metrics for this is "information theoretic scores" that derive from the idea of likelihood (log-likelihood). Below are some well-known information theoretic scores:
1- log-likelihood (LL) score
2- minimum description length (MDL) score
3- minimum message length (MML) score
4- Akaike Information Criterion (AIC) score
5- Bayesian Information Criterion (BIC) score
Note that 2, 3, 4, and 5 use some complexity penalisation factor over the LL score. This is good practice to combat over-fitting.
