Generative Modeling

References

• Generative Model Wikipedia Article

Notes

In statistical classification, two main approaches are called the generative approach and the discriminative approach. These compute classifiers by different approaches, differing in the degree of statistical modelling. Terminology is inconsistent, but three major types can be distinguished:

1. A generative model is a statistical model of the joint probability distribution on a given observable variable and target variable ; A generative model can be used to generate random instances (outcomes) of an observation .
2. A discriminative model is a model of the conditional probability of the target , given an observation . It can be used to discriminate the value of the target variable , given an observation .
3. Classifiers computed without using a probability model are also referred to loosely as discriminative

The distinction between these two classes is not consistently made.

Standard examples of each, all of which are linear classifiers, are:

• generative classifiers
• • naive Bayes classifier
  • linear discriminant analysis
• discriminative model
• • logistic regression

In application to classification, one wishes to go from an observation to label (or probability distribution on labels). One can compute this directly, without using a probability distribution; one can estimate the probability of a label given an observation (discriminative model) and base classification on that, or one can estimate the joint distribution (generative model).

An alternative division defines these symmetrically as:

• a generative model is a model of the conditional probability of the observable , given a target , symbolically,
• a discriminative model is a model of the conditional probability of the target , given an observation , symbolically,

Regardless of precise definition, the terminology is constitutional because a generative model can be used to generate random instances (outcomes), either of an observation and target , or of an observation given a target value , while a discriminative model or discriminative classifier (without a model) can be used to discriminate the value of the target variable m given an observation .

The term generative model is also used to describe models that generate instances of output variables in a way that has no clear relationship to probability distributions over potential samples of input variables. Generative adversarial networks are examples of this class of generative models, and are judged primarily of particular outputs to potential inputs.