Classification

References

• Statistical Classification Wikipedia Article

Notes

Statistical Classification: When classification is performed by a computer, statistical methods are normally used to develop the algorithm.

• Classification is the activity of assigning objects to some pre-existing classes or categories. This is distinct from the task of establishing the classes themselves.

Often, the individual observations are analyzed into a set of quantifiable properties, known as explanatory variables or features. These properties may be categorical, ordinal, integer-valued, or real-valued. Other classifiers work by comparing observations to previous observations by means of similarity or distance function.

An algorithm that implements classification is known as a classifier.

Classification and clustering are examples of the more general problem of pattern recognition, which is the assignment of some sort of output value to a given input value. A common subclass of classification is probabilistic classification. Algorithms of this nature use statistical inference to find the best class for a given instance. Unlike other algorithms, which simply output the best class, probabilistic algorithms output a probability of the instance being a member of each of the possible classes. Such an algorithm has numerous advantages over non-probabilistic classifiers:

• It can output a confidence value associated with its choice (in general, a classifier that can do this is known as a confidence-weighted classifier).
• Correspondingly, it can abstain when its confidence of choosing any particular output is too low
• Because of the probabilities are generated, probabilistic classifiers can be more effectively incorporated into larger machine-learning tasks, in a way that partially or completely avoids the problem of error propagation

Classification can be thought of as two separate problems - binary classification and multiclass classification. In binary classification, a better understood task, only two classes are involved, whereas multiclass classification involves assigning one object to several classes. Multiclass classification often requires the combined use of multiple binary classifiers since many classification methods were developed specifically for binary classification,

Most algorithms describe an individual instance whose category is to be predicted using a feature vector of individual, measurable properties of the instance. Each property is termed a feature, also known in statistics as an explanatory variable (or independent variable, although features may or may not be statistically independent).

A large number of algorithms for classification can be phrased in terms of a linear function that assigns a score to each possible category by combining the feature vector of an instance with a vector of weights, using a dot product. The predicted category is the one with the highest score. This type of score function is known as a linear predictor function and has he following general form:

where is a feature vector for instance , is the vector of weights corresponding to category , and is the score associated with assigning instance to category . In discrete choice theory, where instances represent people and categories represent choices, the score is considered the utility associated with person and choosing category . Algorithms with this basic setup are known as linear classifiers. What distinguished them is the procedure for determining (training) the optimal weights/coefficients and the wat that the score is interpreted.