Hands On Machine Learning Chapter 3 - Classification

We now turn our attention to classification systems.

In this chapter, we will be using the MNIST dataset, which is a set of 70,000 small images of digits handwritten by high school students and employees of the US Census Bureau. Each image is laveled by the figit it reprents.

Scikit-Learn provides many helper functions to download popular datasets. MNIST is one of them. Datasets loaded by Scikit-Learn generally have a similar dictionary structure including:

• A DESC key describing the dataset
• A data key containung an array with one row per instance and one column per feature
• A target key containing an array with the labels

There are 70,000 images, and each image has 784 features. This is because each image is 28x28 pixels, and each feature simply represents one pixel's intensity, from 0 (white) to 255 (black).

A binary classifier can distinguish between two classes. A good place to start is with a Stochastic Gradient Desent classifier, using Scikit-Learn's SGDClassifier class. The SGDClassifier relies on randomness during training. If you want reproducible results, you should set the random_state parameter.

Evaluating a classifier is significantly trickier than evaluating a regressor, so we will spend a large part of this chapter on this topic.
If you look at the code below, the classifier that we trained with SGDClassifier had an average accuracy of approximately 96% while the classifier that we trained to trained to always return False had an accuracy of 90%. (This is for the binary classifier testing whether a digit represents a '5' or not. Also, we were testing the performance of the classification using 3-fold cross-validation with scoring="accuracy"). This demonstrates why accuracy is generally not the preferred performance measure for classifiers, especially when you are dealing with skewed datasets (when some classes are much more frequent than others.)

A much better way to evaluatre the performance of a classifier is to look at the confusion matrix. The general idea is to count the number of times instances of class A are classified as class B. [...] To predict the confusion matrix, you can use the cross_val_predict() function.

The cross_val_predict() function works like cross_val_score() in that it performs K-fold cross validation, but instead of returning teh evaluation scores, it returns the predictions made on each test fold. This means that you get a clean prediction for each instance in the training set ("clean" means that the prediction is made by a model that never saw the data during training).

Each row in a confusion matrix represents an actual class, while each column represents a predicted class. A perfect classifier would have only true positives and true negatives, so its cvonfusion matrix would have nonzero values only on its main diagonal. The confusion matrix gives you a lot of information, but sometimes you may prefer a more concise metric. An interesting one to look at is the accuracy of the positive predictions; this is called the precision of the classifier:

Precision is trypically used along with another metric named recall, also called sensitivity or true positive rate (TPR): this is the ratio of positive instances that are correctly detected by the classifier:

Scikit-Learn provides several functions to compute classifier matrix, including precision and recall (with precision_score and recall_score). It is often convenient to combine the precision and recall into a single metric called the $F_{1}F\_1$F1​ score, in particular if you need a simple way to compare two classifiers. The $F_{1}F\_1$F1​ score is the harmonic mean of precision and recall. The Harmonic mean gives much more weight to low values.As a result, the classifier will only get a high $F_{1}F\_1$F1​ score if both recall and precision are high. To compute the $F_{1}F\_1$F1​ score, simply call the f1_score() function. The $F_{1}F\_1$F1​ score favors classifiers that have similar precision and recall. This is not always what you want: in some contexts you mostly care about precision and in other contexts what you really care about is recall. Increasing precision reduces recall, and vice versa. This is called the precision/recall tradeoff.

The SGDClassifier computes a score based on the decision function. If the score is greater than a threshold, it assigns the instance to the positive class, else it assigns it to a negative class. Scikit-Learn gives you access to the decsions scores that it uses to make predictions with the decision_function() method, which returns a score for each instance. You can then make predictions based on those scores using any threshold you want.

The reason [that the precision curve is bumpier than the recall curve] is that precision may sometimes go down when you raise the threshold. On the other hand, recall can only go down when the threshold is increased.

The ROC Curve

The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers. It is very similar to the precision/recall curve, but instead of plotting precision vs recall, the ROC curve plots the true positive rate (another name for recall) against the false positive rate. The FPR is the ratio of negative instances that are incorrectly classified as positive. It is equal to one minus the true negative rate, which is the ratio of negative instances that are correctly classified as negative. The TNR is also called specificity. Hence the ROC curve plots sensitivity (recall) versus 1-specificity. You can get the TPR and FPR for various threshold values using the roc_curve() function.

Once again there is a tradeoff: the higher the recall (TPR), the more false positives (FPR) the classifier produces. The dotted line represents the ROC curve of a purely random classifier: A good classifier stays as far away from that line as possible.

One way to compare classifiers is to measure the area under the curve (AUC). A perfect classifier will have a ROC AUC equal to 1, whereas a purely random classifier will have a ROC AUC equal to 1. As a rule of thumb, you should prefer the Precision-Recall curve whenever the positive class is rare or when you care more about the false positives than the false negatives, and the ROC curve otherwise. The RandomForestClassifier class does not have a decision_function() method. Instead, it has a predict_proba() method. Scikit-Learn classifiers generally have one or the other. The predict_proba() method returns an array containing a row per instance and a column per class, each containing the probability that the given instance belongs to the given class.

As you can see below, the RandomForestClassifier's ROC curve looks much better than the SGDClassifiers.

Whereas binary classifiers distinguish between two classes, multiclass classifiers (also called multinomial classifiers) can distinguish between more than two classes. Some algorithms (such as Random Forest Classifiers or naive Bayes classifiers) are capable of handling multiple classes directly. Others (such as Support vector Machine classifiers or Linear classifiers) are strictly binary classifiers. There are various strategies that you can use to perform multiclass classification using multiple binary classifiers:

1. One-versus-all (OvA) or One-versus-the-rest: One way to create a system that can classify the digit images into 10 classes is to train 10 binary classifiers, one for each digit.l Then, you want to classify an image, you get the decision score from each classifier and ou select the class whose classifier outputs the highest score.
2. One-versus-one (OvO): Train a binary classifier for every pair of digits: one to distinguish 0s and 1s, another to distinguish 1s and 2s, and so on. If there are $NN$N classes, you need to train $N\times (N-1)\mathrm{/}2N\; \backslash times\; (N\; -\; 1)\; /2$N×(N−1)/2 classifiers.

Some algorithms, such as Support Vector Machine classifiers, scale poorly with the size of the training set, so for these algorithms OvO is preferred since it is faster to train many classifiers on small training sets than training very few classifiers on large training sets. For most binary classification algorithms. however, OvA is preferred.

Scikit-Learn detects when you try to use a binary classification algorithm for a multiclass classification task, and it automatically runs OvA (except for SVM classifiers which it uses OvO).

If you want to force Scikit-Learn to use one-versus-one or ove-versus-all, you can use the OneVsOneClassifier or OneVsRestClassifier classes. Scikit-Learn does not have to run OvA or OvO when using RandomForestClassifiers because Random Forest Classifiers can directly classify instances into multiple classes.

One way to tune classification models is to analyze the types of errors that it makes. The confusion matrix is helpful for this. It is often convenient to look at an image representation of the confusion matrix, using Matplotlib's matshow() function. The confusion matrix below looks fairly good, since most images are on the main diagonal, which means they were classified correctly. The 5s look slightly darker than the other digits, which could mean that there are fewer images of 5s in the dataset or that the classifier does not perform as well on 5s as on other digits. Comparing the rates of errors instead of the total number of error is also a good idea. Keep in mind that rows represent teh actual classes, while columns represent the predicted classes. Analyzing the confusion matrix often gives you insights on ways to improve your classifier. Analyzing individual errors can also be a good way to gain insights on what your classifier is doing and why it is failing, but it is more difficult and time-consuming.

The ConfusionMatrixDisplay from sklearn.metrics is worth looking into for plotting confusion matrices.

In some cases you may want your classifier to output multiple classes for each instance. Such a classification system that outputs multiple binary tags is called a multilabel classification system. There are many ways to evaluate a multilabel classifier, and selecting the right metric really depends on your project.

The last type of classification task we are going to discuss is called the multioutput-multiclass classification (or simply (multioutput classification*)). It is simply a generalization of multilabel classification where each label can be multiclass (it can have more than two possible values).