Kernel Method
Also known as the Kernel trick, this is a concept used often in Machine Learning, so I want to read more about it here.
References
Notes
In Machine Learning, kernel machines are a class of algorithms for pattern analysis whose best known member is the support-vector-machine (SVM). These methods involve using linear classifiers to solve nonlinear problems. The general task of pattern analysis is to find and study general types of relations (clusters, rankings, principal components, correlations, classifications) in datasets. For many algorithms that solve these tasks, the data in raw representation have to be explicitly transformed into feature vector representations via a user-specified feature map: in contrast, kernel methods require only a user-specified kernel, i.e. a similarity function over all pairs of data points computed using inner products. The feature map in kernel machines is infinite dimensional but only requires a finite dimensional matrix from user-input according to the Representer theorem. Kernel machines are slow to compute for datasets larger than a couple thousand examples without parallel processing.
Kernel methods owe their name to the use of kernel functions, which enable them to operate in a high-dimensional, implicit feature space without ever computing the coordinates of the data in that space, but rather by simply computing the inner products between the images of all pairs of data in the feature space. This operation is often computationally cheaper than the explicit computation of the coordinates, This approach is called the kernel trick
.
Algorithms capable of operating with kernels:
- kernel perceptron
- support vector machines
- Gaussian processes
- Principal component analysis
- Canonical Correlation analysis
- Ridge Regression
- Spectral clustering
- linear adaptive filters
Motivation and Informal Explanation
- Kernel methods can be thought of as instance-based learners: rather than learning some fixed set of parameters corresponding to the features of their inputs, they instead
remember
the -th training example and learn for it a corresponding weight . - Prediction for unlabeled inputs - those not in the training set - is treated by the application of a similarity function , called the kernel, between the unlabeled input and each of the training inputs .
- Kernel classifiers were described as early as the 1960s, with the invention of kernel perceptron. They rose to great prominence with the popularity of the support vector machine in the1990s, when the SVM was found to be competitive with neural networks on tasks such as MNIST.
Mathematics: the kernel trick
The kernel trick avoids the explicit mapping that is needed to get linear learning algorithms to learn a nonlinear function or decision boundary. For all and in the input space , certain functions can be expressed as an inner product in another space . The function us often referred to as a kernel or a kernel function. The word "kernel" is used in mathematics to denote a weighing function for a weighted sum or integral.