Kernel Principal Component Analysis

Looking for more research papers to read, I scanned my Hands-On Machine Learning notes for the many papers that were referenced there. This is one of those papers. These papers are mainly on machine learning and deep learning topics.

Reference Kernel Principal Component Analysis Paper

DOWNLOAD TEX

Date Created: 25 10, 2025

Last Edited: 54 22, 2025

1 38

Introduction

A new method for performing a nonlinear form of Principal Component Analysis is proposed. By the use of integral operator kernel functions, one can efficiently compute principal components in high-dimensional feature spaces, related to input space by some nonlinear map; for instance the space of all possible $d$ -pixel products in images. This paper gives the derivation of the method of presents experimental results on polynomial feature extraction for pattern recognition.

Assume for the moment that our data mapped into feature space, $ϕ (x_{1}), \dots, ϕ (x_{ℓ})$ , is centered, i.e. $\sum_{k = 1}^{ℓ} ϕ (x_{ℓ}) = 0$ . To do PCA for the covariance matrix

\overset{�}{C} = \frac{1}{ℓ} \sum_{j = 1}^{ℓ} ϕ (x_{j}) ϕ {(x_{j})}^{T}

we have to find Eigenvalues $λ \geq 0$ and Eigenvectors $V \in F {}$ satisfying $λ X = \overset{�}{C} V$ . All solutions $V$ lie in the span of $ϕ (x_{1}), \dots, ϕ (x_{ℓ})$ . This implies that we may consider the equivalent system

λ (ϕ (x_{k}) \cdot V) = (ϕ (x_{k}) \cdot \overset{�}{C} V) for all k = 1, \dots, ℓ,

and that there exists equivalent coefficients $α_{1}, \dots, α_{ℓ}$ such that

V = \sum_{i = 1}^{ℓ} α_{i} ϕ (x_{i})

Substituting and defining an $ℓ \times ℓ$ matrix $K$ by

K_{ij} : = (ϕ (x_{i}) \dots ϕ (x_{j}))

we arrive at

ℓλKα = K^{2} α

where $α$ denotes the column vector with entries $α_{1}, \dots, α_{ℓ}$ . To find solutions to the above, we solve the Eigenvalue problem

ℓλα = Kα

for nonzero Eigenvalues. We normalize the solutions $α^{k}$ belonging to nonzero Eigenvalues by requiring that the corresponding vectors in $F$ be normalized, i.e. $(V^{k} \cdot V^{k}) = 1$ . This translates to:

1 = \sum_{i, j = 1}^{ℓ} α_{i}^{k} α_{j}^{k} (ϕ (x_{i}) ϕ (x_{j})) = (α^{k} \cdot K α^{k}) = λ_{k} (α^{k} \cdot α^{k})

For princupal component extraction, we compute projections of the image of a test point $ϕ (x)$ onto the Eigenvectors $V^{k}$ in $F$ according to

(V^{k} \cdot ϕ (x) = \sum_{i = 1}^{ℓ} α_{i = 1}^{k} (ϕ (x_{i}) \cdot ϕ (x))

Kernels that have been successfully used in support vector machines include polynomial kernels, radial basis functions, and sigmoid kernels.

Kernel Principal Component Analysis

Introduction

Comments

User Comments

Inserting Custom HTML:

Basics:

Inserting Code Expressions:

Inserting $\Large{LaTeX}$ Expressions:

SVG Icon Search

AI Code Generation

The Sanitization of HTML