Kernel-independent component analysis

In statistics, kernel-independent component analysis (kernel ICA) is an efficient algorithm for independent component analysis which estimates source components by optimizing a generalized variance contrast function, which is based on representations in a reproducing kernel Hilbert space. Those contrast functions use the notion of mutual information as a measure of statistical independence.

Main idea

Kernel ICA is based on the idea that correlations between two random variables can be represented in a reproducing kernel Hilbert space (RKHS), denoted by ${ℱ}$, associated with a feature map $L_x:{ℱ}\mapsto \mathbb{ℝ}$ defined for a fixed $x\in \mathbb{ℝ}$. The ${ℱ}$-correlation between two random variables $X$ and $Y$ is defined as

${\rho }_{{ℱ}}(X,Y)=\underset{f,g\in {ℱ}}{\max }\mathrm{corr}(⟨L_X,f⟩,⟨L_Y,g⟩)$

where the functions $f,g:\mathbb{ℝ}\to \mathbb{ℝ}$ range over ${ℱ}$ and

$\mathrm{corr}(⟨L_X,f⟩,⟨L_Y,g⟩):=\frac{\mathrm{cov}(f(X),g(Y))}{\mathrm{var}(f(X){)}^{1/2}\mathrm{var}(g(Y){)}^{1/2}}$

for fixed $f,g\in {ℱ}$. Note that the reproducing property implies that $f(x)=⟨L_x,f⟩$ for fixed $x\in \mathbb{ℝ}$ and $f\in {ℱ}$. It follows then that the ${ℱ}$-correlation between two independent random variables is zero. This notion of ${ℱ}$-correlations is used for defining contrast functions that are optimized in the Kernel ICA algorithm. Specifically, if $\mathbf{X}:=(x_{ij})\in {\mathbb{ℝ}}^{n\times m}$ is a prewhitened data matrix, that is, the sample mean of each column is zero and the sample covariance of the rows is the $m\times m$ dimensional identity matrix, Kernel ICA estimates a $m\times m$ dimensional orthogonal matrix $\mathbf{A}$ so as to minimize finite-sample ${ℱ}$-correlations between the columns of $\mathbf{S}:=\mathbf{X}{\mathbf{A}}^{\prime }$.