Fast Estimation of Information Theoretic Learning Descriptors using
  Explicit Inner Product Spaces
  Kernel methods form a theoretically-grounded, powerful and versatile
framework to solve nonlinear problems in signal processing and machine
learning. The standard approach relies on the \emph{kernel trick} to perform
pairwise evaluations of a kernel function, leading to scalability issues for
large datasets due to its linear and superlinear growth with respect to the
training data. Recently, we proposed \emph{no-trick} (NT) kernel adaptive
filtering (KAF) that leverages explicit feature space mappings using
data-independent basis with constant complexity. The inner product defined by
the feature mapping corresponds to a positive-definite finite-rank kernel that
induces a finite-dimensional reproducing kernel Hilbert space (RKHS).
Information theoretic learning (ITL) is a framework where information theory
descriptors based on non-parametric estimator of Renyi entropy replace
conventional second-order statistics for the design of adaptive systems. An
RKHS for ITL defined on a space of probability density functions simplifies
statistical inference for supervised or unsupervised learning. ITL criteria
take into account the higher-order statistical behavior of the systems and
signals as desired. However, this comes at a cost of increased computational
complexity. In this paper, we extend the NT kernel concept to ITL for improved
information extraction from the signal without compromising scalability.
Specifically, we focus on a family of fast, scalable, and accurate estimators
for ITL using explicit inner product space (EIPS) kernels. We demonstrate the
superior performance of EIPS-ITL estimators and combined NT-KAF using EIPS-ITL
cost functions through experiments.
