Revisiting Memory Efficient Kernel Approximation: An Indefinite Learning
  Perspective
  Matrix approximations are a key element in large-scale algebraic machine
learning approaches. The recently proposed method MEKA (Si et al., 2014)
effectively employs two common assumptions in Hilbert spaces: the low-rank
property of an inner product matrix obtained from a shift-invariant kernel
function and a data compactness hypothesis by means of an inherent
block-cluster structure. In this work, we extend MEKA to be applicable not only
for shift-invariant kernels but also for non-stationary kernels like polynomial
kernels and an extreme learning kernel. We also address in detail how to handle
non-positive semi-definite kernel functions within MEKA, either caused by the
approximation itself or by the intentional use of general kernel functions. We
present a Lanczos-based estimation of a spectrum shift to develop a stable
positive semi-definite MEKA approximation, also usable in classical convex
optimization frameworks. Furthermore, we support our findings with theoretical
considerations and a variety of experiments on synthetic and real-world data.
