Learning Bounds for Greedy Approximation with Explicit Feature Maps from
  Multiple Kernels
  Nonlinear kernels can be approximated using finite-dimensional feature maps
for efficient risk minimization. Due to the inherent trade-off between the
dimension of the (mapped) feature space and the approximation accuracy, the key
problem is to identify promising (explicit) features leading to a satisfactory
out-of-sample performance. In this work, we tackle this problem by efficiently
choosing such features from multiple kernels in a greedy fashion. Our method
sequentially selects these explicit features from a set of candidate features
using a correlation metric. We establish an out-of-sample error bound capturing
the trade-off between the error in terms of explicit features (approximation
error) and the error due to spectral properties of the best model in the
Hilbert space associated to the combined kernel (spectral error). The result
verifies that when the (best) underlying data model is sparse enough, i.e., the
spectral error is negligible, one can control the test error with a small
number of explicit features, that can scale poly-logarithmically with data. Our
empirical results show that given a fixed number of explicit features, the
method can achieve a lower test error with a smaller time cost, compared to the
state-of-the-art in data-dependent random features.
