A Distributionally Robust Optimization Method for Adversarial Multiple
  Kernel Learning
  We propose a novel data-driven method to learn a mixture of multiple kernels
with random features that is certifiabaly robust against adverserial inputs.
Specifically, we consider a distributionally robust optimization of the
kernel-target alignment with respect to the distribution of training samples
over a distributional ball defined by the Kullback-Leibler (KL) divergence. The
distributionally robust optimization problem can be recast as a min-max
optimization whose objective function includes a log-sum term. We develop a
mini-batch biased stochastic primal-dual proximal method to solve the min-max
optimization. To debias the minibatch algorithm, we use the Gumbel perturbation
technique to estimate the log-sum term. We establish theoretical guarantees for
the performance of the proposed multiple kernel learning method. In particular,
we prove the consistency, asymptotic normality, stochastic equicontinuity, and
the minimax rate of the empirical estimators. In addition, based on the notion
of Rademacher and Gaussian complexities, we establish distributionally robust
generalization bounds that are tighter than previous known bounds. More
specifically, we leverage matrix concentration inequalities to establish
distributionally robust generalization bounds. We validate our kernel learning
approach for classification with the kernel SVMs on synthetic dataset generated
by sampling multvariate Gaussian distributions with differernt variance
structures. We also apply our kernel learning approach to the MNIST data-set
and evaluate its robustness to perturbation of input images under different
adversarial models. More specifically, we examine the robustness of the
proposed kernel model selection technique against FGSM, PGM, C\&W, and DDN
adversarial perturbations, and compare its performance with alternative
state-of-the-art multiple kernel learning paradigms.
