Learning Infinite-Layer Networks: Without the Kernel Trick
  Infinite--Layer Networks (ILN) have recently been proposed as an architecture
that mimics neural networks while enjoying some of the advantages of kernel
methods. ILN are networks that integrate over infinitely many nodes within a
single hidden layer. It has been demonstrated by several authors that the
problem of learning ILN can be reduced to the kernel trick, implying that
whenever a certain integral can be computed analytically they are efficiently
learnable.
  In this work we give an online algorithm for ILN, which avoids the kernel
trick assumption. More generally and of independent interest, we show that
kernel methods in general can be exploited even when the kernel cannot be
efficiently computed but can only be estimated via sampling.
  We provide a regret analysis for our algorithm, showing that it matches the
sample complexity of methods which have access to kernel values. Thus, our
method is the first to demonstrate that the kernel trick is not necessary as
such, and random features suffice to obtain comparable performance.
