Enhanced Convolutional Neural Tangent Kernels
  Recent research shows that for training with $\ell_2$ loss, convolutional
neural networks (CNNs) whose width (number of channels in convolutional layers)
goes to infinity correspond to regression with respect to the CNN Gaussian
Process kernel (CNN-GP) if only the last layer is trained, and correspond to
regression with respect to the Convolutional Neural Tangent Kernel (CNTK) if
all layers are trained. An exact algorithm to compute CNTK (Arora et al., 2019)
yielded the finding that classification accuracy of CNTK on CIFAR-10 is within
6-7% of that of that of the corresponding CNN architecture (best figure being
around 78%) which is interesting performance for a fixed kernel. Here we show
how to significantly enhance the performance of these kernels using two ideas.
(1) Modifying the kernel using a new operation called Local Average Pooling
(LAP) which preserves efficient computability of the kernel and inherits the
spirit of standard data augmentation using pixel shifts. Earlier papers were
unable to incorporate naive data augmentation because of the quadratic training
cost of kernel regression. This idea is inspired by Global Average Pooling
(GAP), which we show for CNN-GP and CNTK is equivalent to full translation data
augmentation. (2) Representing the input image using a pre-processing technique
proposed by Coates et al. (2011), which uses a single convolutional layer
composed of random image patches. On CIFAR-10, the resulting kernel, CNN-GP
with LAP and horizontal flip data augmentation, achieves 89% accuracy, matching
the performance of AlexNet (Krizhevsky et al., 2012). Note that this is the
best such result we know of for a classifier that is not a trained neural
network. Similar improvements are obtained for Fashion-MNIST.
