A Random Matrix Perspective on Mixtures of Nonlinearities for Deep
  Learning
  One of the distinguishing characteristics of modern deep learning systems is
that they typically employ neural network architectures that utilize enormous
numbers of parameters, often in the millions and sometimes even in the
billions. While this paradigm has inspired significant research on the
properties of large networks, relatively little work has been devoted to the
fact that these networks are often used to model large complex datasets, which
may themselves contain millions or even billions of constraints. In this work,
we focus on this high-dimensional regime in which both the dataset size and the
number of features tend to infinity. We analyze the performance of random
feature regression with features $F=f(WX+B)$ for a random weight matrix $W$ and
random bias vector $B$, obtaining exact formulae for the asymptotic training
and test errors for data generated by a linear teacher model. The role of the
bias can be understood as parameterizing a distribution over activation
functions, and our analysis directly generalizes to such distributions, even
those not expressible with a traditional additive bias. Intriguingly, we find
that a mixture of nonlinearities can improve both the training and test errors
over the best single nonlinearity, suggesting that mixtures of nonlinearities
might be useful for approximate kernel methods or neural network architecture
design.
