Merging Two Cultures: Deep and Statistical Learning
  Merging the two cultures of deep and statistical learning provides insights
into structured high-dimensional data. Traditional statistical modeling is
still a dominant strategy for structured tabular data. Deep learning can be
viewed through the lens of generalized linear models (GLMs) with composite link
functions. Sufficient dimensionality reduction (SDR) and sparsity performs
nonlinear feature engineering. We show that prediction, interpolation and
uncertainty quantification can be achieved using probabilistic methods at the
output layer of the model. Thus a general framework for machine learning arises
that first generates nonlinear features (a.k.a factors) via sparse
regularization and stochastic gradient optimisation and second uses a
stochastic output layer for predictive uncertainty. Rather than using shallow
additive architectures as in many statistical models, deep learning uses layers
of semi affine input transformations to provide a predictive rule. Applying
these layers of transformations leads to a set of attributes (a.k.a features)
to which predictive statistical methods can be applied. Thus we achieve the
best of both worlds: scalability and fast predictive rule construction together
with uncertainty quantification. Sparse regularisation with un-supervised or
supervised learning finds the features. We clarify the duality between shallow
and wide models such as PCA, PPR, RRR and deep but skinny architectures such as
autoencoders, MLPs, CNN, and LSTM. The connection with data transformations is
of practical importance for finding good network architectures. By
incorporating probabilistic components at the output level we allow for
predictive uncertainty. For interpolation we use deep Gaussian process and ReLU
trees for classification. We provide applications to regression, classification
and interpolation. Finally, we conclude with directions for future research.
