From Hard to Soft: Understanding Deep Network Nonlinearities via Vector
  Quantization and Statistical Inference
  Nonlinearity is crucial to the performance of a deep (neural) network (DN).
To date there has been little progress understanding the menagerie of available
nonlinearities, but recently progress has been made on understanding the r\^ole
played by piecewise affine and convex nonlinearities like the ReLU and absolute
value activation functions and max-pooling. In particular, DN layers
constructed from these operations can be interpreted as {\em max-affine spline
operators} (MASOs) that have an elegant link to vector quantization (VQ) and
$K$-means. While this is good theoretical progress, the entire MASO approach is
predicated on the requirement that the nonlinearities be piecewise affine and
convex, which precludes important activation functions like the sigmoid,
hyperbolic tangent, and softmax. {\em This paper extends the MASO framework to
these and an infinitely large class of new nonlinearities by linking
deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs).} We show
that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute
value, and max-pooling can be interpreted as solutions to certain natural
"hard" VQ inference problems, while sigmoid, hyperbolic tangent, and softmax
can be interpreted as solutions to corresponding "soft" VQ inference problems.
We further extend the framework by hybridizing the hard and soft VQ
optimizations to create a $\beta$-VQ inference that interpolates between hard,
soft, and linear VQ inference. A prime example of a $\beta$-VQ DN nonlinearity
is the {\em swish} nonlinearity, which offers state-of-the-art performance in a
range of computer vision tasks but was developed ad hoc by experimentation.
Finally, we validate with experiments an important assertion of our theory,
namely that DN performance can be significantly improved by enforcing
orthogonality in its linear filters.
