Neural networks with trainable matrix activation functions
  The training process of neural networks usually optimize weights and bias
parameters of linear transformations, while nonlinear activation functions are
pre-specified and fixed. This work develops a systematic approach to
constructing matrix activation functions whose entries are generalized from
ReLU. The activation is based on matrix-vector multiplications using only
scalar multiplications and comparisons. The proposed activation functions
depend on parameters that are trained along with the weights and bias vectors.
Neural networks based on this approach are simple and efficient and are shown
to be robust in numerical experiments.
