On Expressivity and Trainability of Quadratic Networks
  Inspired by the diversity of biological neurons, quadratic artificial neurons
can play an important role in deep learning models. The type of quadratic
neurons of our interest replaces the inner-product operation in the
conventional neuron with a quadratic function. Despite promising results so far
achieved by networks of quadratic neurons, there are important issues not well
addressed. Theoretically, the superior expressivity of a quadratic network over
either a conventional network or a conventional network via quadratic
activation is not fully elucidated, which makes the use of quadratic networks
not well grounded. Practically, although a quadratic network can be trained via
generic backpropagation, it can be subject to a higher risk of collapse than
the conventional counterpart. To address these issues, we first apply the
spline theory and a measure from algebraic geometry to give two theorems that
demonstrate better model expressivity of a quadratic network than the
conventional counterpart with or without quadratic activation. Then, we propose
an effective and efficient training strategy referred to as ReLinear to
stabilize the training process of a quadratic network, thereby unleashing the
full potential in its associated machine learning tasks. Comprehensive
experiments on popular datasets are performed to support our findings and
evaluate the performance of quadratic deep learning.
