Learn Like The Pro: Norms from Theory to Size Neural Computation
  The optimal design of neural networks is a critical problem in many
applications. Here, we investigate how dynamical systems with polynomial
nonlinearities can inform the design of neural systems that seek to emulate
them. We propose a Learnability metric and its associated features to quantify
the near-equilibrium behavior of learning dynamics. Equating the Learnability
of neural systems with equivalent parameter estimation metric of the reference
system establishes bounds on network structure. In this way, norms from theory
provide a good first guess for neural structure, which may then further adapt
with data. The proposed approach neither requires training nor training data.
It reveals exact sizing for a class of neural networks with multiplicative
nodes that mimic continuous- or discrete-time polynomial dynamics. It also
provides relatively tight lower size bounds for classical feed-forward networks
that is consistent with simulated assessments.
