ResNets, NeuralODEs and CT-RNNs are Particular Neural Regulatory
  Networks
  This paper shows that ResNets, NeuralODEs, and CT-RNNs, are particular neural
regulatory networks (NRNs), a biophysical model for the nonspiking neurons
encountered in small species, such as the C.elegans nematode, and in the retina
of large species. Compared to ResNets, NeuralODEs and CT-RNNs, NRNs have an
additional multiplicative term in their synaptic computation, allowing them to
adapt to each particular input. This additional flexibility makes NRNs $M$
times more succinct than NeuralODEs and CT-RNNs, where $M$ is proportional to
the size of the training set. Moreover, as NeuralODEs and CT-RNNs are $N$ times
more succinct than ResNets, where $N$ is the number of integration steps
required to compute the output $F(x)$ for a given input $x$, NRNs are in total
$M\,{\cdot}\,N$ more succinct than ResNets. For a given approximation task,
this considerable succinctness allows to learn a very small and therefore
understandable NRN, whose behavior can be explained in terms of well
established architectural motifs, that NRNs share with gene regulatory
networks, such as, activation, inhibition, sequentialization, mutual exclusion,
and synchronization. To the best of our knowledge, this paper unifies for the
first time the mainstream work on deep neural networks with the one in biology
and neuroscience in a quantitative fashion.
