On the Modularity of Hypernetworks
  In the context of learning to map an input $I$ to a function
$h_I:\mathcal{X}\to \mathbb{R}$, two alternative methods are compared: (i) an
embedding-based method, which learns a fixed function in which $I$ is encoded
as a conditioning signal $e(I)$ and the learned function takes the form $h_I(x)
= q(x,e(I))$, and (ii) hypernetworks, in which the weights $\theta_I$ of the
function $h_I(x) = g(x;\theta_I)$ are given by a hypernetwork $f$ as
$\theta_I=f(I)$. In this paper, we define the property of modularity as the
ability to effectively learn a different function for each input instance $I$.
For this purpose, we adopt an expressivity perspective of this property and
extend the theory of Devore et al. 1996 and provide a lower bound on the
complexity (number of trainable parameters) of neural networks as function
approximators, by eliminating the requirements for the approximation method to
be robust. Our results are then used to compare the complexities of $q$ and
$g$, showing that under certain conditions and when letting the functions $e$
and $f$ be as large as we wish, $g$ can be smaller than $q$ by orders of
magnitude. This sheds light on the modularity of hypernetworks in comparison
with the embedding-based method. Besides, we show that for a structured target
function, the overall number of trainable parameters in a hypernetwork is
smaller by orders of magnitude than the number of trainable parameters of a
standard neural network and an embedding method.
