A Structured Dictionary Perspective on Implicit Neural Representations
  Implicit neural representations (INRs) have recently emerged as a promising
alternative to classical discretized representations of signals. Nevertheless,
despite their practical success, we still do not understand how INRs represent
signals. We propose a novel unified perspective to theoretically analyse INRs.
Leveraging results from harmonic analysis and deep learning theory, we show
that most INR families are analogous to structured signal dictionaries whose
atoms are integer harmonics of the set of initial mapping frequencies. This
structure allows INRs to express signals with an exponentially increasing
frequency support using a number of parameters that only grows linearly with
depth. We also explore the inductive bias of INRs exploiting recent results
about the empirical neural tangent kernel (NTK). Specifically, we show that the
eigenfunctions of the NTK can be seen as dictionary atoms whose inner product
with the target signal determines the final performance of their
reconstruction. In this regard, we reveal that meta-learning has a reshaping
effect on the NTK analogous to dictionary learning, building dictionary atoms
as a combination of the examples seen during meta-training. Our results permit
to design and tune novel INR architectures, but can also be of interest for the
wider deep learning theory community.
