Structure Inducing Pre-Training
  We present a theoretical analysis from first principles that establishes a
novel connection between relational inductive bias of pre-training and
fine-tuning performance while providing an extended view on general
pre-training models. We further explore how existing pre-training methods
impose relational inductive biases, finding that the vast majority of existing
approaches focus almost exclusively on modelling relationships in an
intra-sample manner, rather than a per-sample manner. We build upon these
findings with simulations and empirical studies on standard benchmarks spanning
3 data modalities and 10 downstream tasks. These investigations validate our
theoretical analyses, and provides a recipe to produce new pre-training methods
which incorporate provably richer inductive biases than do existing methods in
line with user specified relational graphs.
