The curious case of developmental BERTology: On sparsity, transfer
  learning, generalization and the brain
  In this essay, we explore a point of intersection between deep learning and
neuroscience, through the lens of large language models, transfer learning and
network compression. Just like perceptual and cognitive neurophysiology has
inspired effective deep neural network architectures which in turn make a
useful model for understanding the brain, here we explore how biological neural
development might inspire efficient and robust optimization procedures which in
turn serve as a useful model for the maturation and aging of the brain.
