Learning Inductive Biases with Simple Neural Networks
  People use rich prior knowledge about the world in order to efficiently learn
new concepts. These priors - also known as "inductive biases" - pertain to the
space of internal models considered by a learner, and they help the learner
make inferences that go beyond the observed data. A recent study found that
deep neural networks optimized for object recognition develop the shape bias
(Ritter et al., 2017), an inductive bias possessed by children that plays an
important role in early word learning. However, these networks use
unrealistically large quantities of training data, and the conditions required
for these biases to develop are not well understood. Moreover, it is unclear
how the learning dynamics of these networks relate to developmental processes
in childhood. We investigate the development and influence of the shape bias in
neural networks using controlled datasets of abstract patterns and synthetic
images, allowing us to systematically vary the quantity and form of the
experience provided to the learning algorithms. We find that simple neural
networks develop a shape bias after seeing as few as 3 examples of 4 object
categories. The development of these biases predicts the onset of vocabulary
acceleration in our networks, consistent with the developmental process in
children.
