On the Origins of the Block Structure Phenomenon in Neural Network
  Representations
  Recent work has uncovered a striking phenomenon in large-capacity neural
networks: they contain blocks of contiguous hidden layers with highly similar
representations. This block structure has two seemingly contradictory
properties: on the one hand, its constituent layers exhibit highly similar
dominant first principal components (PCs), but on the other hand, their
representations, and their common first PC, are highly dissimilar across
different random seeds. Our work seeks to reconcile these discrepant properties
by investigating the origin of the block structure in relation to the data and
training methods. By analyzing properties of the dominant PCs, we find that the
block structure arises from dominant datapoints - a small group of examples
that share similar image statistics (e.g. background color). However, the set
of dominant datapoints, and the precise shared image statistic, can vary across
random seeds. Thus, the block structure reflects meaningful dataset statistics,
but is simultaneously unique to each model. Through studying hidden layer
activations and creating synthetic datapoints, we demonstrate that these simple
image statistics dominate the representational geometry of the layers inside
the block structure. We explore how the phenomenon evolves through training,
finding that the block structure takes shape early in training, but the
underlying representations and the corresponding dominant datapoints continue
to change substantially. Finally, we study the interplay between the block
structure and different training mechanisms, introducing a targeted
intervention to eliminate the block structure, as well as examining the effects
of pretraining and Shake-Shake regularization.
