Transformers with Competitive Ensembles of Independent Mechanisms
  An important development in deep learning from the earliest MLPs has been a
move towards architectures with structural inductive biases which enable the
model to keep distinct sources of information and routes of processing
well-separated. This structure is linked to the notion of independent
mechanisms from the causality literature, in which a mechanism is able to
retain the same processing as irrelevant aspects of the world are changed. For
example, convnets enable separation over positions, while attention-based
architectures (especially Transformers) learn which combination of positions to
process dynamically. In this work we explore a way in which the Transformer
architecture is deficient: it represents each position with a large monolithic
hidden representation and a single set of parameters which are applied over the
entire hidden representation. This potentially throws unrelated sources of
information together, and limits the Transformer's ability to capture
independent mechanisms. To address this, we propose Transformers with
Independent Mechanisms (TIM), a new Transformer layer which divides the hidden
representation and parameters into multiple mechanisms, which only exchange
information through attention. Additionally, we propose a competition mechanism
which encourages these mechanisms to specialize over time steps, and thus be
more independent. We study TIM on a large-scale BERT model, on the Image
Transformer, and on speech enhancement and find evidence for semantically
meaningful specialization as well as improved performance.
