Language Through a Prism: A Spectral Approach for Multiscale Language
  Representations
  Language exhibits structure at different scales, ranging from subwords to
words, sentences, paragraphs, and documents. To what extent do deep models
capture information at these scales, and can we force them to better capture
structure across this hierarchy? We approach this question by focusing on
individual neurons, analyzing the behavior of their activations at different
timescales. We show that signal processing provides a natural framework for
separating structure across scales, enabling us to 1) disentangle
scale-specific information in existing embeddings and 2) train models to learn
more about particular scales. Concretely, we apply spectral filters to the
activations of a neuron across an input, producing filtered embeddings that
perform well on part of speech tagging (word-level), dialog speech acts
classification (utterance-level), or topic classification (document-level),
while performing poorly on the other tasks. We also present a prism layer for
training models, which uses spectral filters to constrain different neurons to
model structure at different scales. Our proposed BERT + Prism model can better
predict masked tokens using long-range context and produces multiscale
representations that perform better at utterance- and document-level tasks. Our
methods are general and readily applicable to other domains besides language,
such as images, audio, and video.
