Modern Hopfield Networks and Attention for Immune Repertoire
  Classification
  A central mechanism in machine learning is to identify, store, and recognize
patterns. How to learn, access, and retrieve such patterns is crucial in
Hopfield networks and the more recent transformer architectures. We show that
the attention mechanism of transformer architectures is actually the update
rule of modern Hopfield networks that can store exponentially many patterns. We
exploit this high storage capacity of modern Hopfield networks to solve a
challenging multiple instance learning (MIL) problem in computational biology:
immune repertoire classification. Accurate and interpretable machine learning
methods solving this problem could pave the way towards new vaccines and
therapies, which is currently a very relevant research topic intensified by the
COVID-19 crisis. Immune repertoire classification based on the vast number of
immunosequences of an individual is a MIL problem with an unprecedentedly
massive number of instances, two orders of magnitude larger than currently
considered problems, and with an extremely low witness rate. In this work, we
present our novel method DeepRC that integrates transformer-like attention, or
equivalently modern Hopfield networks, into deep learning architectures for
massive MIL such as immune repertoire classification. We demonstrate that
DeepRC outperforms all other methods with respect to predictive performance on
large-scale experiments, including simulated and real-world virus infection
data, and enables the extraction of sequence motifs that are connected to a
given disease class. Source code and datasets: https://github.com/ml-jku/DeepRC
