ProFormer: Towards On-Device LSH Projection Based Transformers
  At the heart of text based neural models lay word representations, which are
powerful but occupy a lot of memory making it challenging to deploy to devices
with memory constraints such as mobile phones, watches and IoT. To surmount
these challenges, we introduce ProFormer -- a projection based transformer
architecture that is faster and lighter making it suitable to deploy to memory
constraint devices and preserve user privacy. We use LSH projection layer to
dynamically generate word representations on-the-fly without embedding lookup
tables leading to significant memory footprint reduction from O(V.d) to O(T),
where V is the vocabulary size, d is the embedding dimension size and T is the
dimension of the LSH projection representation.
  We also propose a local projection attention (LPA) layer, which uses
self-attention to transform the input sequence of N LSH word projections into a
sequence of N/K representations reducing the computations quadratically by
O(K^2). We evaluate ProFormer on multiple text classification tasks and
observed improvements over prior state-of-the-art on-device approaches for
short text classification and comparable performance for long text
classification tasks. In comparison with a 2-layer BERT model, ProFormer
reduced the embedding memory footprint from 92.16 MB to 1.3 KB and requires 16
times less computation overhead, which is very impressive making it the fastest
and smallest on-device model.
