Superbloom: Bloom filter meets Transformer
  We extend the idea of word pieces in natural language models to machine
learning tasks on opaque ids. This is achieved by applying hash functions to
map each id to multiple hash tokens in a much smaller space, similarly to a
Bloom filter. We show that by applying a multi-layer Transformer to these Bloom
filter digests, we are able to obtain models with high accuracy. They
outperform models of a similar size without hashing and, to a large degree,
models of a much larger size trained using sampled softmax with the same
computational budget. Our key observation is that it is important to use a
multi-layer Transformer for Bloom filter digests to remove ambiguity in the
hashed input. We believe this provides an alternative method to solving
problems with large vocabulary size.
