Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding
  Most existing set encoding algorithms operate under the implicit assumption
that all the set elements are accessible, and that there are ample
computational and memory resources to load the set into memory during training
and inference. However, both assumptions fail when the set is excessively large
such that it is impossible to load all set elements into memory, or when data
arrives in a stream. To tackle such practical challenges in large-scale set
encoding, the general set-function constraints of permutation invariance and
equivariance are not sufficient. We introduce a new property termed Mini-Batch
Consistency (MBC) that is required for large scale mini-batch set encoding.
Additionally, we present a scalable and efficient attention-based set encoding
mechanism that is amenable to mini-batch processing of sets, and capable of
updating set representations as data arrives. The proposed method adheres to
the required symmetries of invariance and equivariance as well as maintaining
MBC for any partition of the input set. We perform extensive experiments and
show that our method is computationally efficient and results in rich set
encoding representations for set-structured data.
