mcBERT: Momentum Contrastive Learning with BERT for Zero-Shot Slot
  Filling
  Zero-shot slot filling has received considerable attention to cope with the
problem of limited available data for the target domain. One of the important
factors in zero-shot learning is to make the model learn generalized and
reliable representations. For this purpose, we present mcBERT, which stands for
momentum contrastive learning with BERT, to develop a robust zero-shot slot
filling model. mcBERT uses BERT to initialize the two encoders, the query
encoder and key encoder, and is trained by applying momentum contrastive
learning. Our experimental results on the SNIPS benchmark show that mcBERT
substantially outperforms the previous models, recording a new
state-of-the-art. Besides, we also show that each component composing mcBERT
contributes to the performance improvement.
