Sentence Embeddings by Ensemble Distillation
  This paper contributes a new State Of The Art (SOTA) for Semantic Textual
Similarity (STS). We compare and combine a number of recently proposed sentence
embedding methods for STS, and propose a novel and simple ensemble knowledge
distillation scheme that improves on previous approaches. Our experiments
demonstrate that a model trained to learn the average embedding space from
multiple ensemble students outperforms all the other individual models with
high robustness. Utilizing our distillation method in combination with previous
methods, we significantly improve on the SOTA unsupervised STS, and by proper
hyperparameter tuning of previous methods we improve the supervised SOTA
scores.
