Federated Word2Vec: Leveraging Federated Learning to Encourage
  Collaborative Representation Learning
  Large scale contextual representation models have significantly advanced NLP
in recent years, understanding the semantics of text to a degree never seen
before. However, they need to process large amounts of data to achieve
high-quality results. Joining and accessing all these data from multiple
sources can be extremely challenging due to privacy and regulatory reasons.
Federated Learning can solve these limitations by training models in a
distributed fashion, taking advantage of the hardware of the devices that
generate the data. We show the viability of training NLP models, specifically
Word2Vec, with the Federated Learning protocol. In particular, we focus on a
scenario in which a small number of organizations each hold a relatively large
corpus. The results show that neither the quality of the results nor the
convergence time in Federated Word2Vec deteriorates as compared to centralised
Word2Vec.
