title: How to handle words that are not in word2vec's vocab optimally
body: <p>I have a list of ~10 million sentences, where each of them contains up to 70 words.</p>
  
  <p>I'm running gensim word2vec on every word, and then taking the simple average of each sentence. The problem is that I use min_count=1000, so a lot of words are not in the vocab. </p>
  
  <p>To solve that, I intersect the vocab array (that contains about 10000 words) with every sentence, and if there's at least one element left in that intersection, it returns its the simple average, otherwise, it returns a vector of zeros.</p>
  
  <p>The issue is that calculating every average takes a very long time when I run it on the whole dataset, even when splitting into multiple threads, and I would like to get a better solution that could run faster.</p>
  
  <p>I'm running this on an EC2 r4.4xlarge instance.</p>
  
  <p>I already tried switching to doc2vec, which was way faster, but the results were not as good as word2vec's simple average.</p>
  
  <pre><code>word2vec_aug_32x = Word2Vec(sentences=sentences, 
                          min_count=1000, 
                          size=32, 
                          window=2,
                          workers=16, 
                          sg=0)
  
  vocab_arr = np.array(list(word2vec_aug_32x.wv.vocab.keys()))
  
  def get_embedded_average(sentence):
      sentence = np.intersect1d(sentence, vocab_arr)
      if sentence.shape[0] &gt; 0:
          return np.mean(word2vec_aug_32x[sentence], axis=0).tolist()
      else:
          return np.zeros(32).tolist()
  
  pool = multiprocessing.Pool(processes=16)
  
  w2v_averages = np.asarray(pool.map(get_embedded_average, np.asarray(sentences)))
  pool.close()
  </code></pre>
  
  <p>If you have any suggestions of different algorithms or techniques that have the same purpose of sentence embedding and could solve my problem, I would love to read about it.</p>