title: Python Calculating similarity between two documents using word2vec, doc2vec
body: <p>I am trying to calculate similarity between two documents which are comprised of more than thousands sentences.</p>
  
  <p>Baseline would be calculating cosine similarity using BOW.</p>
  
  <p>However, I want to capture more of semantic difference between documents.</p>
  
  <p>Hence, I built word embedding and calculated documents similarity by generating document vectors by simply averaging all the word vectors in each of documents and measure cosine similarity between these documents vectors. </p>
  
  <p>However, since the size of each input document is rather big, the results I get from using the method above are very similar to simple BOW cosine similarity.</p>
  
  <p>I have two questions, </p>
  
  <p>Q1. I found gensim module offers soft cosine similarity. But I am having hard time understanding the difference from the methods I used above, and I think it may not be the mechanism to calculate similarity between million pairs of documents.</p>
  
  <p>Q2. I found Doc2Vec by gensim would be more appropriate for my purpose. But I recognized that training Doc2Vec requires more RAM than I have (32GB) (the size of my entire documents is about 100GB). Would there be any way that I train the model with small part(like 20GB of them) of entire corpus, and use this model to calculate pairwise similarities of entire corpus?
  If yes, then what would be the desirable train set size, and is there any tutorial that I can follow?</p>