title: How do you calculate the Cosine Similarity Between Two Corresponding Rows using TF-IDF
body: <p><em>For working Example copy and paste code based on number above code in the Update section and example section.</em></p>
  <p>How would you go about calculating the cosine similarity between two corresponding columns of text. For instance (headline, article body).</p>
  <p>Headlines   articleBodies</p>
  <p>headline1 | articleBody 1</p>
  <p>headline2 | articleBody 2
  .
  .
  .</p>
  <p>I'm only trying to get the cosine similarity between corresponding headline and articles. For example,</p>
  <p>Headlines |  articleBodies  | cosine_similarity</p>
  <p>headline1 | articleBody 1   |  value</p>
  <p>headline2 | articleBody 2   |   Value
  .
  .
  .</p>
  <p>It seems to work, but</p>
  <pre><code>def computeTFIDF(train_headlines, train_bodies):
    vec_body = TfidfVectorizer(ngram_range=(1,2),  lowercase=True, 
               stop_words='english', max_df=.5, max_features = 10000)
    tfidf_body = vec_body.fit_transform(train_bodies)
   
    vec_headline = TfidfVectorizer(ngram_range=(1,2), lowercase=True, 
                   stop_words='english',max_df=.5, max_features=10000)
    tfidf_headline = vec_headline.fit_transform(train_headlines)
  
    return tfidf_headline, tfidf_body 
  
  from sklearn.metrics.pairwise import cosine_similarity
  def arr_convert_1d(arr): 
      arr = np.array(arr) 
      arr = np.concatenate( arr, axis=0 ) 
      arr = np.concatenate( arr, axis=0 ) 
      return arr 
  
  def compute_cos_hd_body(headline, body):
    result = []
    for i, (head, body) in enumerate(zip(headline, body)):
      result.append(cosine_similarity(head, body))
    return arr_convert_1d(result)
  
  tr_cos_tfidf = compute_cos_hd_body(train_headlines_tfidf, train_bodies_tfidf)
  </code></pre>
  <p>Now, this seems to work on a large dataset ~50,000 rows of headlines and corresponding articles.
  However, running it on a much smaller data frame of  3 rows of headline article body pairs returns this error:</p>
  <pre><code>/usr/local/lib/python3.6/dist-packages/sklearn/metrics/pairwise.py in check_pairwise_arrays(X, Y, precomputed, dtype, accept_sparse, force_all_finite, copy)
      153         raise ValueError(&quot;Incompatible dimension for X and Y matrices: &quot;
      154                          &quot;X.shape[1] == %d while Y.shape[1] == %d&quot; % (
  --&gt; 155                              X.shape[1], Y.shape[1]))
      156 
      157     return X, Y
  
  ValueError: Incompatible dimension for X and Y matrices: X.shape[1] == 27 while Y.shape[1] == 707
  </code></pre>
  <p>Of course, I understand what the error is, but I'm confused why it's occurring when it worked on a larger set.
  Is it that it's incorrectly calculating the cosine similarity? Should I concatenate the headline, articleBody pairs and vec.fit_transform(headBody) then trasnform the headline and articleBody individually?</p>
  <hr />
  <hr />
  <p><strong>Update: I've edited the code (Copy code from here down, and it should provide working example)</strong>:</p>
  <p><em>1)</em></p>
  <pre><code> # whole_set is the headline and articleBody concatenated into
   # one column
   def compute_tfidf(whole_set, train_headlines, train_bodies, max_feat):
     vec_whole = TfidfVectorizer(ngram_range=(1,2),  lowercase=True, stop_words='english', max_df=.5, max_features = max_feat)
     tfidf_whole = vec_whole.fit_transform(whole_set)
     tfidf_body = vec_whole.transform(train_bodies)
     tfidf_headline = vec_whole.transform(train_headlines)
     return tfidf_headline, tfidf_body 
  
  def arr_convert_1d(arr): 
      arr = np.array(arr) 
      arr = np.concatenate( arr, axis=0 )
      # Commenting this out seemed to produce a set of values,
      #  but I'm not sure if the values are correct.
      # arr = np.concatenate( arr, axis=0 )
      
      return arr 
  def compute_cos_hd_body( headline = None, body = None):
    result = []
    for i, (head, body) in enumerate(zip(headline, body)):
       #print(&quot;Head {}\n&quot;.format(headline))
       #print(&quot;\nBody: {}\n&quot;.format(body))
         
       result.append(cosine_similarity(head, body))
     return arr_convert_1d(result)
     
  </code></pre>
  <p>Example dataFrame :</p>
  <p><em>2)</em></p>
  <pre><code>list_of_lists = []
  list_of_lists.append([&quot;Therefore feminist blogger aborted boy probably fake&quot;,
                        &quot;Amazon boss Jeff Bezos primed ready fresh assault streaming video space. The e commerce giant roll new ad supported streaming offering early next year separate 99 year Prime membership includes video service sources said. The ad supported option ‚Äî part overhaul media offerings ‚Äî poses serious challenge streaming rivals Hulu Netflix analysts said. If ad supported service decouple Prime Netflix killer Wedbush Securities analyst Michael Pachter said. It 99 year. Pachter suggested Amazon would undercut Netflix current monthly price 7.99. Who switch poor cord cutter added. Although separate Prime ad supported service ultimately bid Amazon lure people eventually pay Prime membership said one ad source familiar Amazon plans. The main point bring users eventually sell Prime get broader audience want pay Prime order increase video share source said. The Wall Street Journal reported March Amazon weighing move sources confirmed definite go. Amazon prepping new attack time video service gaining ground sources said. Amazon disclose number Prime members RBC Capital analyst Mark Mahaney estimates 50 million global customers. While Prime big draw two day shipping half Prime subscribers use video service&quot;,
                        &quot;therefore feminist blogger aborted boy probably fake amazon boss jeff bezos primed ready fresh assault streaming video space. the e commerce giant roll new ad supported streaming offering early next year separate 99 year prime membership includes video service sources said. the ad supported option ‚äî part overhaul media offerings ‚äî poses serious challenge streaming rivals hulu netflix analysts said. if ad supported service decouple prime netflix killer wedbush securities analyst michael pachter said. it 99 year. pachter suggested amazon would undercut netflix current monthly price 7.99. who switch poor cord cutter added. although separate prime ad supported service ultimately bid amazon lure people eventually pay prime membership said one ad source familiar amazon plans. the main point bring users eventually sell prime get broader audience want pay prime order increase video share source said. the wall street journal reported march amazon weighing move sources confirmed definite go. amazon prepping new attack time video service gaining ground sources said. amazon disclose number prime members rbc capital analyst mark mahaney estimates 50 million global customers. while prime big draw two day shipping half prime subscribers use video service&quot;
  ])
  list_of_lists.append([&quot;Seth Rogen Is Woz&quot;,
                        &quot;Danny Boyle directing untitled film Seth Rogen eyed play Apple co founder Steve Wozniak Sony Steve Jobs biopic. Danny Boyle directing untitled film based Walter Isaacson book adapted Aaron Sorkin one anticipated biopics recent years. Negotiations yet begun even clear Rogen official offer producers ‚Äî Scott Rudin Guymon Casady Mark Gordon ‚Äî set sights talent talks. Of course may naught Christian Bale actor play Jobs still midst closing deal. Sources say dealmaking process sensitive stage. Insiders say Boyle flying Los Angeles meet actress play one female leads assistant Jobs. Insiders say Jessica Chastain one actresses meeting list. Wozniak known Woz co founded Apple Jobs Ronald Wayne. He first met Jobs worked Atari later responsible creating early Apple computers.&quot;,
                        &quot;seth rogen is woz danny boyle directing untitled film seth rogen eyed play apple co founder steve wozniak sony steve jobs biopic. danny boyle directing untitled film based walter isaacson book adapted aaron sorkin one anticipated biopics recent years. negotiations yet begun even clear rogen official offer producers ‚äî scott rudin guymon casady mark gordon ‚äî set sights talent talks. of course may naught christian bale actor play jobs still midst closing deal. sources say dealmaking process sensitive stage. insiders say boyle flying los angeles meet actress play one female leads assistant jobs. insiders say jessica chastain one actresses meeting list. wozniak known woz co founded apple jobs ronald wayne. he first met jobs worked atari later responsible creating early apple computers.&quot;
  ])
  list_of_lists.append([&quot;CARLISLE HEAD SLAPPER ATTACKS SNEEZING WOMEN&quot;,
                        &quot;A man head slapping people Carlisle sneeze. Cumbria police said 82 year old woman reported slapped sneezed Scotch Street near Costa 11.30am today. The suspect man believed late 50s. It reported similar incident happened yesterday victim yet traced police. Anyone may witnessed incident information asked contact PC Lori Tallantire 101.&quot;,
                        &quot;carlisle head slapper attacks sneezing women a man head slapping people carlisle sneeze. cumbria police said 82 year old woman reported slapped sneezed scotch street near costa 11.30am today. the suspect man believed late 50s. it reported similar incident happened yesterday victim yet traced police. anyone may witnessed incident information asked contact pc lori tallantire 101.&quot;
                        ])
  
  train_hl_ab_merged = pd.DataFrame(list_of_lists, columns=[&quot;Headline&quot;,'first_sent_180', 'headBody']) 
  </code></pre>
  <p>When I run this:</p>
  <p><em>3)</em></p>
  <pre><code>tfidf_headline, tfidf_body, tfidf_whole = compute_tfidf(train_hl_ab_merged['headBody'], train_hl_ab_merged['Headline'], train_hl_ab_merged['first_sent_180'], 10000)
      
  tr_cos_tfidf = compute_cos_hd_body(train_hl_ab_merged, tfidf_headline, tfidf_body, 'tfidf_semantic' )
      
  tr_cos_tfidf, len(tr_cos_tfidf)
  </code></pre>
  <p>I get:</p>
  <pre><code> (array([[0.        ],
           [0.13459548],
           [0.06392028]]), 3)
  </code></pre>
  <p>--
  Update 2:
  It seems that it was a syntax error somewhere in my code that was causing the issue and actually does work.</p>
  <pre><code>def arr_convert_1d(arr): 
        arr = np.array(arr) 
        arr = np.concatenate( arr, axis=0 )
        arr = np.concatenate( arr, axis=0 )
        # I've uncommented the line above as the result above showed 
        # that it should have been peeled an additional array.
        return arr
  </code></pre>
  <p>Running code block <strong>3)</strong>, but excluding len(tr_cos_tfidf)</p>
  <pre><code>tfidf_headline, tfidf_body, tfidf_whole = compute_tfidf(train_hl_ab_merged['headBody'], train_hl_ab_merged['Headline'], train_hl_ab_merged['first_sent_180'], 10000)
              
  tr_cos_tfidf = compute_cos_hd_body(train_hl_ab_merged, tfidf_headline, tfidf_body, 'tfidf_semantic' )
              
  tr_cos_tfidf
  #len(tr_cos_tfidf) # Not necessary
  </code></pre>
  <p>with the enabled code in arr_convert_1d(arr) above returns:</p>
  <blockquote>
  <p>(array([0.        , 0.13459548, 0.06392028])</p>
  </blockquote>