title: How to compute the similarity between two text columns in dataframes with pyspark?
body: <p>I have 2 data frames with different # of rows. Both them has a column as text. My aim to compare them and find similarities and find a ratio of similarity and add this score in final data set. Comparison is between title from df1 and headline from df2. Position of these text rows are different.</p>
  <p><strong>df1</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>duration</th>
  <th>title</th>
  <th>publish_start_date</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>129.33</td>
  <td>Smuggler's Run fr...</td>
  <td>2021-10-29T10:21:...</td>
  </tr>
  <tr>
  <td>49.342</td>
  <td>anchises. Founded...</td>
  <td>2021-10-29T06:00:...</td>
  </tr>
  <tr>
  <td>69.939</td>
  <td>by Diego Angel in...</td>
  <td>2021-10-29T00:33:...</td>
  </tr>
  <tr>
  <td>102.60</td>
  <td>Orange County sch...</td>
  <td>2021-10-28T10:24:...</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><strong>df2</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>DataSource</th>
  <th>Post Id</th>
  <th>headline</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Linkedin</td>
  <td>L1904055</td>
  <td>in English versi...</td>
  </tr>
  <tr>
  <td>Linkedin</td>
  <td>F6955268</td>
  <td>in other language...</td>
  </tr>
  <tr>
  <td>Facebook</td>
  <td>F1948698</td>
  <td>Its combined edit...</td>
  </tr>
  <tr>
  <td>Twitter</td>
  <td>T7954991</td>
  <td>Emma Raducanu: 10...</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Basically, I am trying to find a similarities between 2 data sets row by row (on text). Is there any way to do this?</p>
  <p>number of final data set = number of first data set x number of second data set</p>