title: Merge overlapping date ranges in pyspark
body: <p>I have a dataframe with the schema (ID, start_date, end_date)</p>
  <p>Sample dataframe:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">id</th>
  <th style="text-align: center;">start_date</th>
  <th style="text-align: right;">end_date</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">1</td>
  <td style="text-align: center;">03/05/2020</td>
  <td style="text-align: right;">07/05/2020</td>
  </tr>
  <tr>
  <td style="text-align: left;">1</td>
  <td style="text-align: center;">05/05/2020</td>
  <td style="text-align: right;">07/05/2020</td>
  </tr>
  <tr>
  <td style="text-align: left;">1</td>
  <td style="text-align: center;">01/05/2020</td>
  <td style="text-align: right;">07/05/2020</td>
  </tr>
  <tr>
  <td style="text-align: left;">1</td>
  <td style="text-align: center;">01/05/2020</td>
  <td style="text-align: right;">13/05/2020</td>
  </tr>
  <tr>
  <td style="text-align: left;">2</td>
  <td style="text-align: center;">06/05/2020</td>
  <td style="text-align: right;">09/05/2020</td>
  </tr>
  <tr>
  <td style="text-align: left;">1</td>
  <td style="text-align: center;">18/05/2020</td>
  <td style="text-align: right;">20/05/2020</td>
  </tr>
  <tr>
  <td style="text-align: left;">2</td>
  <td style="text-align: center;">09/05/2020</td>
  <td style="text-align: right;">17/05/2020</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I want the result in below format</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">id</th>
  <th style="text-align: center;">start_dates</th>
  <th style="text-align: right;">days</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">1</td>
  <td style="text-align: center;">[03/05/2020, 05/05/2020, 01/05/2020, 01/05/2020,   18/05/2020]</td>
  <td style="text-align: right;">16</td>
  </tr>
  <tr>
  <td style="text-align: left;">2</td>
  <td style="text-align: center;">[06/05/2020, 09/05/2020]</td>
  <td style="text-align: right;">12</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>days is the sum of non overlapping date intervals (start_date - end_date).</p>
  <p>The approach that i tried is to group by id, collect start_date and end_date as lists and then write a UDF to get the days column. I would like to know if there is a better way to achieve this.</p>