title: get all the dates between two dates in Spark DataFrame
body: <p>I have a DF in which I have <code>bookingDt</code> and <code>arrivalDt</code> columns. I need to find all the dates between these two dates.</p>
  
  <p><strong>Sample code:</strong></p>
  
  <pre class="lang-python prettyprint-override"><code>df = spark.sparkContext.parallelize(
              [Row(vyge_id=1000, bookingDt='2018-01-01', arrivalDt='2018-01-05')]).toDF()
  diffDaysDF = df.withColumn("diffDays", datediff('arrivalDt', 'bookingDt'))
  diffDaysDF.show()
  </code></pre>
  
  <p><strong>code output:</strong></p>
  
  <pre class="lang-python prettyprint-override"><code>+----------+----------+-------+--------+
  | arrivalDt| bookingDt|vyge_id|diffDays|
  +----------+----------+-------+--------+
  |2018-01-05|2018-01-01|   1000|       4|
  +----------+----------+-------+--------+
  </code></pre>
  
  <p>What I tried was finding the number of days between two dates and calculate all the dates using <code>timedelta</code> function and <code>explode</code> it.</p>
  
  <pre class="lang-python prettyprint-override"><code>dateList = [str(bookingDt + timedelta(i)) for i in range(diffDays)]
  </code></pre>
  
  <p><strong>Expected output:</strong></p>
  
  <p>Basically, I need to build a DF with a record for each date in between <code>bookingDt</code> and <code>arrivalDt</code>, inclusive.</p>
  
  <pre class="lang-python prettyprint-override"><code>+----------+----------+-------+----------+
  | arrivalDt| bookingDt|vyge_id|txnDt     |
  +----------+----------+-------+----------+
  |2018-01-05|2018-01-01|   1000|2018-01-01|
  +----------+----------+-------+----------+
  |2018-01-05|2018-01-01|   1000|2018-01-02|
  +----------+----------+-------+----------+
  |2018-01-05|2018-01-01|   1000|2018-01-03|
  +----------+----------+-------+----------+
  |2018-01-05|2018-01-01|   1000|2018-01-04|
  +----------+----------+-------+----------+
  |2018-01-05|2018-01-01|   1000|2018-01-05|
  +----------+----------+-------+----------+
  </code></pre>