title: Filtering a pyspark dataframe to get rows from last N days doesn't work correctly and returns some data before the stated date
body: <p>I added a new column to my spark dataframe by combining these 4 columns  <code>year,month,day,time_hour</code> as follow:</p>
  <pre><code>df = df.withColumn(&quot;tmp_timestamp&quot;, concat_ws(&quot; &quot;, concat_ws(&quot;_&quot;, col(&quot;year&quot;), col(&quot;month&quot;), col(&quot;day&quot;)), col(&quot;time_hour&quot;)))\
         .withColumn(&quot;tsmp&quot;, unix_timestamp(&quot;tmp_timestamp&quot;, &quot;yyyy_MM_dd HH&quot;))\
         .withColumn(&quot;date_time_EET&quot;, from_utc_timestamp(col(&quot;tsmp&quot;).cast('timestamp'), &quot;EET&quot;))\
         .drop(&quot;tmp_timestamp&quot;)
  </code></pre>
  <p>And here is the result:</p>
  <pre><code>+-------+--------+------+---------+-------------+----------+---------------------+
  |year   |month   |day   |time_hour|tmp_timestamp|tsmp      |date_time_EET        |
  +-------+--------+------+---------+-------------+----------+---------------------+
  |2021   |1       |22    |0        |2021_1_22 0  |1611273600|2021-01-22 02:00:00.0|
  |2021   |1       |22    |9        |2021_1_22 9  |1611306000|2021-01-22 11:00:00.0|
  |2021   |1       |22    |4        |2021_1_22 4  |1611288000|2021-01-22 06:00:00.0|
  |2021   |1       |22    |8        |2021_1_22 8  |1611302400|2021-01-22 10:00:00.0|
  |2021   |1       |22    |23       |2021_1_22 23 |1611356400|2021-01-23 01:00:00.0|
  |2021   |1       |22    |7        |2021_1_22 7  |1611298800|2021-01-22 09:00:00.0|
  |2021   |1       |22    |10       |2021_1_22 10 |1611309600|2021-01-22 12:00:00.0|
  |2021   |1       |22    |4        |2021_1_22 4  |1611288000|2021-01-22 06:00:00.0|
  |2021   |1       |22    |8        |2021_1_22 8  |1611302400|2021-01-22 10:00:00.0|
  |2021   |1       |22    |23       |2021_1_22 23 |1611356400|2021-01-23 01:00:00.0|
  +-------+--------+------+---------+-------------+----------+---------------------+
  only showing top 10 rows
  </code></pre>
  <p>Now, I need to filter over this new dataframe and get the rows their <code>date_time_EET</code> value is higher than a specific date, e.g. getting the data for the last 10 days. This is how I'm doing it:</p>
  <pre><code>N = 10
  date_N_days_ago = start_day - timedelta(days=N)  # In this example date_N_days_ago will be 2021-01-24
  
  df.filter(col('date_time_EET')&gt; date_N_days_ago)\
      .select(&quot;year&quot;,&quot;month&quot;,&quot;day&quot;,&quot;time_hour&quot;,&quot;date_time_EET&quot;, )\
      .dropDuplicates([&quot;ts_year&quot;,&quot;ts_month&quot;,&quot;ts_day&quot;,&quot;time_hour&quot;])\
      .orderBy(&quot;ts_year&quot;,&quot;ts_month&quot;,&quot;ts_day&quot;,&quot;time_hour&quot;,)\
      .show(100, False)
  </code></pre>
  <p>which gives me the following result:</p>
  <pre><code>+-------+--------+------+---------+---------------------+
  |year   |month   |day   |time_hour|date_time_EET        |
  +-------+--------+------+---------+---------------------+
  |2021   |1       |23    |21       |2021-01-23 23:00:00.0|
  |2021   |1       |23    |22       |2021-01-24 00:00:00.0|
  |2021   |1       |23    |23       |2021-01-24 01:00:00.0|
  |2021   |1       |24    |0        |2021-01-24 02:00:00.0|
  |2021   |1       |24    |1        |2021-01-24 03:00:00.0|
  |2021   |1       |24    |2        |2021-01-24 04:00:00.0|
  |2021   |1       |24    |3        |2021-01-24 05:00:00.0|
  |2021   |1       |24    |4        |2021-01-24 06:00:00.0|
  |2021   |1       |24    |5        |2021-01-24 07:00:00.0|
  |2021   |1       |24    |6        |2021-01-24 08:00:00.0|
  |2021   |1       |24    |7        |2021-01-24 09:00:00.0|
  |2021   |1       |24    |8        |2021-01-24 10:00:00.0|
  ...
  </code></pre>
  <p>Why do I get the first row where date_time_EET is &quot;2021-01-23 23:00:00.0&quot;, while <code>date_N_days_ago</code> is <code>2021-01-24</code>? Also, when I add another condition to the <code>filter</code> to returns dates before the <code>start_date</code> it doesn't return the data for the last hour (<code>2021-02-02 23:00:00.0</code> is missing):</p>
  <pre><code>df.filter((col('date_time_EET')&gt; date_N_days_ago) &amp; (col('date_time_EET') &lt;= start_date))\
      .select(&quot;year&quot;,&quot;month&quot;,&quot;day&quot;,&quot;time_hour&quot;,&quot;date_time_EET&quot;, )\
      .dropDuplicates([&quot;ts_year&quot;,&quot;ts_month&quot;,&quot;ts_day&quot;,&quot;time_hour&quot;])\
      .orderBy(&quot;ts_year&quot;,&quot;ts_month&quot;,&quot;ts_day&quot;,&quot;time_hour&quot;,)\
      .show(100, False)
  
  +-------+--------+------+---------+---------------------+
  |year   |month   |day   |time_hour|date_time_EET        |
  +-------+--------+------+---------+---------------------+
  ....
  |2021   |2       |2     |16       |2021-02-02 18:00:00.0|
  |2021   |2       |2     |17       |2021-02-02 19:00:00.0|
  |2021   |2       |2     |18       |2021-02-02 20:00:00.0|
  |2021   |2       |2     |19       |2021-02-02 21:00:00.0|
  |2021   |2       |2     |20       |2021-02-02 22:00:00.0|
  +-------+--------+------+---------+---------------------+
  </code></pre>