title: Filter on joined dataframe is not working in pyspark
body: <p>I have a dataframe with the 3 following columns</p>
  
  <ul>
  <li>student_id</li>
  <li>name</li>
  <li>timestamp</li>
  </ul>
  
  <p>There are multiple rows for a student id with different names along with the timestamp the record actually updated. I wanted to get 2 different dataframes. </p>
  
  <ul>
  <li><p>unique_data (rows of all student ids with latest timestamp for that student id)</p></li>
  <li><p>duplicate_data (all the rows from input dataframe except the above mentioned unique_data rows)</p></li>
  </ul>
  
  <p>I have the following code which generates 2 dataframes</p>
  
  <pre><code>input_frame.show()
  +----------+----------+---------+
  |student_id|name      |timestamp|
  +----------+----------+---------+
  |        s1|testuser  |       t1|
  |        s1|sampleuser|       t2|
  |        s2|test123   |       t1|
  |        s2|sample123 |       t2|
  +----------+----------+---------+
  
  # Assuming t2 &gt; t1
  
  unique_data = input_frame.sort(sf.desc(timestamp))drop_duplicates("student_id")
  unique_data.show()
  +----------+----------+---------+
  |student_id|name      |timestamp|
  +----------+----------+---------+
  |        s1|sampleuser|       t2|
  |        s2|sample123 |       t2|
  +----------+----------+---------+
  
  input_frame = input_frame.alias('input_frame')
  unique_frame = unique_frame.alias('unique_frame')
  
  joined_data = input_frame.join(unique_data, input_frame["student_id"] == unique_data["student_id"], how="left")
  joined_data.show()
  +----------+----------+---------+----------+----------+---------+
  |student_id|name      |timestamp|student_id|name      |timestamp|
  +----------+----------+---------+----------+----------+---------+
  |        s1|testuser  |       t1|        s1|sampleuser|       t2|
  |        s1|sampleuser|       t2|        s1|sampleuser|       t2|
  |        s2|test123   |       t1|        s2|sample123 |       t2|
  |        s2|sample123 |       t2|        s2|sample123 |       t2|
  +----------+----------+---------+----------+----------+---------+
  
  
  
  duplicate_data = joined_data.filter(input_frame["timestamp"] != unique_data["timestamp"]).select("input_frame.*")
  duplicate_data.show()
  +----------+----+---------+
  |student_id|name|timestamp|
  +----------+----+---------+
  +----------+----+---------+
  </code></pre>