title: Get IDs for duplicate rows (considering all other columns) in Apache Spark
body: <p>I have a Spark sql dataframe, consisting of an <code>ID</code> column and <code>n</code> "data" columns, i.e.</p>
  
  <pre><code>id | dat1 | dat2 | ... | datn
  </code></pre>
  
  <p>The <code>id</code> columnn is uniquely determined, whereas, looking at <code>dat1 ... datn</code> there may be duplicates.</p>
  
  <p>My goal is to find the <code>id</code>s of those duplicates.</p>
  
  <p>My approach so far:</p>
  
  <ul>
  <li><p>get the duplicate rows using <code>groupBy</code>:</p>
  
  <p><code>dup_df = df.groupBy(df.columns[1:]).count().filter('count &gt; 1')
  </code></p></li>
  <li><p>join the <code>dup_df</code> with the entire <code>df</code> to get the duplicate rows <strong>including</strong> <code>id</code>:</p>
  
  <p><code>df.join(dup_df, df.columns[1:])
  </code></p></li>
  </ul>
  
  <p>I am quite certain that this is basically correct, it fails because the <code>dat1 ... datn</code> columns contain <code>null</code> values. </p>
  
  <p>To do the <code>join</code> on <code>null</code> values, I found .e.g <a href="https://stackoverflow.com/questions/39120934/comparison-operator-in-pyspark-not-equal">this SO post</a>. But this would require to construct a huge "string join condition".</p>
  
  <p>Thus my questions:</p>
  
  <ol>
  <li>Is there a simple / more generic / more pythonic way to do <code>joins</code> on <code>null</code> values?</li>
  <li>Or, even better, is there another (easier, more beautiful, ...) method to get the desired <code>id</code>s?</li>
  </ol>
  
  <p>BTW: I am using Spark 2.1.0 and Python 3.5.3</p>