title: Spark / Scala - Compare Two Columns In a Dataframe when one is NULL
body: <p>I'm using Spark (Scala) to QA data movement - moving tables from one relational database to another. The QA process involves executing a full outer join between the source table, and the target table. </p>
  
  <p>The source table and target tables are joined in a data frame on the key(s):</p>
  
  <pre><code>val joinColumns = for (i &lt;- 0 to (sourceJoinFields.length - 1)) yield sourceDF.col(sourceJoinFields(i)) &lt;=&gt; targetDF.col(targetJoinFields(i))
  val joinedDF = sourceDF.join(targetDF, joinColumns.reduce((_&amp;&amp;_)), "fullouter")
  </code></pre>
  
  <p>I'm using the following logic to find mismatches:</p>
  
  <pre><code>val mismatchColumns = for (i &lt;- 0 to (sourceDF.columns.length-1)) yield (joinedDF.col(joinedDF.columns(i)) =!= joinedDF.col(joinedDF.columns(i+(sourceDF.columns.length))))
  val mismatchedDF = joinedDF.filter(mismatchColumns.reduce((_||_)))
  </code></pre>
  
  <p>However if there is a key missing from one side of the full outer join:</p>
  
  <pre class="lang-none prettyprint-override"><code>+--------------+--------------+--------------+--------------+
  |source_key    |source_field  |target_key    |target_field  |
  +--------------+--------------+--------------+--------------+
  |null          |null          |XXX           |XXX           |
  </code></pre>
  
  <p>will not be in the mismatchedDF data set. </p>
  
  <p>So my question: is the <code>=!=</code> operator the opposite of the <code>&lt;=&gt;</code> operator? It does not appear to be, so is there an operator that will return FALSE for this case? I can't find much documentation on either operator.</p>