title: OR condition in dataframe full outer join reducing performance spark/scala
body: <p>I'm seeing that if my join condition involves OR clause, it takes a long time vs having an AND clause. I'm doing fullouter join. </p>
  
  <p>My unit test doesn't show a major difference but when run against a large dataset, it runs very slow</p>
  
  <pre><code>df1.join(df2, expr("id1 = id2 AND amount1 = amount2"), "fullouter").cache()
  </code></pre>
  
  <p>The following runs for long time i.e. magnitude of ~10 times slower</p>
  
  <pre><code>df1.join(df2, expr("id1 = id2 OR amount1 = amount2"), "fullouter").cache()
  </code></pre>
  
  <p>any thoughts?</p>