title: How to use NOT IN clause in filter condition in spark
body: <p>I want to filter a column of an RDD source  :</p>
  
  <pre><code>val source = sql("SELECT * from sample.source").rdd.map(_.mkString(","))
  val destination = sql("select * from sample.destination").rdd.map(_.mkString(","))
  
  val source_primary_key = source.map(rec =&gt; (rec.split(",")(0)))
  val destination_primary_key = destination.map(rec =&gt; (rec.split(",")(0)))
  
  val src = source_primary_key.subtractByKey(destination_primary_key)
  </code></pre>
  
  <p>I want to use IN clause in filter condition to filter out only the values present in src from source, something like below(EDITED):</p>
  
  <pre><code>val source = spark.read.csv(inputPath + "/source").rdd.map(_.mkString(","))
  val destination = spark.read.csv(inputPath + "/destination").rdd.map(_.mkString(","))
  
  val source_primary_key = source.map(rec =&gt; (rec.split(",")(0)))
  val destination_primary_key = destination.map(rec =&gt; (rec.split(",")(0)))
  
  val extra_in_source = source_primary_key.filter(rec._1 != destination_primary_key._1)
  </code></pre>
  
  <p>equivalent SQL code is  </p>
  
  <pre><code>SELECT * FROM SOURCE WHERE ID IN (select ID from src)
  </code></pre>
  
  <p>Thank you</p>