title: spark scala - replace text if exists in list
body: <p>I have 2 datasets. 
  One is a dataframe with a bunch of data, one column has comments (a string). 
  The other is a list of words.</p>
  
  <p>If a comment contains a word in the list, I want to replace the word in the comment with @@@@@ and return the comment in full with the replaced words.</p>
  
  <p>Here's some sample data:</p>
  
  <p>CommentSample.txt</p>
  
  <pre><code>1   A badword small town
  2   "Love the truck, though rattle is annoying."
  3   Love the paint!
  4   
  5   "Like that you added the ""oh badword2"" handle to passenger side."
  6   "badword you. specific enough for you, badword3?"   
  7   This car is a piece if badword2
  </code></pre>
  
  <p>ProfanitySample.txt</p>
  
  <pre><code>badword
  badword2
  badword3
  </code></pre>
  
  <p>Here's my code so far:</p>
  
  <pre><code>val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  
  import sqlContext.implicits._
  
  case class Response(UniqueID: Int, Comment: String)
  
  val response = sc.textFile("file:/data/CommentSample.txt").map(_.split("\t")).filter(_.size == 2).map(r =&gt; Response(r(0).trim.toInt, r(1).trim.toString, r(10).trim.toInt)).toDF()
  
  var profanity = sc.textFile("file:/data/ProfanitySample.txt").map(x =&gt; (x.toLowerCase())).toArray();
  
      def replaceProfanity(s: String): String = {
          val l = s.toLowerCase()
          val r = "@@@@@"
          if(profanity.contains(s))
              r
          else
              s
        }
  
      def processComment(s: String): String = {
          val commentWords = sc.parallelize(s.split(' '))
          commentWords.foreach(replaceProfanity)
          commentWords.collect().mkString(" ")
        }
  
      response.select(processComment("Comment")).show(100)
  </code></pre>
  
  <p>It compiles, it runs, but the words are not replaced.
  I don't know how to debug in scala.
  I'm totally new! This is my first project ever!</p>
  
  <p>Many thanks for any pointers.
  -M</p>