title: org.apache.spark.SparkException: Failed to execute user defined function
body: <p>I new to <em>scala</em> and I am trying to execute the following code: </p>
  
  <pre><code>val SetID = udf{(c:String, d: String) =&gt;
      if( c.UpperCase.contains("EXKLUS") == true)
      {d}
      else {""}
  }
  val ParquetWithID = STG1
    .withColumn("ID", SetID( col("line_item"), col("line_item_ID")))
  </code></pre>
  
  <p>both columns (<code>line_item</code> and <code>line_item_id</code>) are defined as <code>Strings</code> in the <code>STG1</code> Schema. </p>
  
  <p>I get the following error when I try to run the code: </p>
  
  <pre><code>`org.apache.spark.SparkException: Failed to execute user defined function($anonfun$1$$anonfun$2: (string, string) =&gt; string)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
  at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:246)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
  at org.apache.spark.scheduler.Task.run(Task.scala:86)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>and </p>
  
  <pre><code>Caused by: java.lang.NullPointerException
      at MyTests$$anonfun$1$$anonfun$2.apply(MyTests.scala:356)
      at MyTests$$anonfun$1$$anonfun$2.apply(MyTests.scala:355)
      ... 16 more
  </code></pre>
  
  <p>I also tried <code>c.UpperCase().contains("EXKLUS")</code> but i got the same error.
  However, if I simply run a "<code>if equals</code>" statement everything works fine. So I guess the problem is in using <code>UpperCase().contains(" ")</code> function in my <code>udf</code> but I do not understand where the problem comes from. Any help would be appriciated! </p>