title: Spark Equivalent of IF Then ELSE
body: <p>I have seen this question earlier here and I have took lessons from that. However I am not sure why I am getting an error when I feel it should work. </p>
  
  <p>I want to create a new column in existing Spark <code>DataFrame</code> by some rules. Here is what I wrote. iris_spark is the data frame with a categorical variable iris_spark with three distinct categories. </p>
  
  <pre><code>from pyspark.sql import functions as F
  
  iris_spark_df = iris_spark.withColumn(
      "Class", 
     F.when(iris_spark.iris_class == 'Iris-setosa', 0, F.when(iris_spark.iris_class == 'Iris-versicolor',1)).otherwise(2))
  </code></pre>
  
  <p>Throws the following error. </p>
  
  <pre><code>---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-157-21818c7dc060&gt; in &lt;module&gt;()
  ----&gt; 1 iris_spark_df=iris_spark.withColumn("Class",F.when(iris_spark.iris_class=='Iris-setosa',0,F.when(iris_spark.iris_class=='Iris-versicolor',1)))
  
  TypeError: when() takes exactly 2 arguments (3 given)
  
  
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-157-21818c7dc060&gt; in &lt;module&gt;()
  ----&gt; 1 iris_spark_df=iris_spark.withColumn("Class",F.when(iris_spark.iris_class=='Iris-setosa',0,F.when(iris_spark.iris_class=='Iris-versicolor',1)))
  
  TypeError: when() takes exactly 2 arguments (3 given)
  </code></pre>
  
  <p>Any idea why?</p>