title: PySpark add new column to dataframe with new list
body: <p>Based on previous questions: <a href="https://stackoverflow.com/questions/40222872/add-list-as-column-to-dataframe-in-pyspark">1</a>, <a href="https://stackoverflow.com/questions/33681487/how-do-i-add-a-new-column-to-a-spark-dataframe-using-pyspark">2</a>. Suppose I have the following dataframe:</p>
  
  <pre><code>df = spark.createDataFrame(
      [(1, "a", 23.0), (3, "B", -23.0)], 
       ("x1", "x2", "x3"))
  </code></pre>
  
  <p>And I want to add new column <code>x4</code> but I have value in a list of Python instead to add to the new column e.g. <code>x4_ls = [35.0, 32.0]</code>. Is there a best way to add new column to the Spark dataframe? (<strong>note</strong> that I use Spark 2.1)</p>
  
  <p>Output should be something like:</p>
  
  <pre><code>## +---+---+-----+----+
  ## | x1| x2|   x3|  x4|
  ## +---+---+-----+----+
  ## |  1|  a| 23.0|35.0|
  ## |  3|  B|-23.0|32.0|
  ## +---+---+-----+----+
  </code></pre>
  
  <p>I can also transform my list to dataframe <code>df_x4 = spark.createDataFrame([Row(**{'x4': x}) for x in x4_ls])</code> (but I don't how to concatenate dataframe together)</p>