title: Error about column literals using udf function on a dataframe with pyspark
body: <p>I'm trying to use a udf function on a dataframe with pyspark but getting error about column literals and suggesting I use 'lit', 'array', 'struct' or 'create_map' function.  I´m not clear how to do this.</p>
  
  <pre><code>from pyspark.sql.types import IntegerType
  from pyspark.sql.functions import udf
  
  def compareElem(elem):
      return elem[1]
  
  def getSmallest(type, final_list):
    final_list.sort(key=compareElem)
    print(final_list)
    l = final_list[0][0]
    print('idx=', l)
    if type == 1:
      l = (((l/4)+1)*4)-1
  
    return l
  </code></pre>
  
  <p>Function works ok on List input</p>
  
  <pre><code>getSmallest(0, [ ( 0, 1), (1, 1.1), (2, 0.5) ])
  </code></pre>
  
  <p>returns</p>
  
  <pre><code>[(2, 0.5), (0, 1), (1, 1.1)] 
  ('idx=', 2)
  </code></pre>
  
  <p>But fails here used with udf and dataframe columns</p>
  
  <pre><code>func_udf = udf(getSmallest, IntegerType())
  
  raw_dataset_df = raw_dataset_df.withColumn('result',func_udf( 
    raw_dataset_df['type'], [ ( 0, raw_dataset_df['Icorr_LBT01_R'] ), (1, raw_dataset_df['Icorr_LBT01_S']) ] ));
  </code></pre>
  
  <p>I get following error</p>
  
  <pre><code>TypeError: Invalid argument, not a string or column: [(0, Column&lt;Icorr_LBT01_R&gt;), (1, Column&lt;Icorr_LBT01_S&gt;)] of type &lt;type 'list'&gt;. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.
  </code></pre>
  
  <p>Not sure what this means or how to fix.  I tried wrapping lit around each Column element but not clear what this should do and it doesn't work for me.  </p>