title: pyspark: passing multiple dataframe fields to udf
body: <p>I am new to spark and python. Any help appreciated. </p>
  
  <p>I am having a UDF and created a spark dataframe with US zipcd, latitude and Longitude</p>
  
  <p>UDF:</p>
  
  <pre><code>import math
  def distance(origin, destination):
  lat1, lon1 = origin
  lat2, lon2 = destination
  radius = 6371 # km
  dlat = math.radians(lat2-lat1)
  dlon = math.radians(lon2-lon1)
  a = math.sin(dlat/2) * math.sin(dlat/2) + math.cos(math.radians(lat1)) \
      * math.cos(math.radians(lat2)) * math.sin(dlon/2) * math.sin(dlon/2)
  c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))
  d = radius * c
  return d
  </code></pre>
  
  <p>sample UDF output:</p>
  
  <pre><code>distance((101,121),(-121,-212)) 
  </code></pre>
  
  <p>15447.812243421227</p>
  
  <p>dataframe:</p>
  
  <pre><code>zip=spark.read.option("sep", ",").csv('wasb://hdiazurepoc@dsazurepoc.blob.core.windows.net/main/zip.txt')
  zip1=zip.select(zip._c0,zip._c1.cast("Double"),zip._c2.cast("Double"))
  </code></pre>
  
  <p>Sample zip1 data:</p>
  
  <pre><code>zip1.first()        
  </code></pre>
  
  <p>Row(_c0=u'00601', _c1=18.180555, _c2=-66.749961)</p>
  
  <p>Now I am trying to pass the latitude and longitude from the df zip1 to the udf distance, but I am getting error like "a float is required".
  I believe the udf is not getting the data from df fields, instead its reading the df column as constant value; and hence I am getting below error.</p>
  
  <pre><code>z=zip1.select(distance((zip1._c1,100.23),(zip1._c2,-99.21)))
  </code></pre>
  
  <p>Traceback (most recent call last):<br>
  File "", line 1, in <br>
  File "", line 5, in distance<br>
  TypeError: a float is required</p>
  
  <p>Please let me know the right way to pass the df fields to udf.</p>