title: Creating PySpark UDFs from python method with numpy array input, to calculate and return a single float value
body: <p>As input I have a csv file with int values in it.</p>
  <pre><code>spark_df = spark.read.option(&quot;header&quot;, &quot;false&quot;).csv(&quot;../int_values.csv&quot;)
  
  df = spark_df.selectExpr(&quot;_c0 as something&quot;)
  
  _df = df.withColumn(&quot;values&quot;, df.something.cast(FloatType())).select(&quot;values&quot;)
  </code></pre>
  <p>I also have some python functions designed for numpy array inputs, that I need to apply on the Spark DataFrame.</p>
  <p><strong>The example one:</strong></p>
  <pre><code>def calc_sum(float_array):
      return np.sum(float_array)
  </code></pre>
  <p><strong>Real function:</strong></p>
  <pre><code>def calc_rms(float_array):
      return np.sqrt(np.mean(np.diff(float_array)**2))
  </code></pre>
  <p><em>For the 1. example you can use SQL sum like:</em></p>
  <pre><code>_df.groupBy().sum().collect()
  </code></pre>
  <p>But, what I need is a standard solution to transform these functions into Spark UDFs</p>
  <p>I tried many ways, like:</p>
  <pre><code>udf_sum = udf(lambda x : calc_sum(x), FloatType())
  
  _df.rdd.flatMap(udf_sum).collect()
  </code></pre>
  <p>but it always failed with:</p>
  <blockquote>
  <p>TypeError: Invalid argument, not a string or column:
  Row(values=1114.0) of type &lt;class 'pyspark.sql.types.Row'&gt;. For column
  literals, use 'lit', 'array', 'struct' or 'create_map' function.</p>
  </blockquote>
  <p><strong>Is it possible to transform the data in a way that works with these functions?</strong></p>
  <p><strong>DataFrame sample:</strong></p>
  <pre><code>In [6]: spark_df.show()
  +----+
  | _c0|
  +----+
  |1114|
  |1113|
  |1066|
  |1119|
  |1062|
  |1089|
  |1093|
  | 975|
  |1099|
  |1062|
  |1062|
  |1162|
  |1057|
  |1123|
  |1141|
  |1089|
  |1172|
  |1096|
  |1164|
  |1146|
  +----+
  only showing top 20 rows
  </code></pre>
  <p><strong>Expected output:</strong></p>
  <p>A Float value returned from the UDF.</p>
  <p>For the Sum function it should be clear.</p>