title: Calculate MAPE and apply to PySpark grouped Dataframe [@pandas_udf]
body: <p>Goal:
  Calculate <code>mean_absolute_percentage_error</code> (MAPE) for each unique <code>ID</code>.</p>
  <ul>
  <li><code>y</code> - real value</li>
  <li><code>yhat</code> - predicted value</li>
  </ul>
  <p>Sample PySpark Dataframe: <code>join_df</code></p>
  <pre><code>+----------+----------+-------+---------+----------+----------+
  |        ID|        ds|      y|     yhat|yhat_upper|yhat_lower|
  +----------+----------+-------+---------+----------+----------+
  |    Ax849b|2021-07-01|1165.59| 1298.809| 1939.1261| 687.48206|
  |    Ax849b|2021-07-02|1120.69| 1295.552| 1892.4929|   693.786|
  |    Ax849b|2021-07-03|1120.69| 1294.079| 1923.0253|  664.1514|
  |    Ax849b|2021-07-04|1120.69|1295.0399| 1947.6392|  639.4879|
  |    Bz383J|2021-07-03|1108.71|1159.4934| 1917.6515| 652.76624|
  |    Bz383J|2021-07-04|1062.77|1191.2385| 1891.9268|  665.9529|
  +----------+----------+-------+---------+----------+----------+
  
  </code></pre>
  <pre><code>final_schema =StructType([
    StructField('ds',DateType()),
    StructField('ID',IntegerType()),
    StructField('y',FloatType()),
    StructField('yhat',FloatType()),
    StructField('yhat_upper',FloatType()),
    StructField('yhat_lower',FloatType()),
    StructField('mape',FloatType())
    ])
  
  </code></pre>
  <p>I have tried by creating an <code>uff</code> and applyed it on <code>ID</code>s using <code>apply</code> function.</p>
  <pre><code>from sklearn.metrics import mean_absolute_percentage_error
  from pyspark.sql.functions import pandas_udf, PandasUDFType
  
  @pandas_udf(final_schema, PandasUDFType.GROUPED_MAP)
  def gr_mape_val(join_df):
    
    mape = mean_absolute_percentage_error(join_df[&quot;y&quot;], join_df[&quot;yhat&quot;]) 
    join_df['mape'] = mape
    
    return join_df
  
  df_apply = join_df.groupby('ID').applyInPandas(gr_mape_val, final_schema)
  df_apply.show()
  
  </code></pre>
  <p>However, I am getting the error:</p>
  <blockquote>
  <p>PythonException: 'TypeError: Return type of the user-defined function should be pandas.DataFrame, but is &lt;class 'numpy.float32'&gt;'</p>
  </blockquote>
  <p>I understand that I am requesting for MAPE as numpy output and it should be dataframe. But I am sure if I know what exactly needs to be done differently in order to get MAPE for each <code>ID</code>.</p>