title: Pandas .apply Loc to Pyspark
body: <p>I Have done some work with Pandas. Now I need to make the same on Pyspark and this task seem to be tricky !</p>
  
  <p>Here is my code:</p>
  
  <pre><code>import pandas as pd
  def av_years(df,start,end):
  return df.loc[df['year'].isin(range(start,end+1))]['B'].mean() 
  </code></pre>
  
  <p>I then created a dataframe:</p>
  
  <pre><code>raw_data = {
      'year': [2010,2011,2012,2013],
      'B': [2,3,5,4],'startyear':[2012,2010,2011,2011],'endyear':
       [2012,2013,2013,2013]}
       df = pd.DataFrame(raw_data)
       df
  </code></pre>
  
  <p>This is my result:</p>
  
  <pre><code>    B   endyear startyear   year
      0   2   2012    2012    2010
      1   3   2013    2010    2011
      2   5   2013    2011    2012
      3   4   2013    2011    2013
  </code></pre>
  
  <p>The final step is t create a new column derived from the dataframe:</p>
  
  <pre><code>  df['av'] = df.apply(lambda row: av_years(df,row['startyear'], 
    row['endyear']), axis=1)
    df
  </code></pre>
  
  <p>My final result is :</p>
  
  <pre><code> B  endyear startyear   year    av
   0  2   2012    2012    2010    5.0
   1  3   2013    2010    2011    3.5
   2  5   2013    2011    2012    4.0
   3  4   2013    2011    2013    4.0
  </code></pre>
  
  <p>I need to obtain the same table using PySpark ! some advice ?</p>
  
  <p>Thank you</p>