title: How to calculate mean and standard deviation given a PySpark DataFrame?
body: <p>I have PySpark DataFrame (<strong>not pandas</strong>) called <code>df</code> that is quite large to use <code>collect()</code>. Therefore the below-given code is not efficient. It was working with a smaller amount of data, however now it fails.</p>
  
  <pre><code>import numpy as np
  
  myList = df.collect()
  total = []
  for product,nb in myList:
      for p2,score in nb:
              total.append(score)
  mean = np.mean(total)
  std = np.std(total)
  </code></pre>
  
  <p>Is there any way to get <code>mean</code> and <code>std</code> as two variables by using <code>pyspark.sql.functions</code> or similar?</p>
  
  <pre><code>from pyspark.sql.functions import mean as mean_, std as std_
  </code></pre>
  
  <p>I could use <code>withColumn</code>, however, this approach applies the calculations row by row, and it does not return a single variable.</p>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>Sample content of <code>df</code>:</p>
  
  <pre><code>+----------+------------------+
  |product_PK|          products|
  +----------+------------------+
  |       680|[[691,1], [692,5]]|
  |       685|[[691,2], [692,2]]|
  |       684|[[691,1], [692,3]]|
  </code></pre>
  
  <p>I should calculate mean and standard deviation of <code>score</code> values, e.g. the value <code>1</code> in <code>[691,1]</code> is one of scores.</p>