title: How to define and use a User-Defined Aggregate Function in Spark SQL?
body: <p>I know how to write a UDF in Spark SQL:</p>
  
  <pre><code>def belowThreshold(power: Int): Boolean = {
          return power &lt; -40
        }
  
  sqlContext.udf.register("belowThreshold", belowThreshold _)
  </code></pre>
  
  <p>Can I do something similar to define an aggregate function? How is this done?</p>
  
  <p>For context, I want to run the following SQL query:</p>
  
  <pre><code>val aggDF = sqlContext.sql("""SELECT span, belowThreshold(opticalReceivePower), timestamp
                                      FROM ifDF
                                      WHERE opticalReceivePower IS NOT null
                                      GROUP BY span, timestamp
                                      ORDER BY span""")
  </code></pre>
  
  <p>It should return something like</p>
  
  <p><code>Row(span1, false, T0)</code></p>
  
  <p>I want the aggregate function to tell me if there's any values for <code>opticalReceivePower</code> in the groups defined by <code>span</code> and <code>timestamp</code> which are below the threshold. Do I need to write my UDAF differently to the UDF I pasted above?</p>