title: Defining a UDF that accepts an Array of objects in a Spark DataFrame?
body: <p>When working with Spark's DataFrames, User Defined Functions (UDFs) are required for mapping data in columns. UDFs require that argument types are explicitly specified. In my case, I need to manipulate a column that is made up of arrays of objects, and I do not know what type to use. Here's an example:</p>
  
  <pre><code>import sqlContext.implicits._
  
  // Start with some data. Each row (here, there's only one row) 
  // is a topic and a bunch of subjects
  val data = sqlContext.read.json(sc.parallelize(Seq(
    """
    |{
    |  "topic" : "pets",
    |  "subjects" : [
    |    {"type" : "cat", "score" : 10},
    |    {"type" : "dog", "score" : 1}
    |  ]
    |}
    """)))
  </code></pre>
  
  <p>It's relatively straightforward to use the built-in <code>org.apache.spark.sql.functions</code> to perform basic operations on the data in the columns</p>
  
  <pre><code>import org.apache.spark.sql.functions.size
  data.select($"topic", size($"subjects")).show
  
  +-----+--------------+
  |topic|size(subjects)|
  +-----+--------------+
  | pets|             2|
  +-----+--------------+
  </code></pre>
  
  <p>and it's generally easy to write custom UDFs to perform arbitrary operations</p>
  
  <pre><code>import org.apache.spark.sql.functions.udf
  val enhance = udf { topic : String =&gt; topic.toUpperCase() }
  data.select(enhance($"topic"), size($"subjects")).show 
  
  +----------+--------------+
  |UDF(topic)|size(subjects)|
  +----------+--------------+
  |      PETS|             2|
  +----------+--------------+
  </code></pre>
  
  <p>But what if I want to use a UDF to manipulate the array of objects in the "subjects" column? What type do I use for the argument in the UDF? For example, if I want to reimplement the size function, instead of using the one provided by spark:</p>
  
  <pre><code>val my_size = udf { subjects: Array[Something] =&gt; subjects.size }
  data.select($"topic", my_size($"subjects")).show
  </code></pre>
  
  <p>Clearly <code>Array[Something]</code> does not work... what type should I use!? Should I ditch <code>Array[]</code> altogether? Poking around tells me <code>scala.collection.mutable.WrappedArray</code> may have something to do with it, but still there's another type I need to provide. </p>