title: Working with nested data in Spark SQL / High order functions
body: <p>I've recently started playing with Spark SQL (2.1) and I'm dealing with nested data.</p>
  
  <p>Here is my schema:</p>
  
  <pre><code> root
   |-- a: string (nullable = true)
   |-- b: map (nullable = true)
   |    |-- bb: string
   |    |-- bbb: string (valueContainsNull = true)
   |-- c: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- cc: map (nullable = true)
   |    |    |    |-- cca: string
   |    |    |    |-- ccb: struct (valueContainsNull = true)
   |    |    |    |    |-- member0: string (nullable = true)
   |    |    |    |    |-- member1: long (nullable = true)
   |    |    |-- ccc: map (nullable = true)
   |    |    |    |-- ccca: string
   |    |    |    |-- cccb: string (valueContainsNull = true)
   |    |    |-- cccc: map (nullable = true)
   |    |    |    |-- cccca: string
   |    |    |    |-- ccccb: string (valueContainsNull = true)
  </code></pre>
  
  <p>I am trying to filter my data as follows: keep all the rows where c.ccc.key == 'data'</p>
  
  <p>I found the very relevant function exists in databricks documentation. But I wonder if there is any similar outside of databricks notebooks? </p>
  
  <p><a href="https://docs.databricks.com/spark/latest/spark-sql/higher-order-functions-lambda-functions.html#exists-array-t-function-t-v-boolean-boolean" rel="nofollow noreferrer">https://docs.databricks.com/spark/latest/spark-sql/higher-order-functions-lambda-functions.html#exists-array-t-function-t-v-boolean-boolean</a></p>
  
  <p>I am willing to use sql or do it programatically, just not sure how as dataframes are not typed objects. </p>
  
  <p>Reading this email thread <a href="http://apache-spark-developers-list.1001551.n3.nabble.com/Will-higher-order-functions-in-spark-SQL-be-pushed-upstream-td21703.html" rel="nofollow noreferrer">http://apache-spark-developers-list.1001551.n3.nabble.com/Will-higher-order-functions-in-spark-SQL-be-pushed-upstream-td21703.html</a> it seems that high order functions from databricks will be available for all soon. But I wonder if there is an intermediate solution anybody can share?</p>