title: Unable to find encoder for type stored in Dataset when attempting to perform flatMap on a DataFrame in Spark 2.0
body: <p>I keep getting the following compile time error:</p>
  
  <pre>
  Unable to find encoder for type stored in a Dataset.  
  Primitive types (Int, String, etc) and Product types (case classes) 
  are supported by importing spark.implicits._  
  Support for serializing other types will be added in future releases.
  </pre>
  
  <p>I've just upgraded from Spark v1.6 to v2.0.2, and a whole bunch of code using <code>DataFrame</code> are complaining about this error. The code where it is complaining looks like the following.</p>
  
  <pre><code>def doSomething(data: DataFrame): Unit = {
   data.flatMap(row =&gt; {
    ...
   })
   .reduceByKey(_ + _)
   .sortByKey(ascending = false)
  }
  </code></pre>
  
  <p>Previous SO posts suggest to </p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/34715611/why-is-the-error-unable-to-find-encoder-for-type-stored-in-a-dataset-when-enco">pull out the case class (defined in object)</a></li>
  <li><a href="https://stackoverflow.com/questions/38664972/why-is-unable-to-find-encoder-for-type-stored-in-a-dataset-when-creating-a-dat">perform implicit imports</a></li>
  </ul>
  
  <p>However, I don't have any case classes, as I am using <code>DataFrame</code> which is equal to <code>DataSet[Row]</code>, and also, I've inlined the 2 implicit imports as follows, without any help to get rid of this message.</p>
  
  <pre><code>val sparkSession: SparkSession = ???
  val sqlContext: SQLContext = ???
  
  import sparkSession.implicits._
  import sqlContext.implicits._
  </code></pre>
  
  <p>Note that I've looked at the documentation for <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset" rel="nofollow noreferrer">DataSet</a> and <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Encoder" rel="nofollow noreferrer">Encoder</a>. The docs says something like the following.</p>
  
  <pre>
  Scala
  
  Encoders are generally created automatically through implicits from a 
  SparkSession, or can be explicitly created by calling static methods on 
  Encoders.
  
  import spark.implicits._
  
  val ds = Seq(1, 2, 3).toDS() // implicitly provided (spark.implicits.newIntEncoder)
  </pre>
  
  <p>However, my method doesn't have access to <code>SparkSession</code>. Also, when I try that line <code>import spark.implicits._</code>, IntelliJ can't even find it. When I say that my DataFrame is a DataSet[Row], I really do mean it. </p>
  
  <p>This question is marked as a possible <a href="https://stackoverflow.com/questions/39433419/encoder-error-while-trying-to-map-dataframe-row-to-updated-row">duplicate</a>, but please let me clarify.</p>
  
  <ul>
  <li>I have no case class or business object associated. </li>
  <li>I am using .flatMap while the other question is using .map </li>
  <li>implicit imports do not seem to help</li>
  <li>passing a RowEncoder produces a compile-time error e.g. <code>data.flatMap(row =&gt; { ... }, RowEncoder(data.schema))</code> (too many arguments)</li>
  </ul>
  
  <p>I'm reading the other posts, and let me add, I guess I don't know how this new Spark 2.0 Datasets/DataFrame API is supposed to work. In the Spark shell, the code below works. Note that I <a href="https://github.com/databricks/spark-csv" rel="nofollow noreferrer">start the spark shell</a> like this <code>$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.10:1.5.0</code></p>
  
  <pre><code>val schema = StructType(Array(
   StructField("x1", StringType, true),
   StructField("x2", StringType, true),
   StructField("x3", StringType, true),
   StructField("x4", StringType, true),
   StructField("x5", StringType, true)))
  
  val df = sqlContext.read
   .format("com.databricks.spark.csv")
   .option("header", "true")
   .schema(schema)
   .load("/Users/jwayne/Downloads/mydata.csv")
  
  df.columns.map(col =&gt; {
   df.groupBy(col)
     .count()
     .map(_.getString(0))
     .collect()
     .toList
   })
   .toList
  </code></pre>
  
  <p>However, when I run this as a part of a test unit, I get the same unable to find encoder error. Why does this work in the shell but not in my test units? </p>
  
  <p>In the shell, I typed in <code>:imports</code> and <code>:implicits</code> and placed those in my scala files/source, but that doesn't help either. </p>