title: How can I load Avros in Spark using the schema on-board the Avro file(s)?
body: <p>I am running CDH 4.4 with Spark 0.9.0 from a Cloudera parcel.</p>
  
  <p>I have a bunch of Avro files that were created via Pig's AvroStorage UDF. I want to load these files in Spark, using a generic record or the schema onboard the Avro files. So far I've tried this:</p>
  
  <pre><code>import org.apache.avro.mapred.AvroKey
  import org.apache.avro.mapreduce.AvroKeyInputFormat
  import org.apache.hadoop.io.NullWritable
  import org.apache.commons.lang.StringEscapeUtils.escapeCsv
  
  import org.apache.hadoop.fs.Path
  import org.apache.hadoop.fs.FileSystem
  import org.apache.hadoop.conf.Configuration
  import java.net.URI
  import java.io.BufferedInputStream
  import java.io.File
  import org.apache.avro.generic.{GenericDatumReader, GenericRecord}
  import org.apache.avro.specific.SpecificDatumReader
  import org.apache.avro.file.DataFileStream
  import org.apache.avro.io.DatumReader
  import org.apache.avro.file.DataFileReader
  import org.apache.avro.mapred.FsInput
  
  val input = "hdfs://hivecluster2/securityx/web_proxy_mef/2014/05/29/22/part-m-00016.avro"
  val inURI = new URI(input)
  val inPath = new Path(inURI)
  
  val fsInput = new FsInput(inPath, sc.hadoopConfiguration)
  val reader =  new GenericDatumReader[GenericRecord]
  val dataFileReader = DataFileReader.openReader(fsInput, reader)
  val schemaString = dataFileReader.getSchema
  
  val buf = scala.collection.mutable.ListBuffer.empty[GenericRecord]
  while(dataFileReader.hasNext)  {
    buf += dataFileReader.next
  }
  sc.parallelize(buf)
  </code></pre>
  
  <p>This works for one file, but it can't scale - I am loading all the data into local RAM and then distributing it across the spark nodes from there.</p>