title: Write data that can be read by ProtobufPigLoader from Elephant Bird
body: <p>For a <a href="http://code.google.com/p/fs-c/" rel="nofollow">project of mine</a>, I want to analyse around 2 TB of <a href="http://code.google.com/p/protobuf/" rel="nofollow">Protobuf</a> objects. I want to consume these objects in a Pig Script via the <a href="https://github.com/kevinweil/elephant-bird" rel="nofollow">"elephant bird"</a> library. However it is not totally clear to my how to write a file to HDFS so that it can be consumed by the ProtobufPigLoader class.</p>
  
  <p>This is what I have:</p>
  
  <p><strong>Pig script:</strong></p>
  
  <pre><code>  register ../fs-c/lib/*.jar // this includes the elephant bird library
    register ../fs-c/*.jar    
    raw_data = load 'hdfs://XXX/fsc-data2/XXX*' using com.twitter.elephantbird.pig.load.ProtobufPigLoader('de.pc2.dedup.fschunk.pig.PigProtocol.File');
  </code></pre>
  
  <p><strong>Import tool (parts of it):</strong></p>
  
  <pre><code>def getWriter(filenamePath: Path) : ProtobufBlockWriter[de.pc2.dedup.fschunk.pig.PigProtocol.File] = {
    val conf = new Configuration()
    val fs = FileSystem.get(filenamePath.toUri(), conf)
    val os = fs.create(filenamePath, true)
    val writer = new ProtobufBlockWriter[de.pc2.dedup.fschunk.pig.PigProtocol.File](os, classOf[de.pc2.dedup.fschunk.pig.PigProtocol.File])
    return writer
  }
  val writer = getWriter(new Path(filename))
  val builder = de.pc2.dedup.fschunk.pig.PigProtocol.File.newBuilder()
  writer.write(builder.build)
  writer.finish()
  writer.close()
  </code></pre>
  
  <p>The import tool runs fine. I had a few problems with the ProtobufPigLoader because I cannot use the hadoop-lzo compression library, and without a fix (see <a href="https://github.com/dmeister/elephant-bird/commit/806808e5a70271fa23cfb0e79c5210618d4e7fa1" rel="nofollow">here</a>) ProtobufPigLoader isn't working. The problem where I have problems is that <code>DUMP raw_data;</code> returns <code>Unable to open iterator for alias raw_data</code> and <code>ILLUSTRATE raw_data;</code> returns <code>No (valid) input data found!</code>. </p>
  
  <p>For me, it looks like the ProtobufBlockWriter data cannot be read by the ProtobufPigLoader. But what to use instead? <em>How to write data in a external tool to HDFS so that it can be processed by ProtobufPigLoader.</em></p>
  
  <p><em>Alternative question:</em> What to use instead? How to write pretty large objects to Hadoop to consume it with Pig? The objects are not very complex, but contain a large list of sub-objects in a list (repeated field in Protobuf). </p>
  
  <ul>
  <li>I want to avoid any text format or JSON because they are simply to large for my data. I expect that it would bloat up the data by a factor of 2 or 3 (lots of integer, lots of byte strings that I would need to encode as Base64)..</li>
  <li>I want to avoid normalizing the data so that the id of the main object is attached to each of the subobjects (this is what is done <a href="https://github.com/dmeister/fs-c/blob/master/contrib/fs-c-ir.pig" rel="nofollow">now</a>) because this also blows up the space consumption and makes joins necessary in the later processing.</li>
  </ul>
  
  <p>Updates:</p>
  
  <ul>
  <li>I didn't use the generation of protobuf loader classes, but use the reflection type loader</li>
  <li>The protobuf classes are in a jar that is registered. <code>DESCRIBE</code> correctly shows the types.</li>
  </ul>