title: how to write a valid decoding file based on a given .proto, reading from a .pb
body: <p>Based on the answer to this <a href="https://stackoverflow.com/questions/22105477/protocol-message-end-group-tag-did-not-match-expected-tag">question</a> I'm thinking that I've provided my .pb file with a "faulty decoder". </p>
  
  <p><a href="http://iesl.cs.umass.edu/riedel/ecml/heldout_relations.tgz" rel="nofollow noreferrer">This is the data I'm trying to decode</a>.</p>
  
  <p><a href="http://iesl.cs.umass.edu/riedel/ecml/Document.proto" rel="nofollow noreferrer">This is my .proto file</a>.</p>
  
  <p>Based on the <a href="https://github.com/google/protobuf/blob/master/examples/ListPeople.java" rel="nofollow noreferrer">ListPeople.java</a> example provided in the <a href="https://developers.google.com/protocol-buffers/docs/javatutorial?csw=1" rel="nofollow noreferrer">Java tutorial documentation</a>, I tried to write something similar to start picking apart that data, I wrote this:</p>
  
  <pre><code>import cc.refectorie.proj.relation.protobuf.DocumentProtos.Document;
  import cc.refectorie.proj.relation.protobuf.DocumentProtos.Document.Sentence;
  
  import java.io.FileInputStream;
  import java.io.IOException;
  import java.io.PrintStream;
  
  
  public class ListDocument
  {
      // Iterates though all people in the AddressBook and prints info about them.
      static void Print(Document document)
      {
          for ( Sentence sentence: document.getSentencesList() )
          {
              for(int i=0; i &lt; sentence.getTokensCount(); i++)
              {
                  System.out.println(" getTokens(" + i + ": " + sentence.getTokens(i) );
              }
          }
      }
  
      // Main function:  Reads the entire address book from a file and prints all
      //   the information inside.
      public static void main(String[] args) throws Exception {
          if (args.length != 1) {
              System.err.println("Usage:  ListPeople ADDRESS_BOOK_FILE");
              System.exit(-1);
          }
  
          // Read the existing address book.
          Document addressBook =
                  Document.parseFrom(new FileInputStream(args[0]));
  
          Print(addressBook);
      }
  }
  </code></pre>
  
  <p>But when I run that I get this error</p>
  
  <pre><code>Exception in thread "main" com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
      at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
      at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
      at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:194)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:210)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:215)
      at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
      at cc.refectorie.proj.relation.protobuf.DocumentProtos$Document.parseFrom(DocumentProtos.java:4770)
      at ListDocument.main(ListDocument.java:40)
  </code></pre>
  
  <p>so, as I said above I think that has to do with me not properly defining the decoder. Is there some way to look at the .proto file I'm trying to use and figure out a way to just read off all that data?</p>
  
  <p>Is there some way to look at that .proto file and see what I'm doing wrong?</p>
  
  <p>These are the first few lines of the file I want to read:</p>
  
  <pre><code>Ü
  &amp;/guid/9202a8c04000641f8000000003221072&amp;/guid/9202a8c04000641f80000000004cfd50NA"Ö
  
  S/m/vinci8/data1/riedel/projects/relation/kb/nyt1/docstore/2007-joint/1850511.xml.pb„€€€øÿÿÿÿƒ€€€øÿÿÿÿ"PERSON-&gt;PERSON"'inverse_false|PERSON|on bass and|PERSON"/inverse_false|with|PERSON|on bass and|PERSON|on"7inverse_false|, with|PERSON|on bass and|PERSON|on drums"$inverse_false|PERSON|IN NN CC|PERSON",inverse_false|with|PERSON|IN NN CC|PERSON|on"4inverse_false|, with|PERSON|IN NN CC|PERSON|on drums"`str:Dave[NMOD]-&gt;|PERSON|[PMOD]-&gt;with[ADV]-&gt;was[ROOT]&lt;-on[PRD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;Barry"]str:Dave[NMOD]-&gt;|PERSON|[PMOD]-&gt;with[ADV]-&gt;was[ROOT]&lt;-on[PRD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;on"Rstr:Dave[NMOD]-&gt;|PERSON|[PMOD]-&gt;with[ADV]-&gt;was[ROOT]&lt;-on[PRD]&lt;-bass[PMOD]&lt;-|PERSON"Adep:[NMOD]-&gt;|PERSON|[PMOD]-&gt;[ADV]-&gt;[ROOT]&lt;-[PRD]&lt;-[PMOD]&lt;-|PERSON"dir:-&gt;|PERSON|-&gt;-&gt;&lt;-&lt;-&lt;-|PERSON"Sstr:PERSON|[PMOD]-&gt;with[ADV]-&gt;was[ROOT]&lt;-on[PRD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;Barry"Adep:PERSON|[PMOD]-&gt;[ADV]-&gt;[ROOT]&lt;-[PRD]&lt;-[PMOD]&lt;-|PERSON|[NMOD]-&gt;"dir:PERSON|-&gt;-&gt;&lt;-&lt;-&lt;-|PERSON|-&gt;"Pstr:PERSON|[PMOD]-&gt;with[ADV]-&gt;was[ROOT]&lt;-on[PRD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;on"Adep:PERSON|[PMOD]-&gt;[ADV]-&gt;[ROOT]&lt;-[PRD]&lt;-[PMOD]&lt;-|PERSON|[NMOD]-&gt;"dir:PERSON|-&gt;-&gt;&lt;-&lt;-&lt;-|PERSON|-&gt;"Estr:PERSON|[PMOD]-&gt;with[ADV]-&gt;was[ROOT]&lt;-on[PRD]&lt;-bass[PMOD]&lt;-|PERSON*ŒThe occasion was suitably exceptional : a reunion of the 1970s-era Sam Rivers Trio , with Dave Holland on bass and Barry Altschul on drums ."¬
  S/m/vinci8/data1/riedel/projects/relation/kb/nyt1/docstore/2007-joint/1849689.xml.pb†€€€øÿÿÿÿ…€€€øÿÿÿÿ"PERSON-&gt;PERSON"'inverse_false|PERSON|on bass and|PERSON"/inverse_false|with|PERSON|on bass and|PERSON|on"7inverse_false|, with|PERSON|on bass and|PERSON|on drums"$inverse_false|PERSON|IN NN CC|PERSON",inverse_false|with|PERSON|IN NN CC|PERSON|on"4inverse_false|, with|PERSON|IN NN CC|PERSON|on drums"cstr:Dave[NMOD]-&gt;|PERSON|[PMOD]-&gt;with[NMOD]-&gt;Trio[NULL]&lt;-on[NMOD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;Barry"`str:Dave[NMOD]-&gt;|PERSON|[PMOD]-&gt;with[NMOD]-&gt;Trio[NULL]&lt;-on[NMOD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;on"Ustr:Dave[NMOD]-&gt;|PERSON|[PMOD]-&gt;with[NMOD]-&gt;Trio[NULL]&lt;-on[NMOD]&lt;-bass[PMOD]&lt;-|PERSON"Cdep:[NMOD]-&gt;|PERSON|[PMOD]-&gt;[NMOD]-&gt;[NULL]&lt;-[NMOD]&lt;-[PMOD]&lt;-|PERSON"dir:-&gt;|PERSON|-&gt;-&gt;&lt;-&lt;-&lt;-|PERSON"Vstr:PERSON|[PMOD]-&gt;with[NMOD]-&gt;Trio[NULL]&lt;-on[NMOD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;Barry"Cdep:PERSON|[PMOD]-&gt;[NMOD]-&gt;[NULL]&lt;-[NMOD]&lt;-[PMOD]&lt;-|PERSON|[NMOD]-&gt;"dir:PERSON|-&gt;-&gt;&lt;-&lt;-&lt;-|PERSON|-&gt;"Sstr:PERSON|[PMOD]-&gt;with[NMOD]-&gt;Trio[NULL]&lt;-on[NMOD]&lt;-bass[PMOD]&lt;-|PERSON|[NMOD]-&gt;on"Cdep:PERSON|[PMOD]-&gt;[NMOD]-&gt;[NULL]&lt;-[NMOD]&lt;-[PMOD]&lt;-|PERSON|[NMOD]-&gt;"dir:PERSON|-&gt;-&gt;&lt;-&lt;-&lt;-|PERSON|-&gt;"Hstr:PERSON|[PMOD]-&gt;with[NMOD]-&gt;Trio[NULL]&lt;-on[NMOD]&lt;-bass[PMOD]&lt;-|PERSON*ÊTonight he brings his energies and expertise to the Miller Theater for the festival 's thrilling finale : a reunion of the 1970s Sam Rivers Trio , with Dave Holland on bass and Barry Altschul on drums .â
  &amp;/guid/9202a8c04000641f80000000004cfd50&amp;/guid/9202a8c04000641f8000000003221072NA"Ù
  </code></pre>
  
  <hr>
  
  <p><strong>EDIT</strong></p>
  
  <hr>
  
  <p>This is a file another researcher used to parse these files, so I was told, is it possible that I could use this?</p>
  
  <pre><code>package edu.stanford.nlp.kbp.slotfilling.multir;
  
  import java.io.BufferedInputStream;
  import java.io.FileInputStream;
  import java.io.IOException;
  import java.io.InputStream;
  import java.util.ArrayList;
  import java.util.Collection;
  import java.util.HashMap;
  import java.util.HashSet;
  import java.util.List;
  import java.util.Map;
  import java.util.Set;
  import java.util.zip.GZIPInputStream;
  
  import edu.stanford.nlp.kbp.slotfilling.classify.MultiLabelDataset;
  import edu.stanford.nlp.kbp.slotfilling.common.Log;
  import edu.stanford.nlp.kbp.slotfilling.multir.DocumentProtos.Relation;
  import edu.stanford.nlp.stats.ClassicCounter;
  import edu.stanford.nlp.stats.Counter;
  import edu.stanford.nlp.util.ErasureUtils;
  import edu.stanford.nlp.util.HashIndex;
  import edu.stanford.nlp.util.Index;
  
  /**
   * Converts Hoffmann's data in protobuf format to our MultiLabelDataset
   * @author Mihai
   *
   */
  public class ProtobufToMultiLabelDataset {
    static class RelationAndMentions {
      String arg1;
      String arg2;
      Set&lt;String&gt; posLabels;
      Set&lt;String&gt; negLabels;
      List&lt;Mention&gt; mentions;
  
      public RelationAndMentions(String types, String a1, String a2) {
        arg1 = a1;
        arg2 = a2;
        String [] rels = types.split(",");
        posLabels = new HashSet&lt;String&gt;();
        for(String r: rels){
          if(! r.equals("NA")) posLabels.add(r.trim());
        }
        negLabels = new HashSet&lt;String&gt;(); // will be populated later
        mentions = new ArrayList&lt;Mention&gt;();
      }
    };
  
    static class Mention {
      List&lt;String&gt; features;
      public Mention(List&lt;String&gt; feats) {
        features = feats;
      }
    }
  
      public static void main(String[] args) throws Exception {
        String input = args[0];
  
        InputStream is = new GZIPInputStream(
          new BufferedInputStream
          (new FileInputStream(input)));
  
        toMultiLabelDataset(is);
        is.close();
      }
  
      public static MultiLabelDataset&lt;String, String&gt; toMultiLabelDataset(InputStream is) throws IOException {
        List&lt;RelationAndMentions&gt; relations = toRelations(is, true);
        MultiLabelDataset&lt;String, String&gt; dataset = toDataset(relations);
        return dataset;
      }
  
      public static void toDatums(InputStream is,
          List&lt;List&lt;Collection&lt;String&gt;&gt;&gt; relationFeatures,
          List&lt;Set&lt;String&gt;&gt; labels) throws IOException {
        List&lt;RelationAndMentions&gt; relations = toRelations(is, false);
        toDatums(relations, relationFeatures, labels);
      }
  
      private static void toDatums(List&lt;RelationAndMentions&gt; relations,
          List&lt;List&lt;Collection&lt;String&gt;&gt;&gt; relationFeatures,
        List&lt;Set&lt;String&gt;&gt; labels) {
      for(RelationAndMentions rel: relations) {
        labels.add(rel.posLabels);
        List&lt;Collection&lt;String&gt;&gt; mentionFeatures = new ArrayList&lt;Collection&lt;String&gt;&gt;();
        for(int i = 0; i &lt; rel.mentions.size(); i ++){
          mentionFeatures.add(rel.mentions.get(i).features);
        }
        relationFeatures.add(mentionFeatures);
      }
      assert(labels.size() == relationFeatures.size());
      }
  
      public static List&lt;RelationAndMentions&gt; toRelations(InputStream is, boolean generateNegativeLabels) throws IOException {
        //
        // Parse the protobuf
        //
      // all relations are stored here
      List&lt;RelationAndMentions&gt; relations = new ArrayList&lt;RelationAndMentions&gt;();
      // all known relations (without NIL)
      Set&lt;String&gt; relTypes = new HashSet&lt;String&gt;();
      Map&lt;String, Map&lt;String, Set&lt;String&gt;&gt;&gt; knownRelationsPerEntity =
        new HashMap&lt;String, Map&lt;String,Set&lt;String&gt;&gt;&gt;();
      Counter&lt;Integer&gt; labelCountHisto = new ClassicCounter&lt;Integer&gt;();
      Relation r = null;
      while ((r = Relation.parseDelimitedFrom(is)) != null) {
        RelationAndMentions relation = new RelationAndMentions(
            r.getRelType(), r.getSourceGuid(), r.getDestGuid());
        labelCountHisto.incrementCount(relation.posLabels.size());
        relTypes.addAll(relation.posLabels);
        relations.add(relation);
  
        for(int i = 0; i &lt; r.getMentionCount(); i ++) {
          DocumentProtos.Relation.RelationMentionRef mention = r.getMention(i);
          // String s = mention.getSentence();
          relation.mentions.add(new Mention(mention.getFeatureList()));
        }
  
        for(String l: relation.posLabels) {
          addKnownRelation(relation.arg1, relation.arg2, l, knownRelationsPerEntity);
        }
      }
      Log.severe("Loaded " + relations.size() + " relations.");
      Log.severe("Found " + relTypes.size() + " relation types: " + relTypes);
      Log.severe("Label count histogram: " + labelCountHisto);
  
      Counter&lt;Integer&gt; slotCountHisto = new ClassicCounter&lt;Integer&gt;();
      for(String e: knownRelationsPerEntity.keySet()) {
        slotCountHisto.incrementCount(knownRelationsPerEntity.get(e).size());
      }
      Log.severe("Slot count histogram: " + slotCountHisto);
      int negativesWithKnownPositivesCount = 0, totalNegatives = 0;
      for(RelationAndMentions rel: relations) {
        if(rel.posLabels.size() == 0) {
          if(knownRelationsPerEntity.get(rel.arg1) != null &amp;&amp;
             knownRelationsPerEntity.get(rel.arg1).size() &gt; 0) {
            negativesWithKnownPositivesCount ++;
          }
          totalNegatives ++;
        }
      }
      Log.severe("Found " + negativesWithKnownPositivesCount + "/" + totalNegatives +
          " negative examples with at least one known relation for arg1.");
  
      Counter&lt;Integer&gt; mentionCountHisto = new ClassicCounter&lt;Integer&gt;();
      for(RelationAndMentions rel: relations) {
        mentionCountHisto.incrementCount(rel.mentions.size());
        if(rel.mentions.size() &gt; 100)
          Log.fine("Large relation: " + rel.mentions.size() + "\t" + rel.posLabels);
      }
      Log.severe("Mention count histogram: " + mentionCountHisto);
  
      //
      // Detect the known negatives for each source entity
      //
      if(generateNegativeLabels) {
        for(RelationAndMentions rel: relations) {
          Set&lt;String&gt; negatives = new HashSet&lt;String&gt;(relTypes);
          negatives.removeAll(rel.posLabels);
          rel.negLabels = negatives;
        }
      }
  
      return relations;
      }
  
      private static MultiLabelDataset&lt;String, String&gt; toDataset(List&lt;RelationAndMentions&gt; relations) {
      int [][][] data = new int[relations.size()][][];
      Index&lt;String&gt; featureIndex = new HashIndex&lt;String&gt;();
      Index&lt;String&gt; labelIndex = new HashIndex&lt;String&gt;();
      Set&lt;Integer&gt; [] posLabels = ErasureUtils.&lt;Set&lt;Integer&gt; []&gt;uncheckedCast(new Set[relations.size()]);
      Set&lt;Integer&gt; [] negLabels = ErasureUtils.&lt;Set&lt;Integer&gt; []&gt;uncheckedCast(new Set[relations.size()]);
  
      int offset = 0, posCount = 0;
      for(RelationAndMentions rel: relations) {
        Set&lt;Integer&gt; pos = new HashSet&lt;Integer&gt;();
        Set&lt;Integer&gt; neg = new HashSet&lt;Integer&gt;();
        for(String l: rel.posLabels) {
          pos.add(labelIndex.indexOf(l, true));
        }
        for(String l: rel.negLabels) {
          neg.add(labelIndex.indexOf(l, true));
        }
        posLabels[offset] = pos;
        negLabels[offset] = neg;
        int [][] group = new int[rel.mentions.size()][];
        for(int i = 0; i &lt; rel.mentions.size(); i ++){
          List&lt;String&gt; sfeats = rel.mentions.get(i).features;
          int [] features = new int[sfeats.size()];
          for(int j = 0; j &lt; sfeats.size(); j ++) {
            features[j] = featureIndex.indexOf(sfeats.get(j), true);
          }
          group[i] = features;
        }
        data[offset] = group;
        posCount += posLabels[offset].size();
        offset ++;
      }
  
      Log.severe("Creating a dataset with " + data.length + " datums, out of which " + posCount + " are positive.");
      MultiLabelDataset&lt;String, String&gt; dataset = new MultiLabelDataset&lt;String, String&gt;(
          data, featureIndex, labelIndex, posLabels, negLabels);
      return dataset;
      }
  
      private static void addKnownRelation(String arg1, String arg2, String label,
          Map&lt;String, Map&lt;String, Set&lt;String&gt;&gt;&gt; knownRelationsPerEntity) {
        Map&lt;String, Set&lt;String&gt;&gt; myRels = knownRelationsPerEntity.get(arg1);
        if(myRels == null) {
          myRels = new HashMap&lt;String, Set&lt;String&gt;&gt;();
          knownRelationsPerEntity.put(arg1, myRels);
        }
        Set&lt;String&gt; mySlots = myRels.get(label);
        if(mySlots == null) {
          mySlots = new HashSet&lt;String&gt;();
          myRels.put(label, mySlots);
        }
        mySlots.add(arg2);
      }
  }
  </code></pre>