title: Protobuf-net lazy streaming deserialization of fields
body: <p><strong>Overall aim</strong>: To skip a very long field when deserializing, and when the field is accessed to read elements from it directly from the stream without loading the whole field. </p>
  
  <p><strong>Example classes</strong> The object being serialized/deserialized is <code>FatPropertyClass</code>.</p>
  
  <pre><code>[ProtoContract]
  public class FatPropertyClass
  {
      [ProtoMember(1)]
      private int smallProperty;
  
      [ProtoMember(2)]
      private FatArray2&lt;int&gt; fatProperty;
  
      [ProtoMember(3)]
      private int[] array;
  
      public FatPropertyClass()
      {
  
      }
  
      public FatPropertyClass(int sp, int[] fp)
      {
          smallProperty = sp;
          fatProperty = new FatArray&lt;int&gt;(fp);
      }
  
      public int SmallProperty
      {
          get { return smallProperty; }
          set { smallProperty = value; }
      }
  
      public FatArray&lt;int&gt; FatProperty
      {
          get { return fatProperty; }
          set { fatProperty = value; }
      }
  
      public int[] Array
      {
          get { return array; }
          set { array = value; }
      }
  }
  
  
  [ProtoContract]
  public class FatArray2&lt;T&gt;
  {
      [ProtoMember(1, DataFormat = DataFormat.FixedSize)]
      private T[] array;
      private Stream sourceStream;
      private long position;
  
      public FatArray2()
      {
      }
  
      public FatArray2(T[] array)
      {
          this.array = new T[array.Length];
          Array.Copy(array, this.array, array.Length);
      }
  
  
      [ProtoBeforeDeserialization]
      private void BeforeDeserialize(SerializationContext context)
      {
          position = ((Stream)context.Context).Position;
      }
  
      public T this[int index]
      {
          get
          {
              // logic to get the relevant index from the stream.
              return default(T);
          }
          set
          {
              // only relevant when full array is available for example.
          }
      }
  }
  </code></pre>
  
  <p>I can deserialize like so: <code>FatPropertyClass d = model.Deserialize(fileStream, null, typeof(FatPropertyClass), new SerializationContext() {Context = fileStream}) as FatPropertyClass;</code> where the <code>model</code> can be for example:</p>
  
  <pre><code>    RuntimeTypeModel model = RuntimeTypeModel.Create();
      MetaType mt = model.Add(typeof(FatPropertyClass), false);
      mt.AddField(1, "smallProperty");
      mt.AddField(2, "fatProperty");
      mt.AddField(3, "array");
      MetaType mtFat = model.Add(typeof(FatArray&lt;int&gt;), false);
  </code></pre>
  
  <p>This will skip the deserialization of <code>array</code> in <code>FatArray&lt;T&gt;</code>. However, I then need to read random elements from that array at a later time. One thing I tried is to remember the stream position before deserialization in the <code>BeforeDeserialize(SerializationContext context)</code> method of <code>FatArray2&lt;T&gt;</code>. As in the above code: <code>position = ((Stream)context.Context).Position;</code>. However this seems to always be the end of the stream. </p>
  
  <p>How can I remember the stream position where <code>FatProperty2</code> begins and how can I read from it at a random index? </p>
  
  <p><strong>Note</strong>: The parameter <code>T</code> in <code>FatArray2&lt;T&gt;</code> can be of other types marked with <code>[ProtoContract]</code>, not just primitives. Also there could be multiple properties of type <code>FatProperty2&lt;T&gt;</code> at various depths in the object graph. </p>
  
  <p><strong>Method 2</strong>: Serialize the field <code>FatProperty2&lt;T&gt;</code> after the serialization of the containing object. So, serialize <code>FatPropertyClass</code> with length prefix, then serialize with length prefix all fat arrays it contains. Mark all of these fat array properties with an attribute, and at deserialization we can remember the stream position for each of them. </p>
  
  <p>Then the question is how do we read primitives out of it? This works OK for classes using <code>T item = Serializer.DeserializeItems&lt;T&gt;(sourceStream, PrefixStyle.Base128, Serializer.ListItemTag).Skip(index).Take(1).ToArray();</code> to get the item at index <code>index</code>. But how does this work for primitives? An array of primitives does not seem to be able to be deserialized using <code>DeserializeItems</code>.</p>
  
  <p>Is <code>DeserializeItems</code> with LINQ used like that even OK? Does it do what I assume it does (internally skip through the stream to the correct element - at worst reading each length prefix and skipping it)?</p>
  
  <p>Regards,
  Iulian</p>