title: How to best read and UTF-8-decode byte buffers?
body: <p>I have a <a href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx" rel="nofollow"><code>Stream</code></a> that produces <a href="http://nl.wikipedia.org/wiki/UTF-8" rel="nofollow">UTF-8</a> encoded strings. The strings represent XML documents that I need to parse. The stream is obtained from a <a href="http://msdn.microsoft.com/en-us/library/system.net.sockets.tcpclient.aspx" rel="nofollow"><code>TcpClient</code></a>.</p>
  
  <p>Suppose I read the stream into buffers of size 64 (a little small, I know). Passing these 64 byte buffers directly to the string decoding step could fail because some UTF-8 encoded characters may be split along the 64 byte boundary. The buffer may end with the first two bytes of a character and the next buffer has the last byte for this character.</p>
  
  <p>What I do now, is concatenate buffers until I perform a read that doesn't read the full 64 bytes, indicating that I have read to the end of something (in my case, an XML document). However, once in a while, an XML documents I read ends exactly at the 64 byte boundary. In such a case, I do not know I can pass the byte array to the decoding step (and I need to wait for the next document).</p>
  
  <p>I realize I can lower the chances by increasing the buffer size. However, a small chance always remains that it happens. I could also increase the buffer size such that any XML document I encounter will fit, but I just wonder whether there is another solution, somehow detecting from the byte stream where the character boundaries are.</p>