title: What is the fastest way to parse text with custom delimiters and some very, very large field values in C#?
body: <p>I've been trying to deal with some delimited text files that have non standard delimiters (not comma/quote or tab delimited).  The delimiters are random ASCII characters that don't show up often between the delimiters.  After searching around, I've seem to have only found no solutions in .NET will suit my needs and the custom libraries that people have written for this seem to have some flaws when it comes to gigantic input (4GB file with some field values having very easily several million characters).  </p>
  
  <p>While this seems to be a bit extreme, it is actually a standard in the Electronic Document Discovery (EDD) industry for some review software to have field values that contain the full contents of a document.  For reference, I've previously done this in python using the csv module with no problems.</p>
  
  <p>Here's an example input:</p>
  
  <pre><code>Field delimiter = 
  quote character = þ
  
  þFieldName1þþFieldName2þþFieldName3þþFieldName4þ
  þValue1þþValue2þþValue3þþSomeVery,Very,Very,Large value(5MB or so)þ
  ...etc...
  </code></pre>
  
  <p>Edit:
  So I went ahead and created a delimited file parser from scratch.  I'm kind of weary using this solution as it may be prone to bugs.  It also doesn't feel "elegant" or correct to have to write my own parser for a task like this.  I also have a feeling that I probably didn't have to write a parser from scratch for this anyway.  </p>