Take the 2-minute tour ×

I have a program that reads each line of file, extracting data according to specific format, defined by a regular expression. Instead of calling Match() multiple times against each line in the file, I could call Match() against the entire contents of the file. Which is a more efficient solution?

The latter choice would require the RegexOptions.Multiline option.


The file is specified by the end-user so it could be large (~37000 lines, ~2MB). It is not necessary for every line to contain a valid entry.

The regular expression I'm using is ^\s*(OPTL_\w*)\s*=>\s*(\d+)\s*$. For example, this would match the a line consisting of the text OPTL_Example => 123, but would not match a line consisting of the text FooBar => 999.

share|improve this question
Is it a really large file? –  Diego Nov 28 '12 at 14:41
please try to improve your question.what have you tried!.be specific...give example –  Anirudha Nov 28 '12 at 14:44
Do you have any metrics about your average file size? This is what you should optimize for. –  Ryan Gates Nov 28 '12 at 15:09

4 Answers 4

up vote 2 down vote accepted

So depends on if you are optimizing for speed or stability.

If this is an end user app and don't have control of file size or memory then I would take the safe route and read line by line to protect memory. Clearly build the regex outside the loop so you are just calling .Match in the loop. ReadLine is pretty fast.

Could set up some parallel processing so it is reading the next line while it is performing the parse. But that simple regex would be so fast not sure it would be faster. Line at a time or entire file the disk IO to read the file is most likely the slowest operation.

If this is a server app with limited distribution and speed it critical then read it all in.

share|improve this answer
My 5 cents :: With just 2MB of data I would not consider to use parallel processing. It would be a waist of resources with no possible improvement. It actually may be slower due to multi-thread management. –  Ωmega Nov 28 '12 at 18:37
@omega but there is no guarantee of 2 mb. File is specified by the user and users can do strange things. And you cannot be sure there is no possibility for improvement with parallel. I suspect not but if they are on 4.5 it is getting pretty efficient at parallel. –  Blam Nov 28 '12 at 19:15
An option could be to check the size of the file prior to processing and decide whether to use parallel processing based on that. –  Dan Stevens Nov 29 '12 at 11:07
@IAmAI If parallel is faster or slower I don't think would be dependent on the file size. There is overhead to parallel and for a simple task the overhead may not be faster than the task. –  Blam Nov 29 '12 at 13:24

There is no general and/or correct answer for it, as it depends on many factors. The major one i speed of your I/O. Why don't you just test both solutions? With size of 2MB I would expect to work with entire content to be faster and more efficient.

share|improve this answer

It depends on memory constraints you need. If you have multiple regexes you can run on the file as whole, it is as efficient to keep the whole file in memory. However if your regexes modify lines (and then repeat this process, with cascading regexes that depend on one another) I'd go for a line by line solution.

share|improve this answer

Choosing the line by line solution could allow you to run regexes in parallel. The question is if all the overhead with parallel processing is worth it. If your regex is complicated and/or you do some other processing of lines, that can be run in parallel I would at least try it.

share|improve this answer

Your Answer


