title: What's a good strategy for processing a queue in parallel?
body: <p>I'm writing a program which needs to recursively search through a folder structure, and would like to do so in parallel with several threads.</p>
  
  <p>I've written the rather trivial synchronous method already - adding the root directory to the queue initially, then dequeuing a directory, queuing its subdirectories, etc., until the queue is empty. I'll use a <a href="http://msdn.microsoft.com/en-us/library/dd267265.aspx" rel="nofollow"><code>ConcurrentQueue&lt;T&gt;</code></a> for my queue, but have already realized that my loops will stop prematurely. The first thread will dequeue the root directory, and immediately every other thread could see that the queue is empty and exit, leaving the first thread as the only one running. I would like each thread to loop until the queue is empty, then wait until another thread queues some more directories, and keep going. I need some sort of checkpoint in my loop so that none of the threads will exit until every thread has reached the end of the loop, but I'm not sure the best way to do this without deadlocking when there really are no more directories to process.</p>