title: Parallel.ForEach fails to execute messages on long running IEnumerable
body: <p>Why will the Parallel.ForEach will not finish executing a series of tasks until MoveNext returns false?</p>
  
  <p>I have a tool that monitors a combination of MSMQ and Service Broker queues for incoming messages. When a message is found, it hands that message off to the appropriate executor.  </p>
  
  <p>I wrapped the check for messages in an IEnumerable, so that I could hand the Parallel.ForEach method the IEnumerable plus a delegate to run. The application is designed to run continuously w/ the IEnumerator.MoveNext processing in a loop until it's able to get work, then the IEnumerator.Current giving it the next item. </p>
  
  <p>Since the MoveNext will never die until I set the CancelToken to true, this should continue to process for ever. Instead what I'm seeing is that once the Parallel.ForEach has picked up all the messages and the MoveNext is no longer returning "true", no more tasks are processed. Instead it seems like the MoveNext thread is the only thread given any work while it waits for it to return, and the other threads (including waiting and scheduled threads) do not do any work. </p>
  
  <ul>
  <li>Is there a way to tell the Parallel to keep working while it waits for a response from the MoveNext? </li>
  <li>If not, is there another way to structure the MoveNext to get what I want? (having it return true and then the Current returning a null object spawns a lot of bogus Tasks) </li>
  <li>Bonus Question: Is there a way to limit how many messages the Parallel pulls off at once? It seems to pull off and schedule a lot of messages at once (the MaxDegreeOfParallelism only seems to limit how much work it does at once, it doesn't stop it from pulling off a lot of messages to be scheduled)</li>
  </ul>
  
  <p>Here is the IEnumerator for what I've written (w/o some extraneous code):</p>
  
  <pre><code>public class DataAccessEnumerator : IEnumerator&lt;TransportMessage&gt; 
  {
      public TransportMessage Current
      {   get { return _currentMessage; } }
  
      public bool MoveNext()
      {
          while (_cancelToken.IsCancellationRequested == false)
          {
              TransportMessage current;
              foreach (var task in _tasks)
              {
                  if (task.QueueType.ToUpper() == "MSMQ")
                      current = _msmq.Get(task.Name);
                  else
                      current = _serviceBroker.Get(task.Name);
  
                  if (current != null)
                  {
                      _currentMessage = current;
                      return true;
                  }
              }
              WaitHandle.WaitAny(new [] {_cancelToken.WaitHandle}, 500); 
          }
  
          return false; 
      }
  
      public DataAccessEnumerator(IDataAccess&lt;TransportMessage&gt; serviceBroker, IDataAccess&lt;TransportMessage&gt; msmq, IList&lt;JobTask&gt; tasks, CancellationToken cancelToken)
      {
          _serviceBroker = serviceBroker;
          _msmq = msmq;
          _tasks = tasks;
          _cancelToken = cancelToken;
      }
  
      private readonly IDataAccess&lt;TransportMessage&gt; _serviceBroker;
      private readonly IDataAccess&lt;TransportMessage&gt; _msmq;
      private readonly IList&lt;JobTask&gt; _tasks;
      private readonly CancellationToken _cancelToken;
      private TransportMessage _currentMessage;
  }
  </code></pre>
  
  <p>Here is the Parallel.ForEach call where _queueAccess is the IEnumerable that holds the above IEnumerator and RunJob processes a TransportMessage that is returned from that IEnumerator: </p>
  
  <pre><code>var parallelOptions = new ParallelOptions
      {
          CancellationToken = _cancelTokenSource.Token,
          MaxDegreeOfParallelism = 8 
      };
  
  Parallel.ForEach(_queueAccess, parallelOptions, x =&gt; RunJob(x));
  </code></pre>