title: Why is Parallel.ForEach much faster then AsParallel().ForAll() even though MSDN suggests otherwise?
body: <p>I've been doing some investigation to see how we can create a multithreaded application that runs through a tree.</p>
  
  <p>To find how this can be implemented in the best way I've created a test application that runs through my C:\ disk and opens all directories.</p>
  
  <pre><code>class Program
  {
      static void Main(string[] args)
      {
          //var startDirectory = @"C:\The folder\RecursiveFolder";
          var startDirectory = @"C:\";
  
          var w = Stopwatch.StartNew();
  
          ThisIsARecursiveFunction(startDirectory);
  
          Console.WriteLine("Elapsed seconds: " + w.Elapsed.TotalSeconds);
  
          Console.ReadKey();
      }
  
      public static void ThisIsARecursiveFunction(String currentDirectory)
      {
          var lastBit = Path.GetFileName(currentDirectory);
          var depth = currentDirectory.Count(t =&gt; t == '\\');
          //Console.WriteLine(depth + ": " + currentDirectory);
  
          try
          {
              var children = Directory.GetDirectories(currentDirectory);
  
              //Edit this mode to switch what way of parallelization it should use
              int mode = 3;
  
              switch (mode)
              {
                  case 1:
                      foreach (var child in children)
                      {
                          ThisIsARecursiveFunction(child);
                      }
                      break;
                  case 2:
                      children.AsParallel().ForAll(t =&gt;
                      {
                          ThisIsARecursiveFunction(t);
                      });
                      break;
                  case 3:
                      Parallel.ForEach(children, t =&gt;
                      {
                          ThisIsARecursiveFunction(t);
                      });
                      break;
                  default:
                      break;
              }
  
          }
          catch (Exception eee)
          {
              //Exception might occur for directories that can't be accessed.
          }
      }
  }
  </code></pre>
  
  <p>What I have encountered however is that when running this in mode 3 (Parallel.ForEach) the code completes in around 2.5 seconds (yes I have an SSD ;) ). Running the code without parallelization it completes in around 8 seconds. And running the code in mode 2 (AsParalle.ForAll()) it takes a near infinite amount of time.</p>
  
  <p>When checking in process explorer I also encounter a few strange facts:</p>
  
  <pre><code>Mode1 (No Parallelization):
  Cpu:     ~25%
  Threads: 3
  Time to complete: ~8 seconds
  
  Mode2 (AsParallel().ForAll()):
  Cpu:     ~0%
  Threads: Increasing by one per second (I find this strange since it seems to be waiting on the other threads to complete or a second timeout.)
  Time to complete: 1 second per node so about 3 days???
  
  Mode3 (Parallel.ForEach()):
  Cpu:     100%
  Threads: At most 29-30
  Time to complete: ~2.5 seconds
  </code></pre>
  
  <p>What I find especially strange is that Parallel.ForEach seems to ignore any parent threads/tasks that are still running while AsParallel().ForAll() seems to wait for the previous Task to either complete (which won't soon since all parent Tasks are still waiting on their child tasks to complete).</p>
  
  <p>Also what I read on MSDN was: "Prefer ForAll to ForEach When It Is Possible"</p>
  
  <p>Source: <a href="http://msdn.microsoft.com/en-us/library/dd997403(v=vs.110).aspx" rel="noreferrer">http://msdn.microsoft.com/en-us/library/dd997403(v=vs.110).aspx</a></p>
  
  <p>Does anyone have a clue why this could be?</p>
  
  <p>Edit 1:</p>
  
  <p>As requested by Matthew Watson I've first loaded the tree in memory before looping through it. Now the loading of the tree is done sequentially.</p>
  
  <p>The results however are the same. Unparallelized and Parallel.ForEach now complete the whole tree in about 0.05 seconds while AsParallel().ForAll still only goes around 1 step per second.</p>
  
  <p>Code:</p>
  
  <pre><code>class Program
  {
      private static DirWithSubDirs RootDir;
  
      static void Main(string[] args)
      {
          //var startDirectory = @"C:\The folder\RecursiveFolder";
          var startDirectory = @"C:\";
  
          Console.WriteLine("Loading file system into memory...");
          RootDir = new DirWithSubDirs(startDirectory);
          Console.WriteLine("Done");
  
  
          var w = Stopwatch.StartNew();
  
          ThisIsARecursiveFunctionInMemory(RootDir);
  
          Console.WriteLine("Elapsed seconds: " + w.Elapsed.TotalSeconds);
  
          Console.ReadKey();
      }        
  
      public static void ThisIsARecursiveFunctionInMemory(DirWithSubDirs currentDirectory)
      {
          var depth = currentDirectory.Path.Count(t =&gt; t == '\\');
          Console.WriteLine(depth + ": " + currentDirectory.Path);
  
          var children = currentDirectory.SubDirs;
  
          //Edit this mode to switch what way of parallelization it should use
          int mode = 2;
  
          switch (mode)
          {
              case 1:
                  foreach (var child in children)
                  {
                      ThisIsARecursiveFunctionInMemory(child);
                  }
                  break;
              case 2:
                  children.AsParallel().ForAll(t =&gt;
                  {
                      ThisIsARecursiveFunctionInMemory(t);
                  });
                  break;
              case 3:
                  Parallel.ForEach(children, t =&gt;
                  {
                      ThisIsARecursiveFunctionInMemory(t);
                  });
                  break;
              default:
                  break;
          }
      }
  }
  
  class DirWithSubDirs
  {
      public List&lt;DirWithSubDirs&gt; SubDirs = new List&lt;DirWithSubDirs&gt;();
      public String Path { get; private set; }
  
      public DirWithSubDirs(String path)
      {
          this.Path = path;
          try
          {
              SubDirs = Directory.GetDirectories(path).Select(t =&gt; new DirWithSubDirs(t)).ToList();
          }
          catch (Exception eee)
          {
              //Ignore directories that can't be accessed
          }
      }
  }
  </code></pre>
  
  <p>Edit 2:</p>
  
  <p>After reading the update on Matthew's comment I've tried to add the following code to the program:</p>
  
  <pre><code>ThreadPool.SetMinThreads(4000, 16);
  ThreadPool.SetMaxThreads(4000, 16);
  </code></pre>
  
  <p>This however does not change how the AsParallel peforms. Still the first 8 steps are being executed in an instant before slowing down to 1 step / second.</p>
  
  <p>(Extra note, I'm currently ignoring the exceptions that occur when I can't access a Directory by the Try Catch block around the Directory.GetDirectories())</p>
  
  <p>Edit 3:</p>
  
  <p>Also what I'm mainly interested in is the difference between Parallel.ForEach and AsParallel.ForAll because to me it's just strange that for some reason the second one creates one Thread for every recursion it does while the first once handles everything in around 30 threads max. (And also why MSDN suggests to use the AsParallel even though it creates so much threads with a ~1 second timeout)</p>
  
  <p>Edit 4:</p>
  
  <p>Another strange thing I found out:
  When I try to set the MinThreads on the Thread pool above 1023 it seems to ignore the value and scale back to around 8 or 16:
  ThreadPool.SetMinThreads(1023, 16);</p>
  
  <p>Still when I use 1023 it does the first 1023 elements very fast followed by going back to the slow pace I've been experiencing all the time.</p>
  
  <p>Note: Also literally more then 1000 threads are now created (compared to 30 for the whole Parallel.ForEach one).</p>
  
  <p>Does this mean Parallel.ForEach is just way smarter in handling tasks?</p>
  
  <p>Some more info, this code prints twice 8 - 8 when you set the value above 1023: (When you set the values to 1023 or lower it prints the correct value)</p>
  
  <pre><code>        int threadsMin;
          int completionMin;
          ThreadPool.GetMinThreads(out threadsMin, out completionMin);
          Console.WriteLine("Cur min threads: " + threadsMin + " and the other thing: " + completionMin);
  
          ThreadPool.SetMinThreads(1023, 16);
          ThreadPool.SetMaxThreads(1023, 16);
  
          ThreadPool.GetMinThreads(out threadsMin, out completionMin);
          Console.WriteLine("Now min threads: " + threadsMin + " and the other thing: " + completionMin);
  </code></pre>
  
  <p>Edit 5:</p>
  
  <p>As of Dean's request I've created another case to manually create tasks:</p>
  
  <pre><code>case 4:
      var taskList = new List&lt;Task&gt;();
      foreach (var todo in children)
      {
          var itemTodo = todo;
          taskList.Add(Task.Run(() =&gt; ThisIsARecursiveFunctionInMemory(itemTodo)));
      }
      Task.WaitAll(taskList.ToArray());
      break;
  </code></pre>
  
  <p>This is also as fast as the Parallel.ForEach() loop. So we still don't have the answer to why AsParallel().ForAll() is so much slower.</p>