title: multiprocessing.pool context and load balancing
body: <p>I've encountered some unexpected behaviour of the python multiprocessing Pool class.</p>
  
  <p>Here are my questions:
  <br>1) When does Pool creates its context, which is later used for serialization? <br><i>The example below runs fine as long as the Pool object is created after the Container definition. If you swap the Pool initializations, serialization error occurs.  In my production code I would like to initialize Pool way before defining the container class. Is it possible to refresh Pool "context" or to achieve this in another way.</i>
  <br>2) Does Pool have its own load balancing mechanism and if so how does it work? 
  <br> <i>If I run a similar example on my i7 machine with the pool of 8 processes I get the following results:
  <br> - For a light evaluation function Pool favours using only one process for computation. It creates 8 processes as requested but for most of the time only one is used (I printed the pid from inside and also see this in htop).
  <br> - For a heavy evaluation function the behaviour is as expected. It uses all 8 processes equally.</i>
  <br>3) When using Pool I always see 4 more processes that I requested (i.e. for Pool(processes=2) I see 6 new processes). What is their role?</p>
  
  <p>I use Linux with Python 2.7.2</p>
  
  <pre><code>from multiprocessing import Pool
  from datetime import datetime
  
  POWER = 10
  
  def eval_power(container):
      for power in xrange(2, POWER):
          container.val **= power
      return container
  
  #processes = Pool(processes=2)
  
  class Container(object):
      def __init__(self, value):
          self.val = value
  
  processes = Pool(processes=2)
  
  if __name__ == "__main__":
      cont = [Container(foo) for foo in xrange(20)]
      then = datetime.now()
      processes.map(eval_power, cont)
      now = datetime.now()
      print "Eval time:", now - then
  </code></pre>
  
  <p><br>
  <strong>EDIT - TO BAKURIU</strong>
  <br>1) I was afraid that that's the case.
  <br>2) I don't understand what the linux scheduler has to do with python assigning computations to processes. My situation can be ilustrated by the example below: </p>
  
  <pre><code>from multiprocessing import Pool
  from os import getpid
  from collections import Counter
  
  
  def light_func(ind):
      return getpid()
  
  
  def heavy_func(ind):
      for foo in xrange(1000000):
          ind += foo
      return getpid()
  
  
  if __name__ == "__main__":
      list_ = range(100)
      pool = Pool(4)
      l_func = pool.map(light_func, list_)
      h_func = pool.map(heavy_func, list_)
  
      print "light func:", Counter(l_func)
      print "heavy func:", Counter(h_func)
  </code></pre>
  
  <p><br>On my i5 machine (4 threads) I get the following results:
  <br><i>light func: Counter({2967: 100})
  <br>heavy func: Counter({2969: 28, 2967: 28, 2968: 23, 2970: 21})</i>
  <br>It seems that the situation is as I've described it. However I still don't understand why python does it this way. My guess would be that it tries to minimise communication expenses, but still the mechanism which it uses for load balancing is unknown. The documentation isn't very helpful either, the multiprocessing module is very poorly documented.
  <br>3) If I run the above code I get 4 more processes as described before. The screen comes from htop: <a href="http://i.stack.imgur.com/PldmM.png" rel="nofollow">http://i.stack.imgur.com/PldmM.png</a></p>