title: Python Multiprocessing.Pool lazy iteration
body: <p>I'm wondering about the way that python's Multiprocessing.Pool class works with map, imap, and map_async.  My particular problem is that I want to map on an iterator that creates memory-heavy objects, and don't want all these objects to be generated into memory at the same time.  I wanted to see if the various map() functions would wring my iterator dry, or intelligently call the next() function only as child processes slowly advanced, so I hacked up some tests as such:</p>
  
  <pre><code>def g():
    for el in xrange(100):
      print el
      yield el
  
  def f(x):
    time.sleep(1)
    return x*x
  
  if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    go = g()
    g2 = pool.imap(f, go)
    g2.next()
  </code></pre>
  
  <p>And so on with map, imap, and map_async.  This is the most flagrant example however, as simply calling next() a single time on g2 prints out all my elements from my generator g(), whereas if imap were doing this 'lazily' I would expect it to only call go.next() once, and therefore print out only '1'.</p>
  
  <p>Can someone clear up what is happening, and if there is some way to have the process pool 'lazily' evaluate the iterator as needed?</p>
  
  <p>Thanks,</p>
  
  <p>Gabe</p>