title: Multiprocessing with dictionary of generator objects, TypeError: cannot pickle 'generator' object
body: <p>How can I use multiprocessing to create a dictionary with generator objects as values? </p>
  
  <p>Here is my problem in greater detail, using basic examples:</p>
  
  <p>I have a large dictionary of lists whereby I am applying functions to compute on the dictionary values using <code>ProcessPoolExecutor</code> in <code>concurrent.futures</code>. (Note I am using <code>ProcessPoolExecutor</code>, not threads---there is no GIL contention here.)</p>
  
  <p>Here is an example dictionary of lists:</p>
  
  <pre><code>example_dict1 = {'key1':[367, 30, 847, 482, 887, 654, 347, 504, 413, 821],
      'key2':[754, 915, 622, 149, 279, 192, 312, 203, 742, 846], 
      'key3':[586, 521, 470, 476, 693, 426, 746, 733, 528, 565]}
  </code></pre>
  
  <p>This is a small example---in reality, it is a massive dictionary with millions of keys and lists with thousands of elements. </p>
  
  <p>Originally, I was processing these value lists using a function which appends to an empty list, e.g. </p>
  
  <pre><code>import concurrent
  
  def manipulate_values(dict_items):
      k, v = dict_items
      return_values = []   ## empty list
      for i in v :
          new_value = i ** 2 - 13      ## compute something; just an example
          return_values.append(new_value)  ## append to list
      return k, return_values
  
  with concurrent.futures.ProcessPoolExecutor() as executor:
          new_dict = dict(executor.map(manipulate_values, example_dict1.items()))
  </code></pre>
  
  <p>However, the resulting dictionary lists <code>return_values</code> are too large for memory. Because I cannot hold these massive lists in memory, I decided to try generator functions, i.e. create a dictionary of keys and generator objects:</p>
  
  <pre><code>## generator function
  def compute_values(v):
      for i in v:
          yield i ** 2 - 13   ## using example above; could be any function call 
  
  def manipulate_values_with_generator(dict_items):
      keys, value_lists = dict_items
      return keys, compute_values(value_lists)
  
  with concurrent.futures.ProcessPoolExecutor() as executor:
          new_dict = dict(executor.map(manipulate_values_with_generator, example_dict1.items()))
  </code></pre>
  
  <p>In the actual "big data" dictionary, this will result in a pickling error:</p>
  
  <pre><code>TypeError: cannot pickle 'generator' object
  </code></pre>
  
  <p>There are other questions on this subject, but I'm not sure how to fix my problem---packages such as <code>dill</code> and <code>pathos</code> do not work with generators. </p>
  
  <p><a href="https://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-multiprocessing-pool-map/7309686">Can&#39;t pickle &lt;type &#39;instancemethod&#39;&gt; when using multiprocessing Pool.map()</a></p>
  
  <p><a href="https://stackoverflow.com/questions/27883574/cant-pickle-pyparsing-expression-with-setparseaction-method-needed-for-multi/27892382">Can&#39;t pickle Pyparsing expression with setParseAction() method. Needed for multiprocessing</a></p>
  
  <p>Naturally, perhaps there's another way to solve my fundamental problem, which is doing multiprocessing on a large dictionary of lists within memory or generators. Maybe another data structure would be useful here? </p>