title: Multiprocessing to return large data sets in Python
body: <p>I have 2 functions in a Python 3.7 script that search 2 separate network nodes and returns very large data sets of strings in a list.  The smaller data set length is ~300K entries, while the larger one is ~1.5M.  This script takes almost an hour to execute because of how it has to compile the data sets as well as having the second data set be significantly larger.  I don't have a way to shorten the run time by changing how the compilation happens, there's no easier way for me to get the data from the network nodes.  But I can cut almost 10 minutes if I can run them simultaneously, so I'm trying to shorten the run time by using multiprocessing so I can run both of them at once.</p>
  <p>I do not need them to necessarily start within the same second or finish at the same second, just want them to run at the same time.</p>
  <p>Here's a breakdown of first attempt at coding for multiprocessing:</p>
  <pre><code>def p_func(arg1, arg2, pval):
       ## Do Stuff
      return pval
  
  def s_func(arg1, sval):
      ## Do Stuff
      return sval
  
  # Creating variables to get return values that multiprocessing can handle
  pval = multiprocessing.Value(list)
  sval = multiprocessing.Value(list)
  
  # setting up multiprocessing Processes for each function and passing arguments
  p1 = multiprocessing.Process(target=p_func, args=(arg1, arg2, pval))
  s2 = multiprocessing.Process(target=s_func, args=(arg3, sval))
  p1.start()
  s1.start()
  p1.join()
  s1.join()
  
  print(&quot;Number of values in pval: &quot;, len(pval))
  print(&quot;Number of values in sval: &quot;, len(sval))
  </code></pre>
  <p>I believe I have solved my <code>list</code> concerns, so....</p>
  <p>Based on comments I've updated my code as follows:</p>
  <pre><code>#! python3
  import multiprocessing as mp
  def p_func(arg1, arg2, pval):
      # takes arg1 and arg2 and queries network node to return list of ~300K
      # values and assigns that list to pval for return to main()
      return pval
  
  def s_func(arg1, sval):
      # takes arg1 and queries network node to return list of ~1.5M 
      # values and assigns that list to sval for return to main()
      return sval
  
  # Creating variables to get return values that multiprocessing can handle in 
  # main()
  with mp.Manager() as mgr
      pval = mgr.list()
      sval = mgr.list()
  
      # setting up multiprocessing Processes for each function and passing 
      # arguments
      p1 = mp.Process(target=p_func, args=(arg1, arg2, pval))
      s1 = mp.Process(target=s_func, args=(arg3, sval))
      p1.start()
      s1.start()
      p1.join()
      s1.join()
  
  # out of with block
  print(&quot;Number of values in pval: &quot;, len(pval))
  print(&quot;Number of values in sval: &quot;, len(sval))
  </code></pre>
  <p>Now getting a <strong>TypeError: can't pickle _thread.lock objects</strong> on the <code>p1.start()</code> invocation.  I'm guessing that one of the variables I have passed in the <code>p1</code> declaration is causing a problem with multiprocessing, but I'm not sure how to read the error or resolve the problem.</p>