title: What is proper way to use shared list in multiprocessing
body: <p>I have implemented a <strong>SharedList</strong> in Python (version 3.7) with the help of <code>Manager, Lock</code> of multiprocessing. I have used it as a shared object to among the process created using multiprocessing <code>Process</code> function call. Shared List is used to storing the values/objects generated by each process Sharing it.</p>
  
  <p>Implementation of <strong>SharedList</strong> with <code>Manager</code> and <code>Lock</code> of <code>multiprocessing</code> of Python </p>
  
  <pre><code>class SharedList(object):
      def __init__(self, limit):
          self.manager = Manager()
          self.results = self.manager.list([])
          self.lock = Lock()
          self.limit = limit
  
      def append(self, new_value):
          with self.lock:
              if len(self.results) == self.limit:
                  return False
              self.results.append(new_value)
              return True
  
      def list(self):
          with self.lock:
              return list(self.results).copy()
  </code></pre>
  
  <p>Usage of created <strong>SharedList</strong> to store the values by multiple processes created using <code>multiprocessing</code></p>
  
  <pre><code>results = SharedList(limit)
  num_processes = min(process_count, limit)
  processes = []
  for i in range(num_processes):
     new_process = Process(target=child_function, args=(results))
     processes.append(new_process)
     new_process.start()
  
  for _process in processes:
     _process.join()
  
  for _process in processes:
     _process.close()
  </code></pre>
  
  <p>Implementation of <code>child_function</code></p>
  
  <pre><code>while True:
    result = func()
    if not (results.append(result)):
       break
  </code></pre>
  
  <p>The implementation works of some scenarios, but hangup, when I have increased the limit.
  I have used the processor count less than the number of CPU and do the same experiment still hang up in the same position.</p>
  
  <p>Are there any better approaches to the above problem, I have looked into different approaches such as using Queue, but that doesn't work as expected, get hang up?</p>
  
  <p><strong>Added the Previous Implementation using Queue</strong></p>
  
  <p>Implementation using Queue</p>
  
  <pre><code>results_out = []
  manager = multiprocessing.Manager()
  results = manager.Queue()
  tasks = manager.Queue()
  num_processes = min(process_count, limit)
  processes = []
  for i in range(num_processes):
      new_process = multiprocessing.Process(target=child_function,
                                              args=(tasks, results)
      processes.append(new_process)
      new_process.start()
  
  sleep(5)
  for i in range(limit):
      tasks.put(0)
  sleep(1)
  
  for i in range(num_processes):
      tasks.put(-1)
  
  num_finished_processes = 0
  while True:
      new_result = results.get()
      if new_result == -1:
          num_finished_processes += 1
          if num_finished_processes == num_processes:
              break
      else:
          results_out.append(new_result)
  
  for process in processes:
      process.join()
  
  for process in processes:
      process.close()
  </code></pre>
  
  <p>In  <code>child_function</code></p>
  
  <pre><code>while True:
      task_val = tasks.get()
      if task_val &lt; 0:
          results.put(-1)
          break
      else:
          result = func()
          results.put(result)
  </code></pre>
  
  <p><strong>Updated</strong></p>
  
  <p>I had gone through the following references, before posting this question, but I am unable to get the desired output. I agree, this code led to a deadlock state, but I am unable to find an implementation without deadlock using multiprocessing in python</p>
  
  <p><strong>References</strong></p>
  
  <ol>
  <li><p><a href="https://stackoverflow.com/questions/23623195/multiprocessing-of-shared-list">Multiprocessing of shared list</a></p></li>
  <li><p><a href="https://pymotw.com/2/multiprocessing/basics.html" rel="nofollow noreferrer">https://pymotw.com/2/multiprocessing/basics.html</a></p></li>
  <li><p><a href="https://stackoverflow.com/questions/17377426/shared-variable-in-pythons-multiprocessing/17393879">Shared variable in python&#39;s multiprocessing</a></p></li>
  <li><p><a href="https://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing" rel="nofollow noreferrer">https://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing</a></p></li>
  <li><p><a href="https://medium.com/@urban_institute/using-multiprocessing-to-make-python-code-faster-23ea5ef996ba" rel="nofollow noreferrer">https://medium.com/@urban_institute/using-multiprocessing-to-make-python-code-faster-23ea5ef996ba</a></p></li>
  <li><p><a href="http://kmdouglass.github.io/posts/learning-pythons-multiprocessing-module/" rel="nofollow noreferrer">http://kmdouglass.github.io/posts/learning-pythons-multiprocessing-module/</a></p></li>
  <li><p><a href="https://stackoverflow.com/questions/30127790/python-multiprocessing-threading-cleanup/30133147">python multiprocessing/threading cleanup</a></p></li>
  </ol>
  
  <p>Based on the suggestion, I was able to modify <strong>SharedList</strong> using <code>Queue</code></p>
  
  <pre><code>class SharedList(object):
      def __init__(self, limit):
          self.manager = Manager()
          self.tasks = self.manager.Queue()
          self.results = self.manager.Queue()
          self.limit = limit
          self.no_of_process = min(process_count, limit)
  
      def setup(self):
          sleep(1)
          for i in range(self.limit):
              self.tasks.put(0)
          sleep(1)
          for i in range(self.no_of_process):
              self.tasks.put(-1)
  
      def append(self, new_value):
          task_val = self.tasks.get()
          if task_val &lt; 0:
              self.results.put(-1)
              return False
          else:
              self.results.put(new_value)
              return True
  
      def list(self):
          results_out = []
          num_finished_processes = 0
          while True:
              new_result = self.results.get()
              if new_result == -1:
                  num_finished_processes += 1
                  if num_finished_processes == self.no_of_process:
                      break
              else:
                  results_out.append(new_result)
          return results_out
  </code></pre>
  
  <p>This implementation works fine, with following implementation change</p>
  
  <pre><code>results = SharedList(limit)
  num_processes = min(process_count, limit)
  processes = []
  for i in range(num_processes):
     new_process = Process(target=child_function, args=(results))
     processes.append(new_process)
     new_process.start()
  
  results.setup()
  
  for _process in processes:
     _process.join()
  
  for _process in processes:
     _process.close()
  </code></pre>
  
  <p>Implementation of <code>child_function</code></p>
  
  <pre><code>while True:
    result = func()
    if not (results.append(result)):
       break
  </code></pre>
  
  <p>But Still, again this ended up in deadlock, where it hangup, after some iterations</p>