title: Performance discrepancy between OSX and Linux for communication using Python multiprocessing
body: <p>I have been trying to learn more about Python's <code>multiprocessing</code> module and to evaluate different techniques for communication between processes. I wrote a benchmark that compares the performance of <code>Pipe</code>, <code>Queue</code>, and <code>Array</code> (all from <code>multiprocessing</code>) for transferring <code>numpy</code> arrays between processes. The full benchmark can be found <a href="https://gist.github.com/drdavella/87a903fdd769711ad7382f072aafde95" rel="noreferrer">here</a>. Here's a snippet of the test for <code>Queue</code>:</p>
  
  <pre><code>def process_with_queue(input_queue, output_queue):
      source = input_queue.get()
      dest = source**2
      output_queue.put(dest)
  
  
  def test_with_queue(size):
  
      source = np.random.random(size)
  
      input_queue = Queue()
      output_queue = Queue()
  
      p = Process(target=process_with_queue, args=(input_queue, output_queue))
      start = timer()
      p.start()
      input_queue.put(source)
      result = output_queue.get()
      end = timer()
  
      np.testing.assert_allclose(source**2, result)
  
      return end - start
  </code></pre>
  
  <p>I ran this test on my Linux laptop and got the following results for an array size of 1000000:</p>
  
  <pre><code>Using mp.Array: time for 20 iters: total=2.4869s, avg=0.12435s
  Using mp.Queue: time for 20 iters: total=0.6583s, avg=0.032915s
  Using mp.Pipe:  time for 20 iters: total=0.63691s, avg=0.031845s
  </code></pre>
  
  <p>I was a little surprised to see <code>Array</code> perform so poorly since it uses shared memory and presumably doesn't require pickling, but I assume there must be some copying in <code>numpy</code> that I can't control.</p>
  
  <p>However, I ran the same test (again for array size 1000000) on a Macbook, and got the following results:</p>
  
  <pre><code>Using mp.Array: time for 20 iters: total=1.6917s, avg=0.084587s
  Using mp.Queue: time for 20 iters: total=2.3478s, avg=0.11739s
  Using mp.Pipe:  time for 20 iters: total=8.7709s, avg=0.43855s
  </code></pre>
  
  <p>The real timing differences aren't that surprising since of course different systems would exhibit different performance. What <strong>is</strong> so surprising are the differences in relative timing.</p>
  
  <p>What could account for this? This is a pretty surprising result to me. I wouldn't be surprised to see such stark differences between Linux and Windows, or OSX and Windows, but I sort of assumed that these things would behave very similarly between OSX and Linux.</p>
  
  <p><a href="https://stackoverflow.com/questions/1289813/python-multiprocessing-vs-threading-for-cpu-bound-work-on-windows-and-linux">This question</a> addresses performance differences between Windows and OSX, which seems more expected.</p>