title: How to pin threads to cores with predetermined memory pool objects? (80 core Nehalem architecture 2Tb RAM)
body: <p>I've run into a minor HPC problem after running some tests on a 80core (160HT) nehalem architecture with 2Tb DRAM:</p>
  
  <p>A server with more than 2 sockets starts to stall a lot (delay) as each thread starts to request information about objects on the "wrong" socket, i.e. requests goes from a thread that is working on some objects on the one socket to pull information that is actually in the DRAM on the other socket.</p>
  
  <p>The cores appear 100% utilized, even though I know that they are waiting for the remote socket to return the request.</p>
  
  <p>As most of the code runs asynchronously it is a lot easier to rewrite the code so I can just parse messages from the threads on the one socket to threads the other (no locked waiting). 
  In addition I want to lock each threads to memory pools, so I can update objects instead of  wasting time (~30%) on the garbage collector.</p>
  
  <p>Hence the question:</p>
  
  <p><strong>How to pin threads to cores with predetermined memory pool objects in Python?</strong></p>
  
  <p>A little more context:</p>
  
  <p>Python has no problem running multicore when you put ZeroMQ in the middle and make an art out of passing messages between the memory pool managed by each ZMQworker. At ZMQ's 8M msg/second it the internal update of the objects take longer than the pipeline can be filled. This is all described here: <a href="http://zguide.zeromq.org/page:all#Chapter-Sockets-and-Patterns" rel="nofollow noreferrer">http://zguide.zeromq.org/page:all#Chapter-Sockets-and-Patterns</a></p>
  
  <p>So, with a little over-simplification, I spawn 80 ZMQworkerprocesses and 1 ZMQrouter and load the context with a large swarm of objects (584 million objects actually). 
  From this "start-point" the objects need to interact to complete the computation.</p>
  
  <p>This is the idea:</p>
  
  <ul>
  <li>If "object X" needs to interact with "Object Y" and is available in
  the local memory pool of the python-thread, then the interaction
  should be done directly. </li>
  <li>If "Object Y" is NOT available in the same pool, then I want it to
  send a message through the ZMQrouter and let the router return a
  response at some later point in time. My architecture is non-blocking so what goes on in the particular python thread just continues without waiting for the zmqRouters response. Even for objects on the same socket but on a different core, I would prefer NOT to interact, as I prefer having clean message exchanges instead of having 2 threads manipulating the same memory object.</li>
  </ul>
  
  <p>To do this I need to know:</p>
  
  <ol>
  <li>how to figure out which socket a given python process (thread)
  runs on. </li>
  <li>how assign a memory pool on that particular socket to the python process (some malloc limit or similar so that the sum of memory pools do not push the memory pool from one socket to another)</li>
  <li>Things I haven't thought of.</li>
  </ol>
  
  <p>But I cannot find references in the python docs on how to do this and on google I must be searching for the wrong thing. </p>
  
  <p><strong>Update:</strong></p>
  
  <p>Regarding the question "why use ZeroMQ on a MPI architecture?", please read the thread: <a href="https://stackoverflow.com/questions/35490/spread-vs-mpi-vs-zeromq">Spread vs MPI vs zeromq?</a> as the application I am working on is being designed for a distributed deployment even though it is tested on a an architecture where MPI <em>is</em> more suitable.</p>
  
  <p><strong>Update 2:</strong></p>
  
  <p>Regarding the question:</p>
  
  <p>"How to pin threads to cores with predetermined memory pools in Python(3)" the answer is in <a href="http://pythonhosted.org/psutil/#psutil.Process.cpu_affinity" rel="nofollow noreferrer">psutils</a>:</p>
  
  <pre><code>&gt;&gt;&gt; import psutil
  &gt;&gt;&gt; psutil.cpu_count()
  4
  &gt;&gt;&gt; p = psutil.Process()
  &gt;&gt;&gt; p.cpu_affinity()  # get
  [0, 1, 2, 3]
  &gt;&gt;&gt; p.cpu_affinity([0])  # set; from now on, this process will run on CPU #0 only
  &gt;&gt;&gt; p.cpu_affinity()
  [0]
  &gt;&gt;&gt;
  &gt;&gt;&gt; # reset affinity against all CPUs
  &gt;&gt;&gt; all_cpus = list(range(psutil.cpu_count()))
  &gt;&gt;&gt; p.cpu_affinity(all_cpus)
  &gt;&gt;&gt;
  </code></pre>
  
  <p>The worker can be pegged to a core whereby the NUMA may be exploited effectively (lookup your CPU type to verify that it is a NUMA architecture!)</p>
  
  <p>The second element is to determine the memory-pool. That can be done with <a href="http://pythonhosted.org/psutil/#psutil.virtual_memory" rel="nofollow noreferrer">psutils</a> as well or the <a href="https://docs.python.org/3.4/library/resource.html" rel="nofollow noreferrer">resource library</a>:</p>