title: Shared-memory IPC synchronization (lock-free)
body: <p>Consider the following scenario:</p>
  
  <p><strong>Requirements:</strong></p>
  
  <ul>
  <li>Intel x64 Server (multiple CPU-sockets => NUMA)</li>
  <li>Ubuntu 12, GCC 4.6</li>
  <li>Two processes sharing large amounts of data over (named) shared-memory</li>
  <li>Classical producer-consumer scenario</li>
  <li>Memory is arranged in a circular buffer (with M elements)</li>
  </ul>
  
  <p><strong>Program sequence (pseudo code):</strong></p>
  
  <p>Process A (Producer):</p>
  
  <pre><code>int bufferPos = 0;
  while( true )
  {
      if( isBufferEmpty( bufferPos ) )
      {
          writeData( bufferPos );
          setBufferFull( bufferPos );
  
          bufferPos = ( bufferPos + 1 ) % M;
      }
  }
  </code></pre>
  
  <p>Process B (Consumer):</p>
  
  <pre><code>int bufferPos = 0;
  while( true )
  {
      if( isBufferFull( bufferPos ) )
      {
          readData( bufferPos );
          setBufferEmpty( bufferPos );
  
          bufferPos = ( bufferPos + 1 ) % M;
      }
  }
  </code></pre>
  
  <p>Now the age-old question: How to synchronize them effectively!?</p>
  
  <ol>
  <li>Protect every read/write access with mutexes</li>
  <li>Introduce a "grace period", to allow writes to complete: Read data in buffer N, when buffer(N+3) has been marked as full (dangerous, but seems to work...)</li>
  <li>?!?</li>
  </ol>
  
  <p>Ideally I would like something along the lines of a memory-barrier, that guarantees that all previous reads/writes are visible across all CPUs, along the lines of:</p>
  
  <pre><code>writeData( i );
  MemoryBarrier();
  
  //All data written and visible, set flag
  setBufferFull( i );
  </code></pre>
  
  <p>This way, I would only have to monitor the buffer flags and then could read the large data chunks safely.</p>
  
  <p>Generally I'm looking for something along the lines of acquire/release fences as described by Preshing here:</p>
  
  <p><a href="http://preshing.com/20130922/acquire-and-release-fences/">http://preshing.com/20130922/acquire-and-release-fences/</a></p>
  
  <p>(if I understand it correctly the C++11 atomics only work for threads of a single process and not along multiple processes.)</p>
  
  <p>However the GCC-own memory barriers (__sync_synchronize in combination with the compiler barrier asm volatile( "" ::: "memory" ) to be sure) don't seem to work as expected, as writes become visible after the barrier, when I expected them to be completed.</p>
  
  <p>Any help would be appreciated...</p>
  
  <p>BTW: Under windows this just works fine using volatile variables (a Microsoft specific behaviour)...</p>