title: Why is writing to memory much slower than reading it?
body: <p>Here's a simple <code>memset</code> bandwidth benchmark:</p>
  
  <pre><code>#include &lt;stdio.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;string.h&gt;
  #include &lt;time.h&gt;
  
  int main()
  {
      unsigned long n, r, i;
      unsigned char *p;
      clock_t c0, c1;
      double elapsed;
  
      n = 1000 * 1000 * 1000; /* GB */
      r = 100; /* repeat */
  
      p = calloc(n, 1);
  
      c0 = clock();
  
      for(i = 0; i &lt; r; ++i) {
          memset(p, (int)i, n);
          printf("%4d/%4ld\r", p[0], r); /* "use" the result */
          fflush(stdout);
      }
  
      c1 = clock();
  
      elapsed = (c1 - c0) / (double)CLOCKS_PER_SEC;
  
      printf("Bandwidth = %6.3f GB/s (Giga = 10^9)\n", (double)n * r / elapsed / 1e9);
  
      free(p);
  }
  </code></pre>
  
  <p>On my system (details below) with a single DDR3-1600 memory module, it outputs:</p>
  
  <blockquote>
    <p>Bandwidth =  4.751 GB/s (Giga = 10^9)</p>
  </blockquote>
  
  <p>This is 37% of the theoretical RAM speed: <code>1.6 GHz * 8 bytes = 12.8 GB/s</code></p>
  
  <p>On the other hand, here's a similar "read" test:</p>
  
  <pre><code>#include &lt;stdio.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;string.h&gt;
  #include &lt;time.h&gt;
  
  unsigned long do_xor(const unsigned long* p, unsigned long n)
  {
      unsigned long i, x = 0;
  
      for(i = 0; i &lt; n; ++i)
          x ^= p[i];
      return x;
  }
  
  int main()
  {
      unsigned long n, r, i;
      unsigned long *p;
      clock_t c0, c1;
      double elapsed;
  
      n = 1000 * 1000 * 1000; /* GB */
      r = 100; /* repeat */
  
      p = calloc(n/sizeof(unsigned long), sizeof(unsigned long));
  
      c0 = clock();
  
      for(i = 0; i &lt; r; ++i) {
          p[0] = do_xor(p, n / sizeof(unsigned long)); /* "use" the result */
          printf("%4ld/%4ld\r", i, r);
          fflush(stdout);
      }
  
      c1 = clock();
  
      elapsed = (c1 - c0) / (double)CLOCKS_PER_SEC;
  
      printf("Bandwidth = %6.3f GB/s (Giga = 10^9)\n", (double)n * r / elapsed / 1e9);
  
      free(p);
  }
  </code></pre>
  
  <p>It outputs:</p>
  
  <blockquote>
    <p>Bandwidth = 11.516 GB/s (Giga = 10^9)</p>
  </blockquote>
  
  <p>I can get close to the theoretical limit for read performance, such as XORing a large array, but writing appears to be much slower. Why?</p>
  
  <p><strong>OS</strong> Ubuntu 14.04 AMD64 (I compile with <code>gcc -O3</code>. Using <code>-O3 -march=native</code> makes the read performance slightly worse, but does not affect <code>memset</code>)</p>
  
  <p><strong>CPU</strong> Xeon E5-2630 v2</p>
  
  <p><strong>RAM</strong> A single "16GB PC3-12800 Parity REG CL11 240-Pin DIMM" (What it says on the box) I think that having a single DIMM makes performance more predictable. I'm assuming that with 4 DIMMs, <code>memset</code> will be <em>up to</em> 4 times faster.</p>
  
  <p><strong>Motherboard</strong> Supermicro X9DRG-QF (Supports 4-channel memory)</p>
  
  <p><strong><em>Additional system</em></strong>: A laptop with 2x 4GB of DDR3-1067 RAM: read and write are both about 5.5 GB/s, but note that it uses 2 DIMMs.</p>
  
  <p><strong><em>P.S.</em></strong> replacing <code>memset</code> with this version results in exactly the same performance</p>
  
  <pre><code>void *my_memset(void *s, int c, size_t n)
  {
      unsigned long i = 0;
      for(i = 0; i &lt; n; ++i)
          ((char*)s)[i] = (char)c;
      return s;
  }
  </code></pre>