title: Unexpectedly good performance with openmp parallel for loop
body: <p><strong>I have edited my question after previous comments (especially @Zboson) for better readability</strong></p>
  
  <p>I have always acted on, and observed, the conventional wisdom that the number of openmp threads should roughly match the number of hyper-threads on a machine for optimal performance. However, I am observing odd behaviour on my new laptop with Intel Core i7 4960HQ, 4 cores - 8 threads. (See <a href="http://ark.intel.com/products/76088" rel="noreferrer">Intel docs here</a>)</p>
  
  <p>Here is my test code:</p>
  
  <pre><code>#include &lt;math.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;omp.h&gt;
  
  int main() {
      const int n = 256*8192*100;
      double *A, *B;
      posix_memalign((void**)&amp;A, 64, n*sizeof(double));
      posix_memalign((void**)&amp;B, 64, n*sizeof(double));
      for (int i = 0; i &lt; n; ++i) {
          A[i] = 0.1;
          B[i] = 0.0;
      }
      double start = omp_get_wtime();
      #pragma omp parallel for
      for (int i = 0; i &lt; n; ++i) {
          B[i] = exp(A[i]) + sin(B[i]);
      }
      double end = omp_get_wtime();
      double sum = 0.0;
      for (int i = 0; i &lt; n; ++i) {
          sum += B[i];
      }
      printf("%g %g\n", end - start, sum);
      return 0;
  }
  </code></pre>
  
  <p>When I compile it using <code>gcc 4.9-4.9-20140209</code>, with the command: <code>gcc -Ofast -march=native -std=c99 -fopenmp -Wa,-q</code> I see the following performance as I change <code>OMP_NUM_THREADS</code> [the points are an average of 5 runs, the error bars (which are hardly visible) are the standard deviations]:
  <img src="https://i.stack.imgur.com/X07B3.png" alt="Performance as a function of thread count"></p>
  
  <p>The plot is clearer when shown as the speed up with respect to OMP_NUM_THREADS=1:
  <img src="https://i.stack.imgur.com/t9j94.png" alt="Speed up as a function of thread count"></p>
  
  <p>The performance more or less monotonically increases with thread number, even when the the number of omp threads very greatly exceeds the core and also hyper-thread count! Usually the performance should drop off when too many threads are used (at least in my previous experience), due to the threading overhead. Especially as the calculation should be cpu (or at least memory) bound and not waiting on I/O.</p>
  
  <p>Even more weirdly, the speed-up is 35 times!</p>
  
  <p>Can anyone explain this?</p>
  
  <p>I also tested this with much smaller arrays 8192*4, and see similar performance scaling.</p>
  
  <p>In case it matters, I am on Mac OS 10.9 and the performance data where obtained by running (under bash):</p>
  
  <pre><code>for i in {1..128}; do
      for k in {1..5}; do
          export OMP_NUM_THREADS=$i;
          echo -ne $i $k "";
          ./a.out;
      done;
  done &gt; out
  </code></pre>
  
  <p><strong>EDIT:</strong> Out of curiosity I decided to try much larger numbers of threads. My OS limits this to 2000. The odd results (both speed up and low thread overhead) speak for themselves!
  <img src="https://i.stack.imgur.com/DYgSh.png" alt="Crazy numbers of threads"></p>
  
  <p><strong>EDIT:</strong> I tried @Zboson latest suggestion in their answer, i.e. putting VZEROUPPER before each math function within the loop, and it did fix the scaling problem! (It also sent the single threaded code from 22 s to 2 s!): </p>
  
  <p><img src="https://i.stack.imgur.com/rOx07.png" alt="correct scaling"></p>