title: Custom allocators - versus std::vector, and strange test results
body: <p>I followed a tutorial <a href="https://developer.ibm.com/tutorials/au-memorymanager/" rel="nofollow noreferrer">here</a> to make a custom memory manager for a simple class. Sure enough, as the page suggests, I had a dramatic increase in performance carrying out the tests. Here is the test code copied over:</p>
  <pre><code>Complex* array[1000];
  for (int i = 0; i &lt; 5000; i++) {
      for (int j = 0; j &lt; 1000; j++) {
          array[j] = new Complex(i,j);
      }
      for (int j = 0; j &lt; 1000; j++) {
          delete array[j];
      }
  }
  </code></pre>
  <p>Where <code>Complex</code> is a fairly straightforward class:</p>
  <pre><code>Class Complex {
  public:
      Complex(double a, double b) :
          r(a),
          c(b)
      {}
  private:
      double r;
      double c;
  }
  </code></pre>
  <p>Before implementation of the manager, this test took an average of 3.30866 seconds to run on my machine.
  After implementation of the manager (following the tutorial pretty much word-for-word - should I copy and paste all the code here?) the test took an average of 0.02842002 seconds.</p>
  <p>This seemed like a great improvement, but I wanted to see how it fared against populating a std::vector, as this is the main container used in my existing code. The results were somewhat surprising:</p>
  <pre><code>for (int i = 0; i &lt; 5000; i++) {
      std::vector&lt;Complex&gt; values;
      for (int j = 0; j &lt; 1000; j++) {
          values.push_back(Complex(i, j));
      }
      values.clear();
  }
  </code></pre>
  <p>This took an average of 0.0434723 seconds to run, with very similar results using overloaded new/delete and with default. Still twice as slow as using the custom manager, but here's where my faith in my own testing breaks down. I tried re-running the original test code without using the custom manager, and it took an average of 0.3028371 seconds.</p>
  <p>While there is still an order of magnitude increase in performance, I instinctively feel suspicious of the whole test now. I have no idea where the sudden increase in performance in the original test came from. Is this a symptom of compiler optimizations? I'm using MSVC v142 toolset on Visual Studio 2019.</p>
  <p>I've seen similar effects to this running production code multiple times, where the first run is slow and successive runs are faster. However, this effect did not occur when carrying out the very first original test - individual results were all grouped around the 3.3 second mark.</p>
  <p>Could anyone shed any light on what is going on here? Is it worth writing custom allocators if the STL containers seem to perform so well? How can I improve my testing technique?</p>