title: Node.js network performance suffers drastically under load testing on AWS
body: <p>I have the following Node.js code written as a very basic HTTP server. It's purpose is to ingest large numbers of requests containing base64 data, and write that data to S3 as an image. The S3-writing aspect is working fantastically and has no issues. However, the initial request seems to take an abnormally long time under load.</p>
  
  <p><strong>server.js</strong></p>
  
  <pre><code>http.createServer(function(req, res){
      if (url.parse(req.url).pathname == '/processimage' &amp;&amp; req.method.toLowerCase() == 'post') {
          var startTime = new Date();
          var rawBody = '';
          req.on('data', function(chunk) { 
              rawBody += chunk;
          });
  
          req.on('end', function() {
              console.log('REQUEST FINISHED: ' + (new Date() - startTime) + ' ms');
              // Process image, upload to S3
              res.writeHead(200);
              res.end('data');
          }
          return;
      } else {
          // Other requests
      }
  }).listen(1347);
  </code></pre>
  
  <p>I am also timing the image processing section, but it is performing fine and not relevant to this question.</p>
  
  <p>To test this, I have written a script that POSTs test data containing the base64 of approximately 500k characters (2-3mb original images). When testing this locally, everything works fine. My output is:</p>
  
  <pre><code>REQUEST FINISHED: 9ms
  REQUEST FINISHED: 23ms
  REQUEST FINISHED: 18ms
  </code></pre>
  
  <p>etc.</p>
  
  <p>However, after deploying the code to AWS on an x-large instance, I see the following:</p>
  
  <pre><code>REQUEST FINISHED: 499ms
  REQUEST FINISHED: 2493ms
  REQUEST FINISHED: 1784ms
  REQUEST FINISHED: 3440ms
  REQUEST FINISHED: 994ms
  REQUEST FINISHED: 36043ms
  </code></pre>
  
  <p>Essentially, when stress testing this, approximately 1 in every 30 requests seems to take 10+ seconds (even 30+ seconds in some cases) just to go through the request pipeline. As you can see in my code, there is zero processing being done on the data before the times are calculated, so this means that somewhere between "req.on('data')" and "req.on('end')", there is a massive delay.</p>
  
  <p>My question is: is there some kind of processing happening between req.on('data') and req.on('end') that would cause this POST to take so long? Is it possible that the host machine is choking on these requests for some reason (Ubuntu 12.04, x-large instance, 14GB memory, 4x CPUs)?</p>