title: Node js - http.request() problems with connection pooling
body: <p>Consider the following simple Node.js application:</p>
  
  <pre><code>var http = require('http');
  http.createServer(function() { }).listen(8124); // Prevent process shutting down
  
  var requestNo = 1;
  var maxRequests = 2000;
  
  function requestTest() {
      http.request({ host: 'www.google.com', method: 'GET' }, function(res) {
          console.log('Completed ' + (requestNo++));
  
          if (requestNo &lt;= maxRequests) {
              requestTest();
          }
      }).end();
  }
  
  requestTest();
  </code></pre>
  
  <p>It makes 2000 HTTP requests to google.com, one after the other. The problem is it gets to request No. 5 and pauses for about 3 mins, then continues processing requests 6 - 10, then pauses for another 3 minutes, then requests 11 - 15, pauses, and so on. <strong>Edit:</strong> <em>I tried changing www.google.com to localhost, an extremely basic Node.js app running my machine that returns "Hello world", I still get the 3 minute pause.</em></p>
  
  <p>Now I read I can increase the connection pool limit:</p>
  
  <pre><code>http.globalAgent.maxSockets = 20;
  </code></pre>
  
  <p>Now if I run it, it processes requests 1 - 20, then pauses for 3 mins, then requests 21 - 40, then pauses, and so on.</p>
  
  <p>Finally, after a bit of research, I learned I could disable connection pooling entirely by setting <code>agent: false</code> in the request options:</p>
  
  <pre><code>http.request({ host: 'www.google.com', method: 'GET', agent: false }, function(res) {
      ...snip....
  </code></pre>
  
  <p>...and it'll run through all 2000 requests just fine.</p>
  
  <p>My question, is it a good idea to do this? Is there a danger that I could end up with too many HTTP connections? And why does it pause for 3 mins, surely if I've finished with the connection it should add it straight back into the pool ready for the next request to use, so why is it waiting 3 mins? Forgive my ignorance.</p>
  
  <p>Failing that, what is the best strategy for a Node.js app making a potentially large number HTTP requests, without locking up, or crashing?</p>
  
  <p>I'm running Node.js version 0.10 on Mac OSX 10.8.2.</p>
  
  <hr>
  
  <p><strong>Edit:</strong> I've found if I convert the above code into a for loop and try to establish a bunch of connections at the same time, I start getting errors after about 242 connections. The error is:</p>
  
  <pre><code>Error was thrown: connect EMFILE
  (libuv) Failed to create kqueue (24)
  </code></pre>
  
  <p>...and the code...</p>
  
  <pre><code>for (var i = 1; i &lt;= 2000; i++) {
      (function(requestNo) {
          var request = http.request({ host: 'www.google.com', method: 'GET', agent: false }, function(res) {
              console.log('Completed ' + requestNo);
          });
  
          request.on('error', function(e) {
              console.log(e.name + ' was thrown: ' + e.message);
          });
  
          request.end();
      })(i);
  }
  </code></pre>
  
  <p>I don't know if a heavily loaded Node.js app could ever reach that many simultaneous connections.</p>