title: Memory leak in a node.js crawler application
body: <p>For over a month I'm struggling with a very annoying memory leak issue and I have no clue how to solve it.</p>
  
  <p>I'm writing a general purpose web crawler based on: http, async, cheerio and nano. From the very beginning I've been struggling with memory leak which was very difficult to isolate. 
  I know it's possible to do a heapdump and analyse it with Google Chrome but I can't understand the output. It's usually a bunch of meaningless strings and objects leading to some anonymous functions telling me exactly nothing (it might be lack of experience on my side).</p>
  
  <p>Eventually I came to a conclusion that the library I had been using at the time (jQuery) had issues and I replaced it with Cheerio. I had an impression that Cheerio solved the problem but now I'm sure it only made it less dramatic. </p>
  
  <p>You can find my code at: <a href="https://github.com/lukaszkujawa/node-web-crawler" rel="nofollow">https://github.com/lukaszkujawa/node-web-crawler</a>. I understand it might be lots of code to analyse but perhaps I'm doing something stupid which can be obvious strait away. I'm suspecting the main agent class which does HTTP requests <a href="https://github.com/lukaszkujawa/node-web-crawler/blob/master/webcrawler/agent.js" rel="nofollow">https://github.com/lukaszkujawa/node-web-crawler/blob/master/webcrawler/agent.js</a> from multiple "threads" (with <code>async.queue</code>). </p>
  
  <p>If you would like to run the code it requires CouchDB and after <code>npm install</code> do:</p>
  
  <p><code>$ node crawler.js -c conf.example.json</code></p>
  
  <p>I know that Node doesn't go crazy with garbage collection but after 10min of heavy crawling used memory can go easily over 1GB.</p>
  
  <p>(tested with v0.10.21 and v0.10.22)</p>