title: NodeJS read + write CSVs in threads/workers
body: <p>I currently have a nodejs script that reads data from CSV files and then writes to a number of different CSVs based on the data in each row.</p>
  
  <p>There are 300 CSVs (around 40Gbs worth)  to process so I added async to my script to read and write the data simultaneously across all cores. </p>
  
  <pre><code>async.mapLimit(filePaths, 4, streamZip, function (err, results) {
      console.log('finished');
  });
  </code></pre>
  
  <p>But it turns out that is not what async does. This code actually takes more time to complete than processing each file individually because it is using just a single core.</p>
  
  <p>There seems to many different ways to use more cores <a href="https://nodejs.org/api/cluster.html" rel="nofollow noreferrer">cluster</a>, <a href="https://nodejs.org/api/child_process.html" rel="nofollow noreferrer">child process</a>, <a href="https://www.npmjs.com/package/webworker-threads" rel="nofollow noreferrer">web workers</a> and <a href="https://github.com/rvagg/node-worker-farm" rel="nofollow noreferrer">worker-farm</a></p>
  
  <p>There have also been other questions asked like this <a href="https://stackoverflow.com/questions/2387724/node-js-on-multi-core-machines">one</a></p>
  
  <p>But they all seem to want to use HTTP or Express and run as a server, or they call on an external program like 'ls'. Rather than just running a multi processing pool like I would in Python.</p>
  
  <p>Can anyone provide an example or help on how to use threads or processes that would read multiple CSV files in parallel and all write to the same fs.createWriteStreams??</p>
  
  <p>Thanks</p>
  
  <p>More of my code is here:</p>
  
  <pre><code>function streamZip(filePath, callback) {
  var stream = fs.createReadStream(filePath)
      .pipe(unzip.Parse())
      .on('entry', function (entry) {
          var fileName = entry.path;
          entry.pipe(csvStream)
      })
  
  var csvStream = csv()
      .on("data", function(data){
          var identifier = data[0];
          if (identifier === '10') {
              10CSV.write(data)
          } else if (identifier === '11') {
              11CSV.write(data)
          } else if (identifier === '15') {
              15CSV.write(data)
          }
      })
      .on("end", function(){
          callback(null, filePath + 'Processed');
      });
  }
  </code></pre>