title: How should I batch upload to s3 and insert to MongoDB from nodeJS webserver with a final callback?
body: <p>I have a web server that accepts images from client, processes them, upload them to S3, batch-insert the urls to my mongoDB, and lastly sending the json result back to the client.</p>
  
  <p>Working with a single image works as follows:</p>
  
  <pre><code>router.post("/upload", function(req, res){
      var form = new multiparty.Form();
  
      form.parse(req,function(err,fields,files){
          s3.upload({
                  Key: filename,
                  Bucket: bucketname,
                  ACL: "public-read",
                  Body: fs.createReadStream(filepath)
           }, function(err, data){
              if(err) //error handle
  
              Model.collection.insert({"name": "name", "url" : data.Location}, function(err, result){
                  if(err) //error handle
                  res.json({result: result})
              })
          })
      })
  })
  </code></pre>
  
  <p>This works very well as I am simply uploading the file data to s3 -> when done, insert s3's output (url) to database -> when done, send mongo's result as jsonarray to the client.</p>
  
  <p>The issue is - my client html excepts multiple <code>type=file</code> input with the same <code>name=images</code>, so that i can access them in my form parser as <code>images[i]</code>. The above algorithm is repeated <code>images.length</code> times. Problem rises when  I have to return the jsonarray result back to the client as I would have to wait for all of the asynchronous S3 upload->mongo insert to finish, and I can't pinpoint how and where the callback would be for this job.</p>
  
  <p>What I've tried are the following:</p>
  
  <ul>
  <li>Iterate through images, upload them to S3 first, populate an array with the resulting url (<code>[data.Location]</code>). Batch insert them to mongoDB and return jsonarray result to client in the call back. This didn't work as mongo insert doesn't wait for S3 upload.</li>
  <li>Iterate through images, upload and insert the images to S3 and mongoDB each iteration. <code>if (currentIndex = images.length)</code>, return jsonarray result. This doesn't work accurately as I do not know which images will end last (different size).</li>
  </ul>
  
  <p>How should I design the algorithm to batch upload s3, batch insert to mongo, return result including s3 urls, filename etc back to client as jsonarray?</p>
  
  <p>Thanks, in advance!</p>