title: Getting status of gsutil cp command in parallel mode
body: <p>This command copies a huge number of files from Google Cloud storage to my local server.</p>
  
  <pre><code>gsutil -m cp -r gs://my-bucket/files/ .
  </code></pre>
  
  <p>There are 200+ files, each of which is over 5GB in size.</p>
  
  <p>Once all files are downloaded, another process kicks in and starts reading the files one by one and extract the info needed.</p>
  
  <p>The problem is, even though the gsutil copy process is fast and downloads files in batches of multiple files at a very high speed, I still need to wait till all the files are downloaded before starting to process them.</p>
  
  <p>Ideally I would like to start processing the first file as soon as it is downloaded. But with multiple cp mode, there seems to be no way of knowing when a file is downloaded (or is there?).</p>
  
  <p>From Google docs, this can be done in individual file copy mode.</p>
  
  <pre><code>if ! gsutil cp ./local-file gs://your-bucket/your-object; then
    &lt;&lt; Code that handles failures &gt;&gt;
  fi
  </code></pre>
  
  <p>That means if I run the cp without -m flag, I can get a boolean on success for that file and I can kick off the file processing.</p>
  
  <p>Problem with this approach is the overall download will take much longer as files are now downloading one by one.</p>
  
  <p>Any insight?</p>