title: Loading a TAR file and extracting its *bz2 contents to sdout with bzcat
body: <p>Following <a href="https://codereview.stackexchange.com/questions/77022/loading-40gb-twitter-json-tar-file-from-archive-org-and-load-into-postgresql?noredirect=1#comment139908_77022">this question</a>, I'm attempting to load a <a href="https://archive.org/details/twitterstream" rel="nofollow noreferrer">40 GB TAR file</a> with bz2-compressed json files into PostgreSQL in an effecient manner.</p>
  
  <p>As per the answer mentioned above, I'm trying to separate the process and use external tools to create the following flow.</p>
  
  <ul>
  <li>Open &amp; Extract the files to SDOUT with TAR (bsdtar in this case, as TAR does not include extracting in its Windows build), only *.bz2 files.</li>
  <li>Extract the *BZ2 files with a call to bzcat (export to sdout)</li>
  <li>Open this in my python script 'file_handling', which maps each incoming line to a tweet and outputs this as a csv to stdout</li>
  <li>Pipe this to PSQL to load it into one COPY command.</li>
  </ul>
  
  <p>I'm currently getting an error when arriving at bzcat, this is what I have to build line that executes the above:</p>
  
  <pre><code>pipeline = [filename[1:3] + " &amp;&amp; ",  # Change drive to H so that TAR can find the file without a drive name (doesn't like absolute paths, apparently).
              '"C:\\Tools\\GnuWin32\\gnuwin32\\bin\\bsdtar" vxOf ' + filename_nodrive + ' "*.bz2"',  # Call to tar, outputs to stdin
              " | C:\\Tools\\GnuWin32\\gnuwin32\\bin\\bzcat.exe"#,  # Forward its output to bzcat
              ' | python "D:\Cloud\Dropbox\Coding\GitHub\pyTwitter\pyTwitter_filehandling.py"', # Extract Tweets
              ' | "C:\Program Files\PostgreSQL\9.4\bin\psql.exe" -1f copy.sql ' + secret_login_d
             ]
  module_call = "".join(pipeline)
  module_call = "H: &amp;&amp; "C:\Tools\GnuWin32\gnuwin32\bin\bsdtar" vxOf "Twitter datastream/Sourcefiles/archiveteam-twitter-stream-2013-01.tar" "*.bz2" | C:\Tools\GnuWin32\gnuwin32\bin\bzcat.exe | python "D:\Cloud\Dropbox\Coding\GitHub\pyTwitter\pyTwitter_filehandling.py" | "C:\Program Files\PostgreSQL\9.4in\psql.exe" -1f copy.sql "user=xxx password=xxx host=localhost port=5432 dbname=xxxxxx""
  </code></pre>
  
  <p>When executing the code for TAR, the TAR file is outputted to the CMD prompt, hinting me that all is well. However, the bzcat line brings an error:</p>
  
  <pre><code>x 01/29/06/39.json.bz2
  bzcat.exe: Data integrity error when decompressing.
      Input file = (stdin), output file = (stdout)
  
  It is possible that the compressed file(s) have become corrupted.
  You can use the -tvv option to test integrity of such files.
  </code></pre>
  
  <p>Running -tvv gives me:</p>
  
  <pre><code>huff+mtf data integrity (CRC) error in data
  </code></pre>
  
  <p>I've tried to extract the same archive with 7-zip (GUI): this still works. Any help on how to troubleshoot this would be greatly appreciated. I'm running Windows 8.1 with GNUWin32.</p>