title: Can gzip in a BASH script give an exit status before a file is completely unzipped? How to prevent this?
body: <p>I am trying to write a script to collect a few statistics on .fastq files sequentially. In my script, I unzip and re-zip each file in a loop (<code>gzip / gzip -d</code>), applying commands to find the stats on each file whilst it is unzipped using i.e. 
  <code>command  "${file_name//.fastq.gz/.fastq}"</code> .</p>
  
  <p>When I do this, my script sometimes fails to collect statistics for certain files, (e.g. giving me line counts of zero). However, it does this seemingly randomly and running the script more than once it sometimes collects stats and sometimes doesn't for the same file.</p>
  
  <p>I believe this is because the gzip is returning an exit status before the file is fully unzipped, meaning that my script continues and sometimes collects stats from half-zipped files. In support of this, I have seen different stats for the file sizes returned for the same file (but not for stats like header line counts, which seem to be either zero or the values I would expect).</p>
  
  <p>Is there a best way to tell BASH to wait until the file is completely unzipped? I have tried using variants of <code>until [ -f "${file_name//.fastq.gz/.fastq}" ]</code> and <code>until [ command "${file_name//.fastq.gz/.fastq}" != 0 ]</code> but I still don't always get the right results this way (I have checked by unzipping filesand applying each command manually), then comparing values to the script each time.</p>
  
  <p>I've posted the script below and linked a picture showing output from two different runs of the script on four files to highlight the issue. For the record, I have been running this script on a sun grid engine and it has been returning no error messages to accompany this.</p>
  
  <pre><code>#!/bin/bash
  #$ -cwd
  #$ -l h_rt=01:00:00
  #$ -l h_vmem=1G
  #$ -o fastq_qc_stats_job_out_file.txt
  #$ -e fastq_qc_stats_job_error_file.txt
  
  #First, make a file to store the output
  if echo "${PWD}/" | grep -iq "/[a-Z]*_[0-9]*/"  # if in numbered batches (i.e. the data to be analysed is split across multiple numbered files)
  then batch=`echo "${PWD}/" | grep -o "/[a-Z]*_[0-9]*/" |  cut -d '_' -f 2 | cut -d '/' -f 1` # get the batch number to name the output file,
  else batch=`basename $PWD`; fi # otherwise just use the final part of the directory name.
  header_line=`echo {'FILE','RUN','SIZE','READ_COUNT','MEAN_READ_LENGTH'} | sed 's/ /,/g'` # make a header line
  echo $header_line &gt; "QC_FASTQ_stats_${batch}.csv" # make a .csv file with the header line (by batch)
  #Now loop through the FASTQ files and add the following information for each of them
  
  for file in `ls *.fastq.gz`
  do gzip -d $file # unzip the file
  f="${file//.fastq.gz/.fastq}"
  accession=`echo ${f} | cut -d '.' -f 1 | cut -d '_' -f 1`
  filesize=`du -h ${f} | awk '{print $1}'`
  readcount=`grep -E '^@[EDS]RR' ${f} | grep -E ' length=' | wc -l`
  averagelength=`grep ' length=' ${f} | awk '{print $NF}' | cut -d '=' -f 2 | awk '{ total += $1 } END { print total/NR }'` # calculates mean
  filestats=`echo $file $accession $filesize $readcount $averagelength | sed 's/ /,/g'`
  echo $filestats &gt;&gt; "QC_FASTQ_stats_${batch}.csv" # add stats for each .fastq file to the .csv file
  gzip ${f} # re-zip the file
  done
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/neVEV.png" rel="nofollow noreferrer">An example of the output variation when run twice for the same files - see 4th file</a></p>