title: how to prevent hadoop corrupted .gz file
body: <p>I'm using following simple code to upload files to hdfs.</p>
  
  <pre><code>FileSystem hdfs = FileSystem.get(config);
  hdfs.copyFromLocalFile(src, dst);
  </code></pre>
  
  <p>The files are generated by webserver java component and rotated and closed by logback in .gz format. I've noticed that sometimes the .gz file is corrupted.</p>
  
  <pre><code> &gt; gunzip logfile.log_2013_02_20_07.close.gz 
   gzip: logfile.log_2013_02_20_07.close.gz: unexpected end of file 
  </code></pre>
  
  <p>But the following command does show me the content of the file</p>
  
  <pre><code>&gt; hadoop fs -text /input/2013/02/20/logfile.log_2013_02_20_07.close.gz
  </code></pre>
  
  <p>The impact of having such files is quite disaster - since the aggregation for the whole day fails, and also several slave nodes is marked as blacklisted in such case.</p>
  
  <p>What can I do in such case?
  Can hadoop copyFromLocalFile() utility corrupt the file?
  Does anyone met similar problem ?</p>