title: How does the ability to compress a stream affect a compression algorithm?
body: <p>I recently backed up my soon-to-expire university home directory by sending it as a tar stream and compressing it on my end: <code>ssh user@host "tar cf - my_dir/" | bzip2 &gt; uni_backup.tar.bz2</code>.</p>
  
  <p>This got me thinking: I only know the basics of how compression works, but I would imagine that this ability to compress a stream of data would lead to poorer compression since the algorithm needs to finish handling a block of data at one point, write this to the output stream and continue to the next block.</p>
  
  <p>Is this the case? Or do these programs simply read a lot of data into memory compress this, write it, and then do this over again? Or are there any clever tricks used in these “stream compressors”? I see that both <em>bzip2</em> and <em>xz</em>'s man pages talk about memory usage, and <em>man bzip2</em> also hints to the fact that little is lost on chopping the data to be compressed into blocks:</p>
  
  <blockquote>
    <p>Larger  block  sizes give rapidly diminishing marginal returns.  Most of the compression comes from the first two or three hundred k of block size, a fact worth bearing in mind when using bzip2 on small machines.  It is also important  to  appreciate  that  the decompression memory requirement is set at compression time by the choice of block size.</p>
  </blockquote>
  
  <p>I would still love to hear if other tricks are used, or about where I can read more about this.</p>