title: Bash while read loop extremely slow compared to cat, why?
body: <p>A simple test script here:</p>
  
  <pre><code>while read LINE; do
          LINECOUNT=$(($LINECOUNT+1))
          if [[ $(($LINECOUNT % 1000)) -eq 0 ]]; then echo $LINECOUNT; fi
  done
  </code></pre>
  
  <p>When I do <code>cat my450klinefile.txt | myscript</code> the CPU locks up at 100% and it can process about 1000 lines a second. About 5 minutes to process what <code>cat my450klinefile.txt &gt;/dev/null</code> does in half a second.</p>
  
  <p>Is there a <strong>more efficient way</strong> to do essentially this. I just need to read a line from stdin, count the bytes, and write it out to a named pipe. But the speed of even this example is impossibly slow. </p>
  
  <p>Every 1Gb of input lines I need to do a few more complex scripting actions (close and open some pipes that the data is being feed to).</p>