note BrowserUk <blockquote><i></i></blockquote> <p>What most the responses so far seem to have missed is that the cast of the in-ram [split]ing and [join]ing of records, pales into insignificance when compared to the time required for the read-a-record, write-a-record flip-flopping of the read head back and forth across the disk. <P>There are two possible ways to speed up your processing: <ol><li>Overlap the [split]ing & [join]ing with either the reading or the writing (or both). <p>This can be achieved by using two or more threads or processes. </li><li>Read and write larger chunks of the file to minimise the number of seeks the read heads need to make. <p>Interleaving 80GB/4k = 20 million reads, and (~) 200,000 write means (at least) 400,000 track to track seeks. <p>If you increase the read & write chunk sizes to 1MB each, that can be reduced to ~1,500 track to track seeks. </li></ol> <p>Try these: <ul><li><code> #! perl -sw use strict; our $B //= 64; $B *= 1024; my $ibuf = ''; while( sysread( *STDIN, $ibuf, $B, length $ibuf ) ) { my $p = 1+rindex( $ibuf, "\n" ); my $rem = substr( $ibuf, $p ); substr( $ibuf, $p ) = ''; open my $RAM, '<', \$ibuf; print while <$RAM>; $ibuf = $rem; } </code> </li><li><code> #! perl -sw use strict; our $B //= 64; $B *= 1024; my $obuf; while( <> ) { my @f = split chr(9); $obuf .= join( chr(9), @f[2,0,5] ) . "\n"; if( length( $obuf ) > $B ) { print $obuf; $obuf = ''; } } print $obuf; ## Corrected C&P error. See [lotus1]'s post below. </code> <p>Usage: <c>ibuf -B=1024 < in.tsv | -B=1024 > out.tsv</c>. <p>Based on my experiments, it might be possible to more than halve your processing time, though YMMV. <p>Experiment with adjusting the <c>-B=nnn</c> (in kbs) parameters up & down in unison and independently to find the sweet spot on your system. <p>Be aware, bigger is not always better. 1024 for both seems to work quite well on my system, anything larger slows it down. </li></ul> <div class="pmsig"><div class="pmsig-171588"> <hr /> <font size=1 > <div>With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'</div> <div>Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.</div> <div>"Science is about questioning the status quo. Questioning authority". </div> <div>In the absence of evidence, opinion is indistinguishable from prejudice. </div> </font> </div></div> 1014517 1014517