title: looking to automate a process of manipulating with very large .csv files
body: <p>here is my dilemma.  I need to process information from large .csv files.  typical size of the file is about 200mb which results in approximately 3-4 million lines.
  files contain human readable information. 5 fields.
  my process originally was to take the file and sort it based on the last field</p>
  <p>&quot;sort -k4 -n -t, &quot;</p>
  <p>then split</p>
  <p>&quot;split -l 1000000&quot;</p>
  <p>however now i need to add a decode from base64 of the second field.</p>
  <p>i used this script for testing</p>
  <p>awk 'BEGIN{FS=OFS=&quot;,&quot;} (cmd=&quot;echo &quot;$2&quot; | base64 --decode&quot;; cmd | getline v;$2=v} 1' originalfilename.csv &gt; newfilename.csv</p>
  <p>-- while running this command on a sample file a I am getting an error message.</p>
  <p>&quot;sh: fork: Resource temporarily unavailable&quot;</p>
  <p>and the process halts.</p>
  <p>my final output are several .csv files I use to import into Excel tabs for processing.</p>
  <p>here are my questions:</p>
  <ol>
  <li><p>why am I getting the error while running awk? is there a different command that will do what i need faster and cleaner</p>
  </li>
  <li><p>can anyone suggest a better way to process my files? im trying to stay away from java but python script may work.</p>
  </li>
  </ol>
  <p>i am not a professional programmer, hence my knowledge ot tools is not up to date.</p>
  <p>any advice/assistance would be greatly appreciated.</p>
  <p>Thanks</p>