title: Parsing Gigantic Log File in Python
body: <p>I am trying to parse a gigantic log file (around 5 GB). </p>
  
  <p>I only want to parse the first 500,000 lines and I don't want to read the whole file into memory. </p>
  
  <p>Basically, I want to do what the below is code is doing but with a <code>while</code> loop instead of a <code>for</code> loop and <code>if</code> conditional. I also want to be sure not read the entire file into memory.</p>
  
  <pre><code>import re
  from collections import defaultdict
  FILE = open('logs.txt', 'r')
  count_words=defaultdict(int)
  import pickle
  i=0
  for line in FILE.readlines():
      if i &lt; 500000:
          m = re.search('key=([^&amp;]*)', line)
          count_words[m.group(1)]+=1
      i+=1
  
  csv=[]
  for k, v in count_words.iteritems():
      csv.append(k+","+str(v))
  print "\n".join(csv)
  </code></pre>