title: Read in large file and make dictionary
body: <p>I have a large file which I need to read in and make a dictionary from.  I would like this to be as fast as possible. However my code in python is too slow.  Here is a minimal example that shows the problem.</p>
  
  <p>First make some fake data</p>
  
  <pre><code>paste &lt;(seq 20000000) &lt;(seq 2 20000001)  &gt; largefile.txt
  </code></pre>
  
  <p>Now here is a minimal piece of python code to read it in and make a dictionary.</p>
  
  <pre><code>import sys
  from collections import defaultdict
  fin = open(sys.argv[1])
  
  dict = defaultdict(list)
  
  for line in fin:
      parts = line.split()
      dict[parts[0]].append(parts[1])
  </code></pre>
  
  <p>Timings:</p>
  
  <pre><code>time ./read.py largefile.txt
  real    0m55.746s
  </code></pre>
  
  <p>However it is not I/O bound as:</p>
  
  <pre><code>time cut -f1 largefile.txt &gt; /dev/null    
  real    0m1.702s
  </code></pre>
  
  <p>If I comment out the <code>dict</code> line it takes <code>9</code> seconds. It seems that almost all the time is spent by <code>dict[parts[0]].append(parts[1])</code>.</p>
  
  <p>Is there any way to speed this up? I don't mind using cython or even C if that is going to make a big difference. Or can pandas help here?</p>
  
  <p>Here is the profile output on a file of size 10000000 lines.</p>
  
  <pre><code>python -m cProfile read.py test.data         20000009 function calls in 42.494 seconds
  
     Ordered by: standard name
  
     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
          1    0.000    0.000    0.000    0.000 bisect.py:1(&lt;module&gt;)
          1    0.000    0.000    0.001    0.001 collections.py:1(&lt;module&gt;)
          1    0.000    0.000    0.000    0.000 collections.py:25(OrderedDict)
          1    0.000    0.000    0.000    0.000 collections.py:386(Counter)
          1    0.000    0.000    0.000    0.000 heapq.py:31(&lt;module&gt;)
          1    0.000    0.000    0.000    0.000 keyword.py:11(&lt;module&gt;)
          1   30.727   30.727   42.494   42.494 read.py:2(&lt;module&gt;)
   10000000    4.855    0.000    4.855    0.000 {method 'append' of 'list' objects}
          1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
   10000000    6.912    0.000    6.912    0.000 {method 'split of 'str' objects}
          1    0.000    0.000    0.000    0.000 {open}
  </code></pre>
  
  <p><strong>Update.</strong>  We can assume that parts[1] is an integer and that parts[0] is a short fixed length string. </p>
  
  <p>My fake data isn't very good as you only get one value per key. Here is a better version.</p>
  
  <pre><code>perl -E 'say int rand 1e7, $", int rand 1e4 for 1 .. 1e7' &gt; largefile.txt
  </code></pre>
  
  <p>The only operation I will do is to query a key to return the list of values associated with it.</p>