title: Converting a 1.2GB list of edges into a sparse matrix
body: <p>I have a 1.2GB list of edges from a graph in a text file. My ubuntu PC has 8GB of RAM. Each line in the input looks like</p>
  
  <pre><code>287111206 357850135
  </code></pre>
  
  <p>I would like to convert it into a sparse adjacency matrix and output that to a file.</p>
  
  <p>Some statistics for my data: </p>
  
  <pre><code>Number of edges: around 62500000
  Number of vertices: around 31250000
  </code></pre>
  
  <p>I asked much the same question before at <a href="https://stackoverflow.com/a/38667644/2179021">https://stackoverflow.com/a/38667644/2179021</a> and got a great answer. The problem is that I can't get it to work.</p>
  
  <p>I first tried np.loadtxt to load in the file but it was very slow and used a huge amount of memory.  So instead I moved to pandas.read_csv which is very fast but this caused it own problems. This is my current code:</p>
  
  <pre><code>import pandas
  import numpy as np
  from scipy import sparse
  
  data = pandas.read_csv("edges.txt", sep=" ", header= None, dtype=np.uint32)
  A = data.as_matrix()
  print type(A)
  k1,k2,k3=np.unique(A,return_inverse=True,return_index=True)
  rows,cols=k3.reshape(A.shape).T
  M=sparse.coo_matrix((np.ones(rows.shape,int),(rows,cols)))
  print type(M)
  </code></pre>
  
  <p>The problem is that the pandas dataframe <code>data</code> is huge  and I am effectively making a copy in A which is inefficient. However things are even worse as the code crashes with </p>
  
  <pre><code>&lt;type 'instancemethod'&gt;
  Traceback (most recent call last):
    File "make-sparse-matrix.py", line 13, in &lt;module&gt;
      rows,cols=k3.reshape(A.shape).T
  AttributeError: 'function' object has no attribute 'shape'
  raph@raph-desktop:~/python$ python make-sparse-matrix.py 
  &lt;type 'numpy.ndarray'&gt;
  Traceback (most recent call last):
    File "make-sparse-matrix.py", line 12, in &lt;module&gt;
      k1,k2,k3=np.unique(A,return_inverse=True,return_index=True)
    File "/usr/local/lib/python2.7/dist-packages/numpy/lib/arraysetops.py", line 209, in unique
      iflag = np.cumsum(flag) - 1
    File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 2115, in cumsum
      return cumsum(axis, dtype, out)
  MemoryError
  </code></pre>
  
  <p>So my questions are:</p>
  
  <ol>
  <li>Can I avoid having both the 1.2GB pandas dataframe and the 1.2GB numpy array copy in memory?</li>
  <li>Is there some way to get the code to complete in 8GB of RAM?</li>
  </ol>
  
  <p>You can reproduce a test input of the size I am trying to process with:</p>
  
  <pre><code>import random
  #Number of edges, vertices
  m = 62500000
  n = m/2
  for i in xrange(m):
      fromnode = str(random.randint(0, n-1)).zfill(9)
      tonode = str(random.randint(0, n-1)).zfill(9)
      print fromnode, tonode
  </code></pre>
  
  <p><strong>Update</strong></p>
  
  <p>I have now tried a number of different approaches, all of which have failed.  Here is a summary.</p>
  
  <ol>
  <li>Using <a href="http://igraph.org/" rel="noreferrer">igraph</a> with <code>g = Graph.Read_Ncol('edges.txt')</code>. This uses a huge amount of RAM which crashes my computer.</li>
  <li>Using <a href="https://networkit.iti.kit.edu/" rel="noreferrer">networkit</a> with <code>G= networkit.graphio.readGraph("edges.txt",
  networkit.Format.EdgeList, separator=" ", continuous=False)</code>. This uses a huge amount of RAM which crashes my computer.</li>
  <li>The code above in this question but using np.loadtxt("edges.txt") instead of pandas. This uses a huge amount of RAM which crashes my computer.</li>
  </ol>
  
  <p>I then wrote separate code which remapped all the vertex names to number from 1..|V| where |V| is the total number of vertices.  This should save the code that imports the edge list from having to build up a table that maps the vertex names.  Using this I tried:</p>
  
  <ol start="4">
  <li>Using this new remapped edge list file I used igraph again with <code>g = Graph.Read_Edgelist("edges-contig.txt")</code>. This now works although it takes 4GB of RAM (which is way more than the theoretical amount it should).  However, there is no igraph function to write out a sparse adjacency matrix from a graph. The recommended solution is to <a href="https://stackoverflow.com/a/28179024/2179021">convert the graph to a coo_matrix</a>. Unfortunately this uses a huge amount of RAM which crashes my computer.</li>
  <li>Using the remapped edge list file I used networkit with <code>G = networkit.readGraph("edges-contig.txt", networkit.Format.EdgeListSpaceOne)</code>. This also works using less than the 4GB that igraph needs. networkit also comes with a function to write Matlab files (which is a form of sparse adjacency matrix that scipy can read). However <code>networkit.graphio.writeMat(G,"test.mat")</code> uses a huge amount of RAM which crashes my computer.</li>
  </ol>
  
  <p>Finally sascha's answer below does complete but takes about 40 minutes.</p>