title: How to calculate a very large correlation matrix
body: <p>I have an np.array of observations z where z.shape is (100000, 60).  I want to efficiently calculate the 100000x100000 correlation matrix and then write to disk the coordinates and values of just those elements > 0.95 (this is a very small fraction of the total).  </p>
  
  <p>My brute-force version of this looks like the following but is, not surprisingly, very slow:</p>
  
  <pre><code>for i1 in range(z.shape[0]):
      for i2 in range(i1+1):
          r = np.corrcoef(z[i1,:],z[i2,:])[0,1]
          if r &gt; 0.95:
              file.write("%6d %6d %.3f\n" % (i1,i2,r))
  </code></pre>
  
  <p>I realize that the correlation matrix itself could be calculated much more efficiently in one operation using np.corrcoef(z), but the memory requirement is then huge.  I'm also aware that one could break up the data set into blocks and calculate bite-size subportions of the correlation matrix at one time, but programming that and keeping track of the indices seems unnecessarily complicated.</p>
  
  <p>Is there another way (e.g., using memmap or pytables) that is both simple to code and doesn't put excessive demands on physical memory?</p>