title: Nested Iteration of HDF5 using PyTables
body: <p>I am have a fairly large dataset that I store in HDF5 and access using PyTables.  One operation I need to do on this dataset are pairwise comparisons between each of the elements.  This requires 2 loops, one to iterate over each element, and an inner loop to iterate over every other element. This operation thus looks at N(N-1)/2 comparisons.</p>
  
  <p>For fairly small sets I found it to be faster to dump the contents into a multdimensional numpy array and then do my iteration.  I run into problems with large sets because of memory issues and need to access each element of the dataset at run time. </p>
  
  <p>Putting the elements into an array gives me about 600 comparisons per second, while operating on hdf5 data itself gives me about 300 comparisons per second.</p>
  
  <p>Is there a way to speed this process up?</p>
  
  <p>Example follows (this is not my real code, just an example):</p>
  
  <p><strong>Small Set</strong>:</p>
  
  <pre><code>with tb.openFile(h5_file, 'r') as f:
      data = f.root.data
  
      N_elements = len(data)
      elements = np.empty((N_elements, 1e5))
  
      for ii, d in enumerate(data):
          elements[ii] = data['element']
  
  D = np.empty((N_elements, N_elements))  
  for ii in xrange(N_elements):
      for jj in xrange(ii+1, N_elements):             
          D[ii, jj] = compare(elements[ii], elements[jj])
  </code></pre>
  
  <p><strong>Large Set</strong>:</p>
  
  <pre><code>with tb.openFile(h5_file, 'r') as f:
      data = f.root.data
  
      N_elements = len(data)        
  
      D = np.empty((N_elements, N_elements))  
      for ii in xrange(N_elements):
          for jj in xrange(ii+1, N_elements):             
               D[ii, jj] = compare(data['element'][ii], data['element'][jj])
  </code></pre>