title: File Size Reduction in Pandas and HDF5
body: <p>I am running a model that outputs data into multiple Pandas frames, and then saves those frames to an HDF5 file. The model is run several hundred times, each time adding new columns (multi-indexed) into the existing HDF5 file's frames. This is done with Pandas <code>merge</code>. Since the frames are different lengths for each run, there ends up being a large number of <code>NaN</code> values in the frames.</p>
  
  <p>After enough model runs are completed, data are dropped from the frames if the rows or columns are associated with a model run that had an error. In that process, the new data frames are put into a new HDF5 file. the following pseudo-python shows this process:</p>
  
  <pre><code>with pandas.HDFStore(filename) as store:
      # figure out which indices should be removed
      indices_to_drop = get_bad_indices(store)
  
      new_store = pandas.HDFStore(reduced_filename) 
      for key in store.keys():
          df = store[key]
          for idx in indices_to_drop:
               df = df.drop(idx, &lt;level and axis info&gt;)
          new_store[key] = df
      new_store.close()
  </code></pre>
  
  <p>The new hdf5 file ends up being about 10% of the size of the original. The only difference in the files is that all the <code>NaN</code> values are no longer equal (but are all numpy float64 values). </p>
  
  <p>My question is, how can this filesize reduction (presumably through managing <code>NaN</code> values) be achieved on an existing hdf5 file? There are times where I don't need to do the above procedure, but I am doing it anyway to get the reduction. Is there an existing Pandas or PyTables command that can do this? Thank you very much in advance.</p>