title: Performance Tips for using Vaex
body: <p>I am using Vaex and looking for performance tips. </p>
  
  <p>My use-case is as follows: </p>
  
  <ul>
  <li><p>I have a large dataframe - let's call it <code>large_df</code>(only a few columns but tens of million rows, and in production, the dataset will be >10x as large). One of the columns is called <code>key</code>, this is a 64 character alphanumeric string. The contents of this dataframe are stored across several HDF5 files. I create the dataframe by doing <code>vaex.open_many(&lt;path/to/hdf5 files/)</code>.</p></li>
  <li><p>On each request, the code receives a small (in the 10s) amount of keys to look up in <code>large_df</code>. I then basically have to look up in <code>large_df</code> to get the rows whose <code>key</code>s match the input list of keys, and then do some processing on the resultant matching df (which will be much smaller). </p></li>
  </ul>
  
  <p>From what I have read, Vaex should be perfect for my use case, however I have struggled to get the performance I was expecting. </p>
  
  <p>My code is essentially this: </p>
  
  <pre><code>import vaex
  df = vaex.open_many(&lt;/path/to/hdf5 files&gt;)
  df = df[df.key.isin(&lt;list of input keys&gt;)].to_pandas_df()
  </code></pre>
  
  <p>When all the HDF5 files are cached on disk ahead of time, this code takes around 80 seconds on an i3.8xlarge instance. The code runs inside a Docker container with the CPUs capped at 30 (of 32 available). I read the article about how Vaex can handle strings very well, and at first glance this seems like the type of task Vaex should be able to parallelize easily and compute in faster than ~80s. </p>
  
  <p>I have also tried pre-indexing a <code>short_id</code> column into the dataset that comprises <code>large_df</code>. Basically, this is an integer representing the first 4 characters in the <code>key</code> column. I then tried pre-filtering the df before doing the full string comparison. This code looks like the following:</p>
  
  <pre><code>import vaex
  df = vaex.open_many(&lt;/path/to/hdf5 files&gt;)
  short_ids = [alphanumeric_string_to_int(key) for key in &lt;input keys&gt;]
  df = df[df.short_id.isin(short_ids)]  # filter df down to a smaller size
  df = df[df.key.isin(&lt;list of input keys&gt;)].to_pandas_df()
  </code></pre>
  
  <p>That shaved off about 10 seconds, but that seems like it should have made things a lot faster. I feel like I am missing something obvious for how to make this blazing fast. </p>
  
  <p>What can I do? Please help - thank you! </p>