title: Fast token-to-index conversion with NumPy
body: <p>Let's say I have two large strings of text that I split into words:</p>
  
  <pre><code>import numpy as np
  
  s1 = 'this is a test test test'
  s2 = 'that is another test'
  
  terms1 = np.array(s1.split())
  terms2 = np.array(s2.split())
  </code></pre>
  
  <p>Now <code>terms1</code> is <code>['this', 'is', 'a', 'test', 'test', 'test']</code> and <code>terms2</code> is <code>['that', 'is', 'another', 'test']</code>.</p>
  
  <p>I now want to assign each unique word an ID and then get an array for each <code>terms</code>-vector that contains the respective IDs, i.e. the indices into the common "vocabulary" of <code>terms1</code> and <code>terms2</code>:</p>
  
  <pre><code>vocab = np.unique(np.concatenate((terms1, terms2)))
  # yields ['a', 'another', 'is', 'test', 'that', 'this']
  
  ind1 = [np.where(t == vocab)[0][0] for t in terms1]
  # yields indices into "vocab": [5, 2, 0, 3, 3, 3]
  ind2 = [np.where(t == vocab)[0][0] for t in terms2]
  # yields indices into "vocab": [4, 2, 1, 3]
  </code></pre>
  
  <p>This basically works. However, the usage of <code>np.where</code> in a for-loop seems inefficient and I wonder if there is a better way in NumPy to do such things?</p>