Accelerating RNN Transducer Inference via One-Step Constrained Beam
  Search
  We propose a one-step constrained (OSC) beam search to accelerate recurrent
neural network (RNN) transducer (RNN-T) inference. The original RNN-T beam
search has a while-loop leading to speed down of the decoding process. The OSC
beam search eliminates this while-loop by vectorizing multiple hypotheses. This
vectorization is nontrivial as the expansion of the hypotheses within the
original RNN-T beam search can be different from each other. However, we found
that the hypotheses expanded only once at each decoding step in most cases;
thus, we constrained the maximum expansion number to one, thereby allowing
vectorization of the hypotheses. For further acceleration, we assign
constraints to the prefixes of the hypotheses to prune the redundant search
space. In addition, OSC beam search has duplication check among hypotheses
during the decoding process as duplication can undesirably shrink the search
space. We achieved significant speedup compared with other RNN-T beam search
methods with lower phoneme and word error rate.
