WER we are and WER we think we are
  Natural language processing of conversational speech requires the
availability of high-quality transcripts. In this paper, we express our
skepticism towards the recent reports of very low Word Error Rates (WERs)
achieved by modern Automatic Speech Recognition (ASR) systems on benchmark
datasets. We outline several problems with popular benchmarks and compare three
state-of-the-art commercial ASR systems on an internal dataset of real-life
spontaneous human conversations and HUB'05 public benchmark. We show that WERs
are significantly higher than the best reported results. We formulate a set of
guidelines which may aid in the creation of real-life, multi-domain datasets
with high quality annotations for training and testing of robust ASR systems.
