End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics
  Optimization by Fully Convolutional Neural Networks
  Speech enhancement model is used to map a noisy speech to a clean speech. In
the training stage, an objective function is often adopted to optimize the
model parameters. However, in most studies, there is an inconsistency between
the model optimization criterion and the evaluation criterion on the enhanced
speech. For example, in measuring speech intelligibility, most of the
evaluation metric is based on a short-time objective intelligibility (STOI)
measure, while the frame based minimum mean square error (MMSE) between
estimated and clean speech is widely used in optimizing the model. Due to the
inconsistency, there is no guarantee that the trained model can provide optimal
performance in applications. In this study, we propose an end-to-end
utterance-based speech enhancement framework using fully convolutional neural
networks (FCN) to reduce the gap between the model optimization and evaluation
criterion. Because of the utterance-based optimization, temporal correlation
information of long speech segments, or even at the entire utterance level, can
be considered when perception-based objective functions are used for the direct
optimization. As an example, we implement the proposed FCN enhancement
framework to optimize the STOI measure. Experimental results show that the STOI
of test speech is better than conventional MMSE-optimized speech due to the
consistency between the training and evaluation target. Moreover, by
integrating the STOI in model optimization, the intelligibility of human
subjects and automatic speech recognition (ASR) system on the enhanced speech
is also substantially improved compared to those generated by the MMSE
criterion.
