Audio Visual Speech Recognition using Deep Recurrent Neural Networks
  In this work, we propose a training algorithm for an audio-visual automatic
speech recognition (AV-ASR) system using deep recurrent neural network
(RNN).First, we train a deep RNN acoustic model with a Connectionist Temporal
Classification (CTC) objective function. The frame labels obtained from the
acoustic model are then used to perform a non-linear dimensionality reduction
of the visual features using a deep bottleneck network. Audio and visual
features are fused and used to train a fusion RNN. The use of bottleneck
features for visual modality helps the model to converge properly during
training. Our system is evaluated on GRID corpus. Our results show that
presence of visual modality gives significant improvement in character error
rate (CER) at various levels of noise even when the model is trained without
noisy data. We also provide a comparison of two fusion methods: feature fusion
and decision fusion.
