Visual Features for Context-Aware Speech Recognition
  Automatic transcriptions of consumer-generated multi-media content such as
"Youtube" videos still exhibit high word error rates. Such data typically
occupies a very broad domain, has been recorded in challenging conditions, with
cheap hardware and a focus on the visual modality, and may have been
post-processed or edited. In this paper, we extend our earlier work on adapting
the acoustic model of a DNN-based speech recognition system to an RNN language
model and show how both can be adapted to the objects and scenes that can be
automatically detected in the video. We are working on a corpus of "how-to"
videos from the web, and the idea is that an object that can be seen ("car"),
or a scene that is being detected ("kitchen") can be used to condition both
models on the "context" of the recording, thereby reducing perplexity and
improving transcription. We achieve good improvements in both cases and compare
and analyze the respective reductions in word error rate. We expect that our
results can be used for any type of speech processing in which "context"
information is available, for example in robotics, man-machine interaction, or
when indexing large audio-visual archives, and should ultimately help to bring
together the "video-to-text" and "speech-to-text" communities.
