Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring
  Automatic Speech Scoring (ASS) is the computer-assisted evaluation of a
candidate's speaking proficiency in a language. ASS systems face many
challenges like open grammar, variable pronunciations, and unstructured or
semi-structured content. Recent deep learning approaches have shown some
promise in this domain. However, most of these approaches focus on extracting
features from a single audio, making them suffer from the lack of
speaker-specific context required to model such a complex task. We propose a
novel deep learning technique for non-native ASS, called speaker-conditioned
hierarchical modeling. In our technique, we take advantage of the fact that
oral proficiency tests rate multiple responses for a candidate. We extract
context vectors from these responses and feed them as additional
speaker-specific context to our network to score a particular response. We
compare our technique with strong baselines and find that such modeling
improves the model's average performance by 6.92% (maximum = 12.86%, minimum =
4.51%). We further show both quantitative and qualitative insights into the
importance of this additional context in solving the problem of ASS.
