Exploiting the Logits: Joint Sign Language Recognition and
  Spell-Correction
  Machine learning techniques have excelled in the automatic semantic analysis
of images, reaching human-level performances on challenging benchmarks. Yet,
the semantic analysis of videos remains challenging due to the significantly
higher dimensionality of the input data, respectively, the significantly higher
need for annotated training examples. By studying the automatic recognition of
German sign language videos, we demonstrate that on the relatively scarce
training data of 2.800 videos, modern deep learning architectures for video
analysis (such as ResNeXt) along with transfer learning on large gesture
recognition tasks, can achieve about 75% character accuracy. Considering that
this leaves us with a probability of under 25% that a 5 letter word is spelled
correctly, spell-correction systems are crucial for producing readable outputs.
The contribution of this paper is to propose a convolutional neural network for
spell-correction that expects the softmax outputs of the character recognition
network (instead of a misspelled word) as an input. We demonstrate that purely
learning on softmax inputs in combination with scarce training data yields
overfitting as the network learns the inputs by heart. In contrast, training
the network on several variants of the logits of the classification output i.e.
scaling by a constant factor, adding of random noise, mixing of softmax and
hardmax inputs or purely training on hardmax inputs, leads to better
generalization while benefitting from the significant information hidden in
these outputs (that have 98% top-5 accuracy), yielding a readable text despite
the comparably low character accuracy.
