Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded
  Inference Based on Softmax Confidence
  We study the tradeoff between computational effort and classification
accuracy in a cascade of deep neural networks. During inference, the user sets
the acceptable accuracy degradation which then automatically determines
confidence thresholds for the intermediate classifiers. As soon as the
confidence threshold is met, inference terminates immediately without having to
compute the output of the complete network. Confidence levels are derived
directly from the softmax outputs of intermediate classifiers, as we do not
train special decision functions. We show that using a softmax output as a
confidence measure in a cascade of deep neural networks leads to a reduction of
15%-50% in the number of MAC operations while degrading the classification
accuracy by roughly 1%. Our method can be easily incorporated into pre-trained
non-cascaded architectures, as we exemplify on ResNet. Our main contribution is
a method that dynamically adjusts the tradeoff between accuracy and computation
without retraining the model.
