Dropout Prediction Variation Estimation Using Neuron Activation Strength
  It is well-known that deep neural networks generate different predictions
even given the same model configuration and training dataset. It thus becomes
more and more important to study prediction variation, the variation of the
predictions on a given input example, in neural network models. Dropout has
been commonly used in various applications to quantify prediction variations.
However, using dropout in practice can be expensive as it requires running
dropout inferences many times to estimate prediction variation.
  We study how to estimate dropout prediction variation in a resource-efficient
manner. We demonstrate that we can use neuron activation strengths to estimate
dropout prediction variation under different dropout settings and on a variety
of tasks using three large datasets, MovieLens, Criteo, and EMNIST. Our
approach provides an inference-once alternative to estimate dropout prediction
variation as an auxiliary task. Moreover, we demonstrate that using activation
features from a subset of the neural network layers can be sufficient to
achieve variation estimation performance almost comparable to that of using
activation features from all layers, thus reducing resources even further for
variation estimation.
