Unsupervised Difficulty Estimation with Action Scores
  Evaluating difficulty and biases in machine learning models has become of
extreme importance as current models are now being applied in real-world
situations. In this paper we present a simple method for calculating a
difficulty score based on the accumulation of losses for each sample during
training. We call this the action score. Our proposed method does not require
any modification of the model neither any external supervision, as it can be
implemented as callback that gathers information from the training process. We
test and analyze our approach in two different settings: image classification,
and object detection, and we show that in both settings the action score can
provide insights about model and dataset biases.
