Anytime Dense Prediction with Confidence Adaptivity
  Anytime inference requires a model to make a progression of predictions which
might be halted at any time. Prior research on anytime visual recognition has
mostly focused on image classification. We propose the first unified and
end-to-end approach for anytime dense prediction. A cascade of "exits" is
attached to the model to make multiple predictions. We redesign the exits to
account for the depth and spatial resolution of the features for each exit. To
reduce total computation, and make full use of prior predictions, we develop a
novel spatially adaptive approach to avoid further computation on regions where
early predictions are already sufficiently confident. Our full method, named
anytime dense prediction with confidence (ADP-C), achieves the same level of
final accuracy as the base model, and meanwhile significantly reduces total
computation. We evaluate our method on Cityscapes semantic segmentation and
MPII human pose estimation: ADP-C enables anytime inference without sacrificing
accuracy while also reducing the total FLOPs of its base models by 44.4% and
59.1%. We compare with anytime inference by deep equilibrium networks and
feature-based stochastic sampling, showing that ADP-C dominates both across the
accuracy-computation curve. Our code is available at
https://github.com/liuzhuang13/anytime .
