Temperature as Uncertainty in Contrastive Learning
  Contrastive learning has demonstrated great capability to learn
representations without annotations, even outperforming supervised baselines.
However, it still lacks important properties useful for real-world application,
one of which is uncertainty. In this paper, we propose a simple way to generate
uncertainty scores for many contrastive methods by re-purposing temperature, a
mysterious hyperparameter used for scaling. By observing that temperature
controls how sensitive the objective is to specific embedding locations, we aim
to learn temperature as an input-dependent variable, treating it as a measure
of embedding confidence. We call this approach "Temperature as Uncertainty", or
TaU. Through experiments, we demonstrate that TaU is useful for
out-of-distribution detection, while remaining competitive with benchmarks on
linear evaluation. Moreover, we show that TaU can be learned on top of
pretrained models, enabling uncertainty scores to be generated post-hoc with
popular off-the-shelf models. In summary, TaU is a simple yet versatile method
for generating uncertainties for contrastive learning. Open source code can be
found at: https://github.com/mhw32/temperature-as-uncertainty-public.
