Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic
  Uncertainty
  Numerous recent works utilize bi-Lipschitz regularization of neural network
layers to preserve relative distances between data instances in the feature
spaces of each layer. This distance sensitivity with respect to the data aids
in tasks such as uncertainty calibration and out-of-distribution (OOD)
detection. In previous works, features extracted with a distance sensitive
model are used to construct feature covariance matrices which are used in
deterministic uncertainty estimation or OOD detection. However, in cases where
there is a distribution over tasks, these methods result in covariances which
are sub-optimal, as they may not leverage all of the meta information which can
be shared among tasks. With the use of an attentive set encoder, we propose to
meta learn either diagonal or diagonal plus low-rank factors to efficiently
construct task specific covariance matrices. Additionally, we propose an
inference procedure which utilizes scaled energy to achieve a final predictive
distribution which is well calibrated under a distributional dataset shift.
