Covariance-aware Feature Alignment with Pre-computed Source Statistics
  for Test-time Adaptation
  The accuracy of deep neural networks is degraded when the distribution of
features in the test environment (target domain) differs from that of the
training (source) environment. To mitigate the degradation, test-time
adaptation (TTA), where a model adapts to the target domain without access to
the source dataset, can be used in the test environment. However, the existing
TTA methods lack feature distribution alignment between the source and target
domains, which unsupervised domain adaptation mainly addresses, because
accessing the source dataset is prohibited in the TTA setting. In this paper,
we propose a novel TTA method, named Covariance-Aware Feature alignment (CAFe),
which explicitly aligns the source and target feature distributions at test
time. To perform alignment without accessing the source data, CAFe uses
auxiliary feature statistics (mean and covariance) pre-computed on the source
domain, which are lightweight and easily prepared. Further, to improve
efficiency and stability, we propose feature grouping, which splits the feature
dimensions into groups according to their correlations by using spectral
clustering to avoid degeneration of the covariance matrix. We empirically show
that CAFe outperforms prior TTA methods on a variety of distribution shifts.
