When Does Contrastive Learning Preserve Adversarial Robustness from
  Pretraining to Finetuning?
  Contrastive learning (CL) can learn generalizable feature representations and
achieve the state-of-the-art performance of downstream tasks by finetuning a
linear classifier on top of it. However, as adversarial robustness becomes
vital in image classification, it remains unclear whether or not CL is able to
preserve robustness to downstream tasks. The main challenge is that in the
self-supervised pretraining + supervised finetuning paradigm, adversarial
robustness is easily forgotten due to a learning task mismatch from pretraining
to finetuning. We call such a challenge 'cross-task robustness
transferability'. To address the above problem, in this paper we revisit and
advance CL principles through the lens of robustness enhancement. We show that
(1) the design of contrastive views matters: High-frequency components of
images are beneficial to improving model robustness; (2) Augmenting CL with
pseudo-supervision stimulus (e.g., resorting to feature clustering) helps
preserve robustness without forgetting. Equipped with our new designs, we
propose AdvCL, a novel adversarial contrastive pretraining framework. We show
that AdvCL is able to enhance cross-task robustness transferability without
loss of model accuracy and finetuning efficiency. With a thorough experimental
study, we demonstrate that AdvCL outperforms the state-of-the-art
self-supervised robust learning methods across multiple datasets (CIFAR-10,
CIFAR-100, and STL-10) and finetuning schemes (linear evaluation and full model
finetuning).
