AugMax: Adversarial Composition of Random Augmentations for Robust
  Training
  Data augmentation is a simple yet effective way to improve the robustness of
deep neural networks (DNNs). Diversity and hardness are two complementary
dimensions of data augmentation to achieve robustness. For example, AugMix
explores random compositions of a diverse set of augmentations to enhance
broader coverage, while adversarial training generates adversarially hard
samples to spot the weakness. Motivated by this, we propose a data augmentation
framework, termed AugMax, to unify the two aspects of diversity and hardness.
AugMax first randomly samples multiple augmentation operators and then learns
an adversarial mixture of the selected operators. Being a stronger form of data
augmentation, AugMax leads to a significantly augmented input distribution
which makes model training more challenging. To solve this problem, we further
design a disentangled normalization module, termed DuBIN
(Dual-Batch-and-Instance Normalization), that disentangles the instance-wise
feature heterogeneity arising from AugMax. Experiments show that AugMax-DuBIN
leads to significantly improved out-of-distribution robustness, outperforming
prior arts by 3.03%, 3.49%, 1.82% and 0.71% on CIFAR10-C, CIFAR100-C, Tiny
ImageNet-C and ImageNet-C. Codes and pretrained models are available:
https://github.com/VITA-Group/AugMax.
