SMILE: Self-Distilled MIxup for Efficient Transfer LEarning
  To improve the performance of deep learning, mixup has been proposed to force
the neural networks favoring simple linear behaviors in-between training
samples. Performing mixup for transfer learning with pre-trained models however
is not that simple, a high capacity pre-trained model with a large
fully-connected (FC) layer could easily overfit to the target dataset even with
samples-to-labels mixed up. In this work, we propose SMILE - Self-Distilled
Mixup for EffIcient Transfer LEarning. With mixed images as inputs, SMILE
regularizes the outputs of CNN feature extractors to learn from the mixed
feature vectors of inputs (sample-to-feature mixup), in addition to the mixed
labels. Specifically, SMILE incorporates a mean teacher, inherited from the
pre-trained model, to provide the feature vectors of input samples in a
self-distilling fashion, and mixes up the feature vectors accordingly via a
novel triplet regularizer. The triple regularizer balances the mixup effects in
both feature and label spaces while bounding the linearity in-between samples
for pre-training tasks. Extensive experiments have been done to verify the
performance improvement made by SMILE, in comparisons with a wide spectrum of
transfer learning algorithms, including fine-tuning, L2-SP, DELTA, and RIFLE,
even with mixup strategies combined. Ablation studies show that the vanilla
sample-to-label mixup strategies could marginally increase the linearity
in-between training samples but lack of generalizability, while SMILE
significantly improve the mixup effects in both label and feature spaces with
both training and testing datasets. The empirical observations backup our
design intuition and purposes.
