Efficient Contrastive Learning via Novel Data Augmentation and
  Curriculum Learning
  We introduce EfficientCL, a memory-efficient continual pretraining method
that applies contrastive learning with novel data augmentation and curriculum
learning. For data augmentation, we stack two types of operation sequentially:
cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum
learning by incrementing the augmentation degree for each difficulty step.
After data augmentation is finished, contrastive learning is applied on
projected embeddings of original and augmented examples. When finetuned on GLUE
benchmark, our model outperforms baseline models, especially for sentence-level
tasks. Additionally, this improvement is capable with only 70% of computational
memory compared to the baseline model.
