Learning by Ignoring, with Application to Domain Adaptation
  Learning by ignoring, which identifies less important things and excludes
them from the learning process, is broadly practiced in human learning and has
shown ubiquitous effectiveness. There has been psychological studies showing
that learning to ignore certain things is a powerful tool for helping people
focus. In this paper, we explore whether this useful human learning methodology
can be borrowed to improve machine learning. We propose a novel machine
learning framework referred to as learning by ignoring (LBI). Our framework
automatically identifies pretraining data examples that have large domain shift
from the target distribution by learning an ignoring variable for each example
and excludes them from the pretraining process. We formulate LBI as a
three-level optimization framework where three learning stages are involved:
pretraining by minimizing the losses weighed by ignoring variables; finetuning;
updating the ignoring variables by minimizing the validation loss. A
gradient-based algorithm is developed to efficiently solve the three-level
optimization problem in LBI. Experiments on various datasets demonstrate the
effectiveness of our framework.
