Multi-Domain Active Learning: Literature Review and Comparative Study
  Multi-domain learning (MDL) refers to learning a set of models
simultaneously, where each model is specialized to perform a task in a
particular domain. Generally, a high labeling effort is required in MDL, as
data needs to be labeled by human experts for every domain. Active learning
(AL) can be utilized in MDL to reduce the labeling effort by only using the
most informative data. The resultant paradigm is termed multi-domain active
learning (MDAL). In this work, we provide an exhaustive literature review for
MDAL on the relevant fields, including AL, cross-domain information sharing
schemes, and cross-domain instance evaluation approaches. It is found that the
few studies which have been directly conducted on MDAL cannot serve as
off-the-shelf solutions on more general MDAL tasks. To fill this gap, we
construct a pipeline of MDAL and present a comprehensive comparative study of
thirty different algorithms, which are established by combining six
representative MDL models and five commonly used AL strategies. We evaluate the
algorithms on six datasets involving textual and visual classification tasks.
In most cases, AL brings notable improvements to MDL, and the naive best vs.
second best (BvSB) Uncertainty strategy can perform competitively with the
state-of-the-art AL strategies. Besides, BvSB with the MAN model can
consistently achieve top or above-average performance on all the datasets.
Furthermore, we qualitatively analyze the behaviors of the well-performed
strategies and models, shedding light on their superior performance in the
comparison. Finally, we recommend to use BvSB with the MAN model in the
application of MDAL due to their good performance in the experiments.
