SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural
  Language Understanding
  Prevailing deep models are single-purpose and overspecialize at individual
tasks. However, when being extended to new tasks, they typically forget
previously learned skills and learn from scratch. We address this issue by
introducing SkillNet-NLU, a general-purpose model that stitches together
existing skills to learn new tasks more effectively. The key feature of our
approach is that it is sparsely activated guided by predefined skills.
Different from traditional dense models that always activate all the model
parameters, SkillNet-NLU only activates parts of the model parameters whose
skills are relevant to the target task. When learning for a new task, our
approach precisely activates required skills and also provides an option to add
new skills. We evaluate on natural language understandings tasks and have the
following findings. First, with only one model checkpoint, SkillNet-NLU
performs better than task-specific fine-tuning and two multi-task learning
baselines (i.e., dense model and Mixture-of-Experts model) on six tasks.
Second, sparsely activated pre-training further improves the overall
performance. Third, SkillNet-NLU significantly outperforms baseline systems
when being extended to new tasks.
