A Flexible Multi-Task Model for BERT Serving
  In this demonstration, we present an efficient BERT-based multi-task (MT)
framework that is particularly suitable for iterative and incremental
development of the tasks. The proposed framework is based on the idea of
partial fine-tuning, i.e. only fine-tune some top layers of BERT while keep the
other layers frozen. For each task, we train independently a single-task (ST)
model using partial fine-tuning. Then we compress the task-specific layers in
each ST model using knowledge distillation. Those compressed ST models are
finally merged into one MT model so that the frozen layers of the former are
shared across the tasks. We exemplify our approach on eight GLUE tasks,
demonstrating that it is able to achieve both strong performance and
efficiency. We have implemented our method in the utterance understanding
system of XiaoAI, a commercial AI assistant developed by Xiaomi. We estimate
that our model reduces the overall serving cost by 86%.
