Pool of Experts: Realtime Querying Specialized Knowledge in Massive
  Neural Networks
  In spite of the great success of deep learning technologies, training and
delivery of a practically serviceable model is still a highly time-consuming
process. Furthermore, a resulting model is usually too generic and heavyweight,
and hence essentially goes through another expensive model compression phase to
fit in a resource-limited device like embedded systems. Inspired by the fact
that a machine learning task specifically requested by mobile users is often
much simpler than it is supported by a massive generic model, this paper
proposes a framework, called Pool of Experts (PoE), that instantly builds a
lightweight and task-specific model without any training process. For a
realtime model querying service, PoE first extracts a pool of primitive
components, called experts, from a well-trained and sufficiently generic
network by exploiting a novel conditional knowledge distillation method, and
then performs our train-free knowledge consolidation to quickly combine
necessary experts into a lightweight network for a target task. Thanks to this
train-free property, in our thorough empirical study, PoE can build a fairly
accurate yet compact model in a realtime manner, whereas it takes a few minutes
per query for the other training methods to achieve a similar level of the
accuracy.
