MetaGater: Fast Learning of Conditional Channel Gated Networks via
  Federated Meta-Learning
  While deep learning has achieved phenomenal successes in many AI
applications, its enormous model size and intensive computation requirements
pose a formidable challenge to the deployment in resource-limited nodes. There
has recently been an increasing interest in computationally-efficient learning
methods, e.g., quantization, pruning and channel gating. However, most existing
techniques cannot adapt to different tasks quickly. In this work, we advocate a
holistic approach to jointly train the backbone network and the channel gating
which enables dynamical selection of a subset of filters for more efficient
local computation given the data input. Particularly, we develop a federated
meta-learning approach to jointly learn good meta-initializations for both
backbone networks and gating modules, by making use of the model similarity
across learning tasks on different nodes. In this way, the learnt meta-gating
module effectively captures the important filters of a good meta-backbone
network, based on which a task-specific conditional channel gated network can
be quickly adapted, i.e., through one-step gradient descent, from the
meta-initializations in a two-stage procedure using new samples of that task.
The convergence of the proposed federated meta-learning algorithm is
established under mild conditions. Experimental results corroborate the
effectiveness of our method in comparison to related work.
