Balancing Expert Utilization in Mixture-of-Experts Layers Embedded in
  CNNs
  This work addresses the problem of unbalanced expert utilization in
sparsely-gated Mixture of Expert (MoE) layers, embedded directly into
convolutional neural networks. To enable a stable training process, we present
both soft and hard constraint-based approaches. With hard constraints, the
weights of certain experts are allowed to become zero, while soft constraints
balance the contribution of experts with an additional auxiliary loss. As a
result, soft constraints handle expert utilization better and support the
expert specialization process, hard constraints mostly maintain generalized
experts and increase the model performance for many applications. Our findings
demonstrate that even with a single dataset and end-to-end training, experts
can implicitly focus on individual sub-domains of the input space. Experts in
the proposed models with MoE embeddings implicitly focus on distinct domains,
even without suitable predefined datasets. As an example, experts trained for
CIFAR-100 image classification specialize in recognizing different domains such
as sea animals or flowers without previous data clustering. Experiments with
RetinaNet and the COCO dataset further indicate that object detection experts
can also specialize in detecting objects of distinct sizes.
