Towards a Universal Gating Network for Mixtures of Experts
  The combination and aggregation of knowledge from multiple neural networks
can be commonly seen in the form of mixtures of experts. However, such
combinations are usually done using networks trained on the same tasks, with
little mention of the combination of heterogeneous pre-trained networks,
especially in the data-free regime. This paper proposes multiple data-free
methods for the combination of heterogeneous neural networks, ranging from the
utilization of simple output logit statistics, to training specialized gating
networks. The gating networks decide whether specific inputs belong to specific
networks based on the nature of the expert activations generated. The
experiments revealed that the gating networks, including the universal gating
approach, constituted the most accurate approach, and therefore represent a
pragmatic step towards applications with heterogeneous mixtures of experts in a
data-free regime. The code for this project is hosted on github at
https://github.com/cwkang1998/network-merging.
