Model Composition: Can Multiple Neural Networks Be Combined into a
  Single Network Using Only Unlabeled Data?
  The diversity of deep learning applications, datasets, and neural network
architectures necessitates a careful selection of the architecture and data
that match best to a target application. As an attempt to mitigate this
dilemma, this paper investigates the idea of combining multiple trained neural
networks using unlabeled data. In addition, combining multiple models into one
can speed up the inference, result in stronger, more capable models, and allows
us to select efficient device-friendly target network architectures. To this
end, the proposed method makes use of generation, filtering, and aggregation of
reliable pseudo-labels collected from unlabeled data. Our method supports using
an arbitrary number of input models with arbitrary architectures and
categories. Extensive performance evaluations demonstrated that our method is
very effective. For example, for the task of object detection and without using
any ground-truth labels, an EfficientDet-D0 trained on Pascal-VOC and an
EfficientDet-D1 trained on COCO, can be combined to a RetinaNet-ResNet50 model,
with a similar mAP as the supervised training. If fine-tuned in a
semi-supervised setting, the combined model achieves +18.6%, +12.6%, and +8.1%
mAP improvements over supervised training with 1%, 5%, and 10% of labels.
