S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural
  Networks via Guided Distribution Calibration
  Previous studies dominantly target at self-supervised learning on real-valued
networks and have achieved many promising results. However, on the more
challenging binary neural networks (BNNs), this task has not yet been fully
explored in the community. In this paper, we focus on this more difficult
scenario: learning networks where both weights and activations are binary,
meanwhile, without any human annotated labels. We observe that the commonly
used contrastive objective is not satisfying on BNNs for competitive accuracy,
since the backbone network contains relatively limited capacity and
representation ability. Hence instead of directly applying existing
self-supervised methods, which cause a severe decline in performance, we
present a novel guided learning paradigm from real-valued to distill binary
networks on the final prediction distribution, to minimize the loss and obtain
desirable accuracy. Our proposed method can boost the simple contrastive
learning baseline by an absolute gain of 5.5~15% on BNNs. We further reveal
that it is difficult for BNNs to recover the similar predictive distributions
as real-valued models when training without labels. Thus, how to calibrate them
is key to address the degradation in performance. Extensive experiments are
conducted on the large-scale ImageNet and downstream datasets. Our method
achieves substantial improvement over the simple contrastive learning baseline,
and is even comparable to many mainstream supervised BNN methods. Code is
available at https://github.com/szq0214/S2-BNN.
