Training Restricted Boltzmann Machines with Binary Synapses using the
  Bayesian Learning Rule
  Restricted Boltzmann machines (RBMs) with low-precision synapses are much
appealing with high energy efficiency. However, training RBMs with binary
synapses is challenging due to the discrete nature of synapses. Recently Huang
proposed one efficient method to train RBMs with binary synapses by using a
combination of gradient ascent and the message passing algorithm under the
variational inference framework. However, additional heuristic clipping
operation is needed. In this technical note, inspired from Huang's work , we
propose one alternative optimization method using the Bayesian learning rule,
which is one natural gradient variational inference method. As opposed to
Huang's method, we update the natural parameters of the variational symmetric
Bernoulli distribution rather than the expectation parameters. Since the
natural parameters take values in the entire real domain, no additional
clipping is needed. Interestingly, the algorithm in \cite{huang2019data} could
be viewed as one first-order approximation of the proposed algorithm, which
justifies its efficacy with heuristic clipping.
