Restricted Boltzmann Machines as Models of Interacting Variables
  We study the type of distributions that Restricted Boltzmann Machines (RBMs)
with different activation functions can express by investigating the effect of
the activation function of the hidden nodes on the marginal distribution they
impose on observed binary nodes. We report an exact expression for these
marginals in the form of a model of interacting binary variables with the
explicit form of the interactions depending on the hidden node activation
function. We study the properties of these interactions in detail and evaluate
how the accuracy with which the RBM approximates distributions over binary
variables depends on the hidden node activation function and on the number of
hidden nodes. When the inferred RBM parameters are weak, an intuitive pattern
is found for the expression of the interaction terms which reduces
substantially the differences across activation functions. We show that the
weak parameter approximation is a good approximation for different RBMs trained
on the MNIST dataset. Interestingly, in these cases, the mapping reveals that
the inferred models are essentially low order interaction models.
