Bias-Variance Trade-Off in Hierarchical Probabilistic Models Using
  Higher-Order Feature Interactions
  Hierarchical probabilistic models are able to use a large number of
parameters to create a model with a high representation power. However, it is
well known that increasing the number of parameters also increases the
complexity of the model which leads to a bias-variance trade-off. Although it
is a classical problem, the bias-variance trade-off between hidden layers and
higher-order interactions have not been well studied. In our study, we propose
an efficient inference algorithm for the log-linear formulation of the
higher-order Boltzmann machine using a combination of Gibbs sampling and
annealed importance sampling. We then perform a bias-variance decomposition to
study the differences in hidden layers and higher-order interactions. Our
results have shown that using hidden layers and higher-order interactions have
a comparable error with a similar order of magnitude and using higher-order
interactions produce less variance for smaller sample size.
