Regularized Dynamic Boltzmann Machine with Delay Pruning for
  Unsupervised Learning of Temporal Sequences
  We introduce Delay Pruning, a simple yet powerful technique to regularize
dynamic Boltzmann machines (DyBM). The recently introduced DyBM provides a
particularly structured Boltzmann machine, as a generative model of a
multi-dimensional time-series. This Boltzmann machine can have infinitely many
layers of units but allows exact inference and learning based on its
biologically motivated structure. DyBM uses the idea of conduction delays in
the form of fixed length first-in first-out (FIFO) queues, with a neuron
connected to another via this FIFO queue, and spikes from a pre-synaptic neuron
travel along the queue to the post-synaptic neuron with a constant period of
delay. Here, we present Delay Pruning as a mechanism to prune the lengths of
the FIFO queues (making them zero) by setting some delay lengths to one with a
fixed probability, and finally selecting the best performing model with fixed
delays. The uniqueness of structure and a non-sampling based learning rule in
DyBM, make the application of previously proposed regularization techniques
like Dropout or DropConnect difficult, leading to poor generalization. First,
we evaluate the performance of Delay Pruning to let DyBM learn a
multidimensional temporal sequence generated by a Markov chain. Finally, we
show the effectiveness of delay pruning in learning high dimensional sequences
using the moving MNIST dataset, and compare it with Dropout and DropConnect
methods.
