On better training the infinite restricted Boltzmann machines
  The infinite restricted Boltzmann machine (iRBM) is an extension of the
classic RBM. It enjoys a good property of automatically deciding the size of
the hidden layer according to specific training data. With sufficient training,
the iRBM can achieve a competitive performance with that of the classic RBM.
However, the convergence of learning the iRBM is slow, due to the fact that the
iRBM is sensitive to the ordering of its hidden units, the learned filters
change slowly from the left-most hidden unit to right. To break this dependency
between neighboring hidden units and speed up the convergence of training, a
novel training strategy is proposed. The key idea of the proposed training
strategy is randomly regrouping the hidden units before each gradient descent
step. Potentially, a mixing of infinite many iRBMs with different permutations
of the hidden units can be achieved by this learning method, which has a
similar effect of preventing the model from over-fitting as the dropout. The
original iRBM is also modified to be capable of carrying out discriminative
training. To evaluate the impact of our method on convergence speed of learning
and the model's generalization ability, several experiments have been performed
on the binarized MNIST and CalTech101 Silhouettes datasets. Experimental
results indicate that the proposed training strategy can greatly accelerate
learning and enhance generalization ability of iRBMs.
