Exponentially Increasing the Capacity-to-Computation Ratio for
  Conditional Computation in Deep Learning
  Many state-of-the-art results obtained with deep networks are achieved with
the largest models that could be trained, and if more computation power was
available, we might be able to exploit much larger datasets in order to improve
generalization ability. Whereas in learning algorithms such as decision trees
the ratio of capacity (e.g., the number of parameters) to computation is very
favorable (up to exponentially more parameters than computation), the ratio is
essentially 1 for deep neural networks. Conditional computation has been
proposed as a way to increase the capacity of a deep neural network without
increasing the amount of computation required, by activating some parameters
and computation "on-demand", on a per-example basis. In this note, we propose a
novel parametrization of weight matrices in neural networks which has the
potential to increase up to exponentially the ratio of the number of parameters
to computation. The proposed approach is based on turning on some parameters
(weight matrices) when specific bit patterns of hidden unit activations are
obtained. In order to better control for the overfitting that might result, we
propose a parametrization that is tree-structured, where each node of the tree
corresponds to a prefix of a sequence of sign bits, or gating units, associated
with hidden units.
