Conditional Information Gain Networks
  Deep neural network models owe their representational power to the high
number of learnable parameters. It is often infeasible to run these largely
parametrized deep models in limited resource environments, like mobile phones.
Network models employing conditional computing are able to reduce computational
requirements while achieving high representational power, with their ability to
model hierarchies. We propose Conditional Information Gain Networks, which
allow the feed forward deep neural networks to execute conditionally, skipping
parts of the model based on the sample and the decision mechanisms inserted in
the architecture. These decision mechanisms are trained using cost functions
based on differentiable Information Gain, inspired by the training procedures
of decision trees. These information gain based decision mechanisms are
differentiable and can be trained end-to-end using a unified framework with a
general cost function, covering both classification and decision losses. We
test the effectiveness of the proposed method on MNIST and recently introduced
Fashion MNIST datasets and show that our information gain based conditional
execution approach can achieve better or comparable classification results
using significantly fewer parameters, compared to standard convolutional neural
network baselines.
