Bounded Rational Decision-Making in Feedforward Neural Networks
  Bounded rational decision-makers transform sensory input into motor output
under limited computational resources. Mathematically, such decision-makers can
be modeled as information-theoretic channels with limited transmission rate.
Here, we apply this formalism for the first time to multilayer feedforward
neural networks. We derive synaptic weight update rules for two scenarios,
where either each neuron is considered as a bounded rational decision-maker or
the network as a whole. In the update rules, bounded rationality translates
into information-theoretically motivated types of regularization in weight
space. In experiments on the MNIST benchmark classification task for
handwritten digits, we show that such information-theoretic regularization
successfully prevents overfitting across different architectures and attains
results that are competitive with other recent techniques like dropout,
dropconnect and Bayes by backprop, for both ordinary and convolutional neural
networks.
