Understanding Deep Learning Generalization by Maximum Entropy
  Deep learning achieves remarkable generalization capability with overwhelming
number of model parameters. Theoretical understanding of deep learning
generalization receives recent attention yet remains not fully explored. This
paper attempts to provide an alternative understanding from the perspective of
maximum entropy. We first derive two feature conditions that softmax regression
strictly apply maximum entropy principle. DNN is then regarded as approximating
the feature conditions with multilayer feature learning, and proved to be a
recursive solution towards maximum entropy principle. The connection between
DNN and maximum entropy well explains why typical designs such as shortcut and
regularization improves model generalization, and provides instructions for
future model development.
