Learning credit assignment
  Deep learning has achieved impressive prediction accuracies in a variety of
scientific and industrial domains. However, the nested non-linear feature of
deep learning makes the learning highly non-transparent, i.e., it is still
unknown how the learning coordinates a huge number of parameters to achieve a
decision making. To explain this hierarchical credit assignment, we propose a
mean-field learning model by assuming that an ensemble of sub-networks, rather
than a single network, are trained for a classification task. Surprisingly, our
model reveals that apart from some deterministic synaptic weights connecting
two neurons at neighboring layers, there exist a large number of connections
that can be absent, and other connections can allow for a broad distribution of
their weight values. Therefore, synaptic connections can be classified into
three categories: very important ones, unimportant ones, and those of
variability that may partially encode nuisance factors. Therefore, our model
learns the credit assignment leading to the decision, and predicts an ensemble
of sub-networks that can accomplish the same task, thereby providing insights
toward understanding the macroscopic behavior of deep learning through the lens
of distinct roles of synaptic weights.
