Neurashed: A Phenomenological Model for Imitating Deep Learning Training
  To advance deep learning methodologies in the next decade, a theoretical
framework for reasoning about modern neural networks is needed. While efforts
are increasing toward demystifying why deep learning is so effective, a
comprehensive picture remains lacking, suggesting that a better theory is
possible. We argue that a future deep learning theory should inherit three
characteristics: a \textit{hierarchically} structured network architecture,
parameters \textit{iteratively} optimized using stochastic gradient-based
methods, and information from the data that evolves \textit{compressively}. As
an instantiation, we integrate these characteristics into a graphical model
called \textit{neurashed}. This model effectively explains some common
empirical patterns in deep learning. In particular, neurashed enables insights
into implicit regularization, information bottleneck, and local elasticity.
Finally, we discuss how neurashed can guide the development of deep learning
theories.
