Analysis and Design of Convolutional Networks via Hierarchical Tensor
  Decompositions
  The driving force behind convolutional networks - the most successful deep
learning architecture to date, is their expressive power. Despite its wide
acceptance and vast empirical evidence, formal analyses supporting this belief
are scarce. The primary notions for formally reasoning about expressiveness are
efficiency and inductive bias. Expressive efficiency refers to the ability of a
network architecture to realize functions that require an alternative
architecture to be much larger. Inductive bias refers to the prioritization of
some functions over others given prior knowledge regarding a task at hand. In
this paper we overview a series of works written by the authors, that through
an equivalence to hierarchical tensor decompositions, analyze the expressive
efficiency and inductive bias of various convolutional network architectural
features (depth, width, strides and more). The results presented shed light on
the demonstrated effectiveness of convolutional networks, and in addition,
provide new tools for network design.
