Geometry of Linear Convolutional Networks
  We study the family of functions that are represented by a linear
convolutional neural network (LCN). These functions form a semi-algebraic
subset of the set of linear maps from input space to output space. In contrast,
the families of functions represented by fully-connected linear networks form
algebraic sets. We observe that the functions represented by LCNs can be
identified with polynomials that admit certain factorizations, and we use this
perspective to describe the impact of the network's architecture on the
geometry of the resulting function space. We further study the optimization of
an objective function over an LCN, analyzing critical points in function space
and in parameter space, and describing dynamical invariants for gradient
descent. Overall, our theory predicts that the optimized parameters of an LCN
will often correspond to repeated filters across layers, or filters that can be
decomposed as repeated filters. We also conduct numerical and symbolic
experiments that illustrate our results and present an in-depth analysis of the
landscape for small architectures.
