Growing an architecture for a neural network
  We propose a new kind of automatic architecture search algorithm. The
algorithm alternates pruning connections and adding neurons, and it is not
restricted to layered architectures only. Here architecture is an arbitrary
oriented graph with some weights (along with some biases and an activation
function), so there may be no layered structure in such a network. The
algorithm minimizes the complexity of staying within a given error. We
demonstrate our algorithm on the brightness prediction problem of the next
point through the previous points on an image. Our second test problem is the
approximation of the bivariate function defining the brightness of a black and
white image. Our optimized networks significantly outperform the standard
solution for neural network architectures in both cases.
