Continuously Constructive Deep Neural Networks
  Traditionally, deep learning algorithms update the network weights whereas
the network architecture is chosen manually, using a process of trial and
error. In this work, we propose two novel approaches that automatically update
the network structure while also learning its weights. The novelty of our
approach lies in our parameterization where the depth, or additional
complexity, is encapsulated continuously in the parameter space through control
parameters that add additional complexity. We propose two methods: In tunnel
networks, this selection is done at the level of a hidden unit, and in budding
perceptrons, this is done at the level of a network layer; updating this
control parameter introduces either another hidden unit or another hidden
layer. We show the effectiveness of our methods on the synthetic two-spirals
data and on two real data sets of MNIST and MIRFLICKR, where we see that our
proposed methods, with the same set of hyperparameters, can correctly adjust
the network complexity to the task complexity.
