Disentangling Neural Architectures and Weights: A Case Study in
  Supervised Classification
  The history of deep learning has shown that human-designed problem-specific
networks can greatly improve the classification performance of general neural
models. In most practical cases, however, choosing the optimal architecture for
a given task remains a challenging problem. Recent architecture-search methods
are able to automatically build neural models with strong performance but fail
to fully appreciate the interaction between neural architecture and weights.
This work investigates the problem of disentangling the role of the neural
structure and its edge weights, by showing that well-trained architectures may
not need any link-specific fine-tuning of the weights. We compare the
performance of such weight-free networks (in our case these are binary networks
with {0, 1}-valued weights) with random, weight-agnostic, pruned and standard
fully connected networks. To find the optimal weight-agnostic network, we use a
novel and computationally efficient method that translates the hard
architecture-search problem into a feasible optimization problem.More
specifically, we look at the optimal task-specific architectures as the optimal
configuration of binary networks with {0, 1}-valued weights, which can be found
through an approximate gradient descent strategy. Theoretical convergence
guarantees of the proposed algorithm are obtained by bounding the error in the
gradient approximation and its practical performance is evaluated on two
real-world data sets. For measuring the structural similarities between
different architectures, we use a novel spectral approach that allows us to
underline the intrinsic differences between real-valued networks and
weight-free architectures.
