Neural Architecture Search by Estimation of Network Structure
  Distributions
  The influence of deep learning is continuously expanding across different
domains, and its new applications are ubiquitous. The question of neural
network design thus increases in importance, as traditional empirical
approaches are reaching their limits. Manual design of network architectures
from scratch relies heavily on trial and error, while using existing pretrained
models can introduce redundancies or vulnerabilities. Automated neural
architecture design is able to overcome these problems, but the most successful
algorithms operate on significantly constrained design spaces, assuming the
target network to consist of identical repeating blocks. While such approach
allows for faster search, it does so at the cost of expressivity. We instead
propose an alternative probabilistic representation of a whole neural network
structure under the assumption of independence between layer types. Our matrix
of probabilities is equivalent to the population of models, but allows for
discovery of structural irregularities, while being simple to interpret and
analyze. We construct an architecture search algorithm, inspired by the
estimation of distribution algorithms, to take advantage of this
representation. The probability matrix is tuned towards generating
high-performance models by repeatedly sampling the architectures and evaluating
the corresponding networks, while gradually increasing the model depth. Our
algorithm is shown to discover non-regular models which cannot be expressed via
blocks, but are competitive both in accuracy and computational cost, while not
utilizing complex dataflows or advanced training techniques, as well as
remaining conceptually simple and highly extensible.
