Approximate Neural Architecture Search via Operation Distribution
  Learning
  The standard paradigm in Neural Architecture Search (NAS) is to search for a
fully deterministic architecture with specific operations and connections. In
this work, we instead propose to search for the optimal operation distribution,
thus providing a stochastic and approximate solution, which can be used to
sample architectures of arbitrary length. We propose and show, that given an
architectural cell, its performance largely depends on the ratio of used
operations, rather than any specific connection pattern in typical search
spaces; that is, small changes in the ordering of the operations are often
irrelevant. This intuition is orthogonal to any specific search strategy and
can be applied to a diverse set of NAS algorithms. Through extensive validation
on 4 data-sets and 4 NAS techniques (Bayesian optimisation, differentiable
search, local search and random search), we show that the operation
distribution (1) holds enough discriminating power to reliably identify a
solution and (2) is significantly easier to optimise than traditional
encodings, leading to large speed-ups at little to no cost in performance.
Indeed, this simple intuition significantly reduces the cost of current
approaches and potentially enable NAS to be used in a broader range of
applications.
