Optimal Transport Kernels for Sequential and Parallel Neural
  Architecture Search
  Neural architecture search (NAS) automates the design of deep neural
networks. One of the main challenges in searching complex and non-continuous
architectures is to compare the similarity of networks that the conventional
Euclidean metric may fail to capture. Optimal transport (OT) is resilient to
such complex structure by considering the minimal cost for transporting a
network into another. However, the OT is generally not negative definite which
may limit its ability to build the positive-definite kernels required in many
kernel-dependent frameworks. Building upon tree-Wasserstein (TW), which is a
negative definite variant of OT, we develop a novel discrepancy for neural
architectures, and demonstrate it within a Gaussian process surrogate model for
the sequential NAS settings. Furthermore, we derive a novel parallel NAS, using
quality k-determinantal point process on the GP posterior, to select diverse
and high-performing architectures from a discrete set of candidates.
Empirically, we demonstrate that our TW-based approaches outperform other
baselines in both sequential and parallel NAS.
