Semi-discrete optimization through semi-discrete optimal transport: a
  framework for neural architecture search
  In this paper we introduce a theoretical framework for semi-discrete
optimization using ideas from optimal transport. Our primary motivation is in
the field of deep learning, and specifically in the task of neural architecture
search. With this aim in mind, we discuss the geometric and theoretical
motivation for new techniques for neural architecture search (in a companion
paper we show that algorithms inspired by our framework are competitive with
contemporaneous methods). We introduce a Riemannian-like metric on the space of
probability measures over a semi-discrete space $\mathbb{R}^d \times
\mathcal{G}$ where $\mathcal{G}$ is a finite weighted graph. With such
Riemmanian structure in hand, we derive formal expressions for the gradient
flow of a relative entropy functional, as well as second order dynamics for the
optimization of said energy. Then, with the aim of providing a rigorous
motivation for the gradient flow equations derived formally, we also consider
an iterative procedure known as minimizing movement scheme (i.e., Implicit
Euler scheme, or JKO scheme) and apply it to the relative entropy with respect
to a suitable cost function. For some specific choices of metric and cost, we
rigorously show that the minimizing movement scheme of the relative entropy
functional converges to the gradient flow process provided by the formal
Riemannian structure. This flow coincides with a system of reaction-diffusion
equations on $\mathbb{R}^d$.
