Theory-Inspired Path-Regularized Differential Network Architecture
  Search
  Despite its high search efficiency, differential architecture search (DARTS)
often selects network architectures with dominated skip connections which lead
to performance degradation. However, theoretical understandings on this issue
remain absent, hindering the development of more advanced methods in a
principled way. In this work, we solve this problem by theoretically analyzing
the effects of various types of operations, e.g. convolution, skip connection
and zero operation, to the network optimization. We prove that the
architectures with more skip connections can converge faster than the other
candidates, and thus are selected by DARTS. This result, for the first time,
theoretically and explicitly reveals the impact of skip connections to fast
network optimization and its competitive advantage over other types of
operations in DARTS. Then we propose a theory-inspired path-regularized DARTS
that consists of two key modules: (i) a differential group-structured sparse
binary gate introduced for each operation to avoid unfair competition among
operations, and (ii) a path-depth-wise regularization used to incite search
exploration for deep architectures that often converge slower than shallow ones
as shown in our theory and are not well explored during the search.
Experimental results on image classification tasks validate its advantages.
