Full-attention based Neural Architecture Search using Context
  Auto-regression
  Self-attention architectures have emerged as a recent advancement for
improving the performance of vision tasks. Manual determination of the
architecture for self-attention networks relies on the experience of experts
and cannot automatically adapt to various scenarios. Meanwhile, neural
architecture search (NAS) has significantly advanced the automatic design of
neural architectures. Thus, it is appropriate to consider using NAS methods to
discover a better self-attention architecture automatically. However, it is
challenging to directly use existing NAS methods to search attention networks
because of the uniform cell-based search space and the lack of long-term
content dependencies. To address this issue, we propose a full-attention based
NAS method. More specifically, a stage-wise search space is constructed that
allows various attention operations to be adopted for different layers of a
network. To extract global features, a self-supervised search algorithm is
proposed that uses context auto-regression to discover the full-attention
architecture. To verify the efficacy of the proposed methods, we conducted
extensive experiments on various learning tasks, including image
classification, fine-grained image recognition, and zero-shot image retrieval.
The empirical results show strong evidence that our method is capable of
discovering high-performance, full-attention architectures while guaranteeing
the required search efficiency.
