RepNAS: Searching for Efficient Re-parameterizing Blocks
  In the past years, significant improvements in the field of neural
architecture search(NAS) have been made. However, it is still challenging to
search for efficient networks due to the gap between the searched constraint
and real inference time exists. To search for a high-performance network with
low inference time, several previous works set a computational complexity
constraint for the search algorithm. However, many factors affect the speed of
inference(e.g., FLOPs, MACs). The correlation between a single indicator and
the latency is not strong. Currently, some re-parameterization(Rep) techniques
are proposed to convert multi-branch to single-path architecture which is
inference-friendly. Nevertheless, multi-branch architectures are still
human-defined and inefficient. In this work, we propose a new search space that
is suitable for structural re-parameterization techniques. RepNAS, a one-stage
NAS approach, is present to efficiently search the optimal diverse branch
block(ODBB) for each layer under the branch number constraint. Our experimental
results show the searched ODBB can easily surpass the manual diverse branch
block(DBB) with efficient training.
