NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language
  Processing
  Neural Architecture Search (NAS) is a promising and rapidly evolving research
area. Training a large number of neural networks requires an exceptional amount
of computational power, which makes NAS unreachable for those researchers who
have limited or no access to high-performance clusters and supercomputers. A
few benchmarks with precomputed neural architectures performances have been
recently introduced to overcome this problem and ensure more reproducible
experiments. However, these benchmarks are only for the computer vision domain
and, thus, are built from the image datasets and convolution-derived
architectures. In this work, we step outside the computer vision domain by
leveraging the language modeling task, which is the core of natural language
processing (NLP). Our main contribution is as follows: we have provided search
space of recurrent neural networks on the text datasets and trained 14k
architectures within it; we have conducted both intrinsic and extrinsic
evaluation of the trained models using datasets for semantic relatedness and
language understanding evaluation; finally, we have tested several NAS
algorithms to demonstrate how the precomputed results can be utilized. We
believe that our results have high potential of usage for both NAS and NLP
communities.
