ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves
  Zero-Shot Generalization
  We propose a multitask pretraining approach ZeroPrompt for zero-shot
generalization, focusing on task scaling and zero-shot prompting. While
previous models are trained on only a few dozen tasks, we scale to 1,000 tasks
for the first time using real-world data. This leads to a crucial discovery
that task scaling can be an efficient alternative to model scaling; i.e., the
model size has little impact on performance with an extremely large number of
tasks. Our results show that task scaling can substantially improve training
efficiency by 30 times in FLOPs. Moreover, we present a prompting method that
incorporates a genetic algorithm to automatically search for the best prompt
for unseen tasks, along with a few other improvements. Empirically, ZeroPrompt
substantially improves both the efficiency and the performance of zero-shot
learning across a variety of academic and production datasets.
