FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours
  Protein structure prediction is an important method for understanding gene
translation and protein function in the domain of structural biology. AlphaFold
introduced the Transformer model to the field of protein structure prediction
with atomic accuracy. However, training and inference of the AlphaFold model
are time-consuming and expensive because of the special performance
characteristics and huge memory consumption. In this paper, we propose
FastFold, a highly efficient implementation of the protein structure prediction
model for training and inference. FastFold includes a series of GPU
optimizations based on a thorough analysis of AlphaFold's performance.
Meanwhile, with Dynamic Axial Parallelism and Duality Async Operation, FastFold
achieves high model parallelism scaling efficiency, surpassing existing popular
model parallelism techniques. Experimental results show that FastFold reduces
overall training time from 11 days to 67 hours and achieves 7.5-9.5X speedup
for long-sequence inference. Furthermore, We scaled FastFold to 512 GPUs and
achieved an aggregate of 6.02 PetaFLOPs with 90.1% parallel efficiency. The
implementation can be found at https://github.com/hpcaitech/FastFold
