Scalable Training of Language Models using JAX pjit and TPUv4
  Modern large language models require distributed training strategies due to
their size. The challenges of efficiently and robustly training them are met
with rapid developments on both software and hardware frontiers. In this
technical report, we explore challenges and design decisions associated with
developing a scalable training framework, and present a quantitative analysis
of efficiency improvements coming from adopting new software and hardware
solutions.
