The Technology Behind BLOOM Training
stas
In recent years, training ever larger language models has become the norm. While the issues of those models' not being released for further study is frequently discussed, the hidden knowledge about how to train such models rarely gets any attention. This article aims to change this by shedding some light on the technology and engineering behind training such models both in terms of hardware and software on the example of the 176B parameter language model [BLOOM](https://huggingface.co/bigscience/bloom).

But first we would like to thank the companies and key people and groups that made the amazing feat of training a 176 Billion parameter model by a small group of dedicated people possible.

Then the hardware setup and main technological components will be discussed.

![BLOOM](assets/86_bloom_megatron_deepspeed/bloom-banner.png)

Here's a quick summary of project:

|               |                             |
| :