Bloom: one of the largest open, multilingual NLP models awarded "Best HPC Collaboration" by the HPC Wire Readers' and Editors' Choice Awards

BigScience's global collaborative effort to develop Bloom, one of the world's largest open, multilingual NLP (Natural Language Processing) models, was recognized in the HPCwire Annual Readers' and Editors' Choice presented at the 2022 International Conference on High Performance Computing, Networking, Storage and Analysis (SC22) in Dallas, Texas. The award for best HPC collaboration of the year went to the GENCI, IDRIS and HuggingFace teams together.

BigScience was a community adventure as well as a research and engineering challenge. It brought together over 1,200 researchers from academia and industry (startups, SMEs, major groups) from 38 countries with the aim of developing and training BLOOM using a public HPC infrastructure, the GENCI (Grand Equipement National de Calcul Intensif) Jean Zay supercomputer hosted and operated at IDRIS (Institut du développement et des ressources en informatique scientifique, CNRS).

Orchestrated by Hugging Face, the open source AI start-up, 30 working groups set to work between mid-2021 and mid-2022, tackling all the different stages of building such a large language model (LLM) such as data governance, choice of data and input sources, modeling, model evaluation, engineering, including model optimization and scaling, generalization, ethical AI and legal frameworks, ROOTS open multilingual data introduction and RAIL open AI licensing.
The final and largest version of BLOOM with 176 billion parameters on 70 learned layers from a total amount of 1.61 terabytes of text covering 46 natural languages and 13 programming languages. The engineering workgroup achieved peak throughput with this transformer-based model on the latest nVIDIA A100-80 partition of the Jean Zay supercomputer (offering over 400 A100 GPUs out of >3100 of the total configuration).

reception of the award

From the right to the left : Pierre-François Lavallée (IDRIS), Tom Tabor (HPCWire), Stéphane Requena (GENCI)

With the support of experts from IDRIS, Hugging Face, Microsoft and nVIDIA (using the DeepSpeed-Megatron framework), the model achieved sustained performance of 156 TFlops/GPU (50% of FP32/BF16 peak performance). Training BLOOM-176B took 3.5 months, with 1,082,990 hours of computation on 48 Jean Zay nodes, requiring a total energy consumption of 433 MWh, representing a carbon footprint of just 25 tonnes of CO2 emissions.

BLOOM is available openly on a RAIL (Responsible AI Licenses) which limits the potentially dangerous use cases BLOOM could enable.

More information here:
https://huggingface.co/bigscience/bloom
https://arxiv.org/abs/2211.05100