Tools to parallelize Megatron tokenization
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Is your feature request related to a problem? Please describe.**

I recently heard about the parallel data curation tools in NeMo, which allows one to run download, clean, and de-dup data using multiple nodes.

https://developer.nvidia.com/blog/curating-trillion-token-datasets-introducing-nemo-data-curator/

That's very exciting.  Thanks for putting this together.

The NeMo script to tokenize data for Megatron supports running multiple workers on a single node:

https://github.com/NVIDIA/NeMo/blob/main/scripts/nlp_language_modeling/preprocess_data_for_megatron.py

For large datasets, I'd like to go even further and parallelize this task across multiple nodes.

**Describe the solution you'd like**

About a year ago, I wrote up some tools to do this.  I opened a similar ticket on Megatron-LM with more info:

https://github.com/NVIDIA/Megatron-LM/issues/492

Would you be willing to accept something like this in NeMo?

Those existing files would need to be refreshed, because the indexed_dataset mmap file format has since changed.  I am willing to take some time to do that if that would be helpful.