Add sequence packing and proper attention masking support for LLM pretraining?
Based on DM, this should be already supported in gpt dataset https://github.com/NVIDIA/Megatron-LM/blob/c7a1f82d761577e6ca0338d3521eac82f2aa0904/megatron/core/datasets/gpt_dataset.py#L189
closing
Right now when performing LLM pretraining, different documents are just naively concatenated without proper masking. This can become problematic for model training with longer context window, as noted in e.g. https://arxiv.org/abs/2402.08268. Llama 3 blog post also mentioned they applied proper masking e.g. see https://ai.meta.com/blog/meta-llama-3/.

Seems like https://github.com/NVIDIA/NeMo/pull/7945 added support for packed sequence with proper masking for SFT. Would it be possible to make pretraining support this too?


