Text memmap index file cascade structure
MR https://github.com/NVIDIA/NeMo/pull/6683 added functionality to avoid collisions but in the case of bionemo the shared data on daraco OCI is structured in a way where every .csv has a .idx in the same directory. We would benefit by having a flag https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py#L474 where if True use the non cascade logic and the default can be set to False. This would allow bionemo to reuse its previously done indices and not introduce errors that the cascade introduction was designed to fix.

Current with Nemo 1.20
index_mapping_dir = /data/uniref202104_esm2/uf50/train 
fn = /data/uniref202104_esm2/uf50/train/x048.csv 
idx_fn = /data/uniref202104_esm2/uf50/train/data/uniref202104_esm2/uf50/train/x048.csv.idx

Desired: Rest of Bionemo / nemo 1.18
index_mapping_dir = /data/uniref202104_esm2/uf50/train 
fn = /data/uniref202104_esm2/uf50/train/x048.csv 
idx_fn = /data/uniref202104_esm2/uf50/train/x048.csv.idx


BIONEMO_IMAGE="nvcr.io/nvidian/cvai_bnmo_trng/bionemo-esm2-001:dev" 
