fasttext cbow on dclm400
A continuous-bag-of-words model trained on https://huggingface.co/datasets/mlfoundations/dclm-pool-400m-1x
the cbow model was trained with https://github.com/facebookresearch/fastText/
the dataset was downloaded with git-lfs
the dataset commit was: f20ae752116ce7b4ab15d31e1e40b094229bf911
the files decompressed with:
parallel "zstd --keep --stdout -d {} | jq .text > {/}.txt" ::: /root/lfs/dclm-pool-400m-1x/*.jsonl.zst
concatenated with
cat *.txt > CC_SHARD_ALL.jsonl.txt
the sha256sum CC_SHARD_ALL.json.txt is
576e4e79e76b9ca24dc77a8da0df17ad5efc9c5ca16c9a86f62e7b7b4ae8c640 CC_SHARD_ALL.jsonl.txt
then the fasttext model was trained with defaults settings from
compiled with gcc 13.3.1
fasttext-repo (main branch) with the commit hash 1142dc4c4ecbc19cc16eee5cdd28472e689267e6
training command:
prlimit -m 3200000000 fasttext cbow -input CC_SHARD_ALL.jsonl.txt -output fasttext_models/model
the exact fasttext binary is included in this repo as fasttext
the decompression and concatenating took a few hours.
the model training took 100 hours on 8 cores plus a few hours to read in the words (fasttext)