Training a tokenizer
Hi everyone,I have a dataset in which sentences have been segmented into words. How do I use it to train a BPE or SentencePiece tokenizer?Thank you
2022-08-03T09:04:00Z
Check outthisnotebook from huggingface’s github, or the second step ofthisother notebook about how to pretrain a LM from scratch