How to set multiple files in `LineByLineTextDataset`?
In this Colab notebook, “01_how-to-train.ipynb”, when defining the trainingdataset, it uses only onefile_path, as shown below:dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./oscar.eo.txt",
    block_size=128,
)How can I give multiple files to thisdataset? Thank you!InHow to train a new language model from scratch using Transformers and Tokenizerstutorial, it defines a new class and uses the following linesrc_files = Path("./data/").glob("*-eval.txt") if evaluate else Path("./data/").glob("*-train.txt")to achieve this. But this tutorial does not leverage the newTrainerapproach as in the Colab notebook.
2023-02-28T07:30:21Z
You can use this:cat ./* > merged-fileAnd use this file which is created to make dataset