Data format in run_lm_fine_tuning.py
Hello everyone, I would like to ask for help with the following:I want to fine-tuning of a language model for text generation, for this I will userun_lm_fine_tuning.pyI would like to know what is the optimal way to train the model. Can it be trained with a pure csv file, or should the file have some kind of pre-processing?Or is it necessary to train it with an instance of theDatasetclass ?Thanks for the help
2020-09-07T04:24:53Z
Hi@kintaro,run_lm_fine_tuning.pyis now renamed torun_language_modeling.py, you can fine ithere.The data file can be in one of the two formatsLine by line: which means each example is on it’s own line seperated by\n. Set the--line_by_linecommand line argumentOr just a plain text file from which examples will be sampled .Thisnotebookalso walks through how a language model can be trained.
2020-09-08T18:31:21Z
@valhallaThanks a lot for your answer. Let me ask:1.With line_by_line is there any limit in the max number of tokens per line? maybe 512?it seems like the easy way is with the plain text file.