Further pre-train roberta model
I have gone through this codetrain from scratchand understood how to pre-train a model from scratch. I have the following doubts in this codeWhat doesblock_sizeinLineByLineTextDatasetrepresent?If I want to further pretrain Robert-base model (instead of training from scratch) using my own corpus, what are the changes I have to make in the above code besides the following changesfrom transformers import RobertaForMaskedLM, RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
model = RobertaForMaskedLM('roberta-base')I am aware that I need not to train tokenizer from scratch.@thomwolf@julien-c
2020-07-14T12:12:56Z
Hi@mr-nlp, I think you can use the samerun_language_modellingscript to further pre-train roberta, just provide your own datasets.block_sizeis used formax_length