Fine tune Masked Language Model on custom dataset
Hi, I am new in Bert, I want to fine tune a plain text corpus(only text, without any label or other information) to get hidden state layer for specific word/document embedding.Thus I think the suitable model for my task is the masked  language model, however I found this only example demohttps://huggingface.co/blog/how-to-train, which is not universal and clear.The original examples are here, but no mlm model availablehttps://huggingface.co/transformers/master/custom_datasets.htmlDoes anyone have clear example for this case?To be specific, I don’t know how to pass true label in MLM task for loss and gradient update. Also while using the Trainer, where should I insert -mlm command?Thank you in advance
2020-08-20T15:25:27Z
Hi@smalltoken, what is the issue withhttps://huggingface.co/blog/how-to-train?This colab should help you. It walks you through,How to to train tokenizer from scratchCreateRobertaModelusing the configuse theDataCollatorForLanguageModeling, which handle the maskingand train usingTrainer.
2020-08-20T15:46:58Z
Thanks for your kind reply.I put the code and error message in this open issue on github.https://github.com/huggingface/transformers/issues/6616My problem is mainly about ‘index out of range in self’ error, given that I alredy set a very low max_length value in tokenizer. I was also confused using native PyTorch to train before, in which I don’t know how to handle mask and pass labels to model. But if DataCollatorForLanguageModeling and Trainer works well, that doesn’t matter.Could you help verify if there is any mistake in my piece of code? Thank you!
2020-08-20T15:52:28Z
Sure, what is./bert-large-casedin the code, is it a pre-trained bert or did you create it ? If yes, can you post the config ?
2020-08-20T15:52:52Z
AlsoDataCollatorForLanguageModelinghandles masking
2020-08-20T21:09:39Z
Because of proxyerror in my working company, I mannually downloaded the pre trained model froms3.amazonaws.com.(So maybe I ignored some items to be download?) And./bert-large-casedis a folder which contains these following items :bert-large-cased-tf_model.h5config.jsonpytorch_model.binvocab.txtHere is what inside the config:attention_probs_dropout_prob:0.1directionality:“bidi”hidden_act:“gelu”hidden_dropout_prob:0.1hidden_size:1024initializer_range:0.02intermediate_size:4096max_position_embeddings:512num_attention_heads:16num_hidden_layers:24pooler_fc_size:768pooler_num_attention_heads:12pooler_num_fc_layers:3pooler_size_per_head:128pooler_type:“first_token_transform”type_vocab_size:2vocab_size:28996