How pretrain ELECTRA on custom dataset?
I have a 1GB raw text dataset in a niche domain. I want to train an ELECTRA model but I couldn’t find any tutorial/examples to do so. Can anyone help me? I tried using thesimpletransformerspackage, but it has memory issues at the moment and after a few epochs my colab session crashes.
2020-09-03T04:15:04Z
pinging@lysandre
2020-09-03T07:31:47Z
Do you want to fine-tune or pre-train an ELECTRA model? If you want to fine-tune it, you can leverage theexamples/run_language_modeling.pyscript.If you want to pre-train it, your best bet is to use the original implementation (in TF1) and then convert it to our library using our conversion script which ishere.Here’s a PRfor pre-training ELECTRA with our library but it’s not working right now and we don’t have the bandwidth to get back into it right now.
2020-09-03T08:36:30Z
@lysandreI am planning to pretrain the model, but I would like to also give a shot toexamples/run_language_modeling.pyscript. How can I finetune the base Electra model on MLM task on my data? The script is a bit ambigious.
2020-09-06T07:50:22Z
I’m not sure if fine-tuning ELECTRA with MLM is good idea since the main idea behind ELECTRA was to train it as a discriminator rather than generator and overcome the problems of MLM’sAny thoughts@lysandre
2020-09-06T10:53:40Z
ELECTRA training reimplementation and discussionResearchAfter months of development and debugging, I finally successfully train a model from scratch and replicate the official results.ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generatorsby Kevin Clark. Minh-Thang Luong. Quoc V. Le. Christopher D. ManningCode:electra_pytorchAFAIK, the closest reimplementation to the original one, taking care of many easily overlooked details (described below). 


AFAIK, the only one successfully validate itself by replica…