SpanBERT, ELECTRA, MARGE from scratch?
Hey everyone! I am incredibly grateful for this tutorial on training a language model from scratch:How to train a new language model from scratch using Transformers and TokenizersI really want to expand this to contiguous masking of longer token sequences (e.g. [mask-5], [mask-8]). I have begun looking into how to write a custom DataCollator for this, but suspect I will also need to make some changes to the model as well.Has anyone looked into this and can point me to any resources?Thank you!
2022-01-04T22:18:29Z
Found something useful on StackOverflow for this:stackoverflow.comBest way of using hugging face's Mask Filling for more than 1 masked token at a timepython, neural-network, nlp, huggingface-transformersasked byuser3472360on11:50AM - 02 Apr 20 UTC
2022-01-06T14:07:26Z
This tutorial on Keras Code Examples is the most useful thing I have found so far on this:keras.ioKeras documentation: End-to-end Masked Language Modeling with BERTKeras documentation
2022-01-08T21:45:58Z
Hello ConnorGreat to see you here! I will not be able to help you, I’m going to ping@nielsrhere, maybe he could help. Sorry for delay!
2022-01-09T12:50:20Z
Hi,I’ve seen that SpanBERT models are on thehub, but we haven’t added the model itself yet to the library.This would be a great project actually:contribute SpanBERT to HuggingFace Transformers, based on themodeling file. This will be relatively easy, as the authors already used HuggingFace’s implementation of BERT and tweaked it a little bit. The only difference isthis class. We could then call the modelSpanBertModelin the library, and add aSpanBertForPreTrainingsimilar toBertForPreTrainingthat includes the heads necessary for pre-training.add a script to theexamplesdirectory, which could be called run_span_mlm.py (similar to run_mlm.py). This can be based on the files definedhere(Facebook open-sourced everything!).If anyone is interested in contributing, let me know!
2023-07-22T01:35:46Z
@nielsrI am also interested in fine-tuning BERT (or any BERT like pre-trained model) using span masking. Can I know whether this is supported via transformers library. If so can you refer me to any resource available?