Cannot find pre-trained SOP head of ALBERT
I’m trying to do continual MLM & SOP pre-training on my own dataset with a pre-trained ALBERt model, but the modelalbert-base-v2only have a pre-trained MLM head, but not SOP head. (I tried bothAlbertForPreTrainingandAlbertForMaskedLM, and usedmodel.from_pretrained("albert-base-v2")to load pre-trained weight.) Could you help me how to load the weight if it exists? If not, would it be important to do SOP training from scratch with the pre-trained ALBERT (with randomly initialized weight) or MLM would be enough (viarun_language_modeling.py)?