DistilBert weights initialization
I want to train aDistilBertModelfrom scratch with my own corpus, usingBertModelas the teacher model. Following DistilBert paper, what’s the best way to initialize the weights of my DistilBert with part of the teacher model’s weights?It seems both models are constructed using different classes (e.g.BertAttentioninBertModelandMultiheadAttentionin `DistilBertModel). In this case, I don’t know if I can just “assign” the teacher’s layers to the DistilBert’s layers…
2020-08-20T15:16:43Z
Hi@meisyarahd, you can find the distillation examplehere