How to load pretrained model with custom model layers
HiHow can I load pre-trained BART model weights into a custom model layer?I have a custom decoder layer with additionnn.Modulebut pretrain BART model likefacebook/bart-basehas prefixdecoder.layer.keys cant match the custom layer.from transformers.models.bart.modeling_bart import BartEncoder, BartDecoder

class BartDecoder(BartPretrainedModel):
  def __init__(self, config: BartConfig, embed_tokens: Optional[nn.Embedding] = None):
        super().__init__(config)
        # assume here are same 
        # ...

class BartDecoderLayer(nn.Module):
    def __init__(self, config: BartConfig):
        super().__init__()
        # assume we have original modules here 
        # ...
        self.some_new_modules1 = something_new
        self.some_new_modules2 = something_newHow can I load pretrainedfacebook/bart-baseinto this decoder and leave the rest ofself.some_new_modules1andself.some_new_modules2? Thefacebook/bart-basehas prefix keys likeendoer.layer..anddecoder.layer..Thanks
2023-09-06T12:36:23Z
I have same request. Has anyone could give an answer.
2023-09-12T16:30:02Z
I think the easiest way is following:follow the source code structure of pre-trained modelname all attributes containing pre-trained modules the same as in the pre-trained modelcode aPreTrainedModelclass and add thebase_model_prefixattribute to it with the name of the pre-trained model, for example"bert"or"mpnet".An example might be my small project.:GitHub - voorhs/hssa