Loading trained model with new vocab
I’m working with DistilBERT model. I created a few new vocab tokens and trained the model.tokenizer.add_tokens(["[NEW_TOKEN]"], special_tokens=True)
        model.resize_token_embeddings(len(tokenizer))
        tokenizer.save_pretrained(args.tokenizer_dir)Everything is working fine but I’m hitting some issues while loading the trained model from a check-point:$ model = BERTMODEL.from_pretrained(checkpoint_path) 

RuntimeError: Error(s) in loading state_dict for BERTMODEL:
    size mismatch for vocab_projector.weight: copying a param with shape torch.Size([30522, 768]) from checkpoint, the shape in current model is torch.Size([30524, 768]).Size mismatch 30522 vs 30524 because I added 2 new tokens in the vocab. I’m not sure how to pass the new vocab config when loading the model from a checkpoint:model = BERTMODEL.from_pretrained(checkpoint_path)  # <--- ????Any hint regarding what is missing in my code?
2022-03-04T16:48:34Z
I don’t know if this would help, but you could passignore_mismatched_sizes=Truewhen loading model.
2024-04-10T00:36:38Z
newbie001:working fine but I’m hitting some issues while loading the trI got the same issue as you, adding some new tokens to the tokenizer,with code:model.resize_token_embeddings(len(tokenizer))training and save the model is OK, but the problem happened when I try to load the modelthe raising error is:size mismatch for text_model.model.embed_tokens.weight: copying a param with shape torch.Size([65048, 2048]) from checkpoint, the shape in current model is torch.Size([65037, 2048]).Anyone can help please?Thanks a lot in advance!!