Freezing weights of new tokens in the input embedding
HiI have added new tokens to my tokenizer and I would like to freeze the weights of the original set of tokens in the input embedding layer while allowing the weights of the new tokens to be trained. I’ve triedexisting_vocab = tokenizer.get_vocab()
for token_id in existing_vocab.values():
    if token_id < tokenizer.vocab_size - len(new_tokens):
        embedding = model.get_input_embeddings().weight[token_id]
        embedding = embedding.detach()
        embedding.requires_grad = FalseBut subsequently, when I check the weights of the new tokens, they are not frozen.Any ideas how I can work around this?
2023-12-20T10:24:41Z
Got the same problem while trying to freeze part of the parameters in the embedding layer of a pretrained CLM
2024-09-25T09:20:09Z
@KhaiKithave you found a solution for this?