Export the embeddings layer of a pre-trained model as a standalone model
I would like to import a pre-trained model, extract the embeddings layer and export that as a standalone model. I did something like this:from transformers import AutoTokenizer, AutoModelForPreTraining

tokenizer = AutoTokenizer.from_pretrained("deepset/gelectra-large")
model = AutoModelForPreTraining.from_pretrained("deepset/gelectra-large")

embedding_model = model.electra.embeddings

embedding_model.save_pretrained("...")This returns the error that the object has nosave_pretrainedmethod. Any ideas how this is done?One idea that came to mind is to include the extracted embeddings layer in a new model object, but not sure what config to use for its initialisation.
2024-01-31T14:47:45Z
Solved. I just used the save methods from PyTorch on the extracted embedding objects.