What does from_pretrained do?
Hi, I am trying to visualize whatfrom_pretraineddoFor eg:from transformers import BertModel
finbert_bertmodel=BertModel.from_pretrained('ProsusAI/finbert')#TheBert Model transformer outputting raw hidden-states without any specific head on top.#WhileFinbert has BertForSequenceClassification architectureThis is my understanding of from_pretrained for this piece of code. Here, it loads all the corresponding weights of Finbert into the architecture of BertModel.Similarly, for a different model, say:finbert_maskedLM=AutoModelForMaskedLM.from_pretrained('ProsusAI/finbert')Weights of Finbert will be loaded into the architecture of MaskedLM, the weights of last layer of MaskedLM will be randomly initialised as Finbert architecture is of Sequence classification, and not of MaskedLM.Is my understanding correct?Please add additional details and any link which has additional information.
2022-12-20T17:01:22Z
@nielsr@sgugger,Can you please check this question ?
2024-09-10T08:37:02Z
Hi,SoProsusAI/finbert · Hugging Faceindeed has aBertForSequenceClassificationarchitecture as can be seen from the “architectures” field in config.json.When you do:from transformers import BertModel
finbert_bertmodel=BertModel.from_pretrained('ProsusAI/finbert')then what happens is, all weights of the BERT model get initialized from the checkpoint on the hub, and the weights of the sequence classifier on top (which the finbert repo included) will be discarded, as BertModel does not include a sequence classification head on top.When you do:from transformers import BertForMaskedLM
finbert_bertmodel=BertForMaskedLM.from_pretrained('ProsusAI/finbert')then all weights of the BERT model get initialized from the checkpoint on the hub, the sequence classifier weights gets discarded, and a randomly initialized language modeling head on top will be added.