Load weight from local ckpt file
I have download a standard bert ckpt file ,,but how can I load this weights into model?I have read the document:图片829×265 15 KBand  try some method like:config = BertConfig.from_json_file('./bert_model/bert_config.json')model =TFBertModel.from_pretrained('bert_model/bert_model.ckpt',config=config)orconfig = BertConfig.from_json_file('./bert_model/bert_config.json')model = TFBertModel(config).load_weights('bert_model/bert_model.ckpt')but it seem like doesn’t  work.
2021-02-23T10:05:59Z
Hi@Sniper, I’m not very familiar with the TensorFlow API oftransformersbut I think the following should work:config = BertConfig.from_pretrained("path/to/your/bert/directory")
model = TFBertModel.from_pretrained("path/to/your/bert/directory", config=config)If that doesn’t work, can you share the error that you get?
2021-02-23T10:16:26Z
lewtun:from_pretrainedthanks for your reply.I still have some question here.shouldpath/to/your/bert/directoryispath/orpath/model.ckptIf I usepathhere, there is an error:OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5'] found in directory bert_model/ orfrom_ptset to FalseIf I usepath/model.ckpt,there is another error:**OSError: Unable to load weights from h5 file. If you tried to load a TF 2.0 model from a PyTorch checkpoint, please set from_pt=True.It seem like the code can’t recognize my ckpt file.code is like this:config = BertConfig.from_json_file('./bert_model/bert_config.json')model = TFBertModel.from_pretrained('bert_model/bert_model.ckpt', config=config)
2021-02-23T10:42:31Z
As far as I know, thepath/to/your/bert/directoryin the `from_pretrained function should point to the root of the directory where the model / tokenizer files are stored, not individual files.Can you share the contents of your directory? It seems there might be some files missing. Also, are you trying to load a model that was trained in PyTorch or TensorFlow?
2021-02-23T10:54:57Z
Hi,@lewtunI show my directory in the first paragraph. I can show it again.(I download this file atGitHub - google-research/bert: TensorFlow code and pre-trained models for BERT)It can be load by other framework likebert4keras.In fact,I have a runable model write by bert4keras and I try to rewrite the code to tensorflow2(bert4keras base on tf1).
2021-02-23T11:10:28Z
Thanks for clarification - I see in thedocsthat one can indeed pointfrom_pretraineda TF checkpoint file:A path or url to a tensorflow index checkpoint file (e.g,./tf_model/model.ckpt.index). In this case,from_tfshould be set toTrueand a configuration object should be provided asconfigargument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.It seems that what you’re missing is thefrom_tf=Trueargument, so maybe something like the following works:config = BertConfig.from_pretrained("path/to/your/bert/directory")
model = TFBertModel.from_pretrained("path/to/bert_model.ckpt.index", config=config, from_tf=True)I’m not sure whether the config should be loaded withfrom_pretrainedorfrom_json_filebut maybe you can test both to see which one works
2021-02-23T11:22:35Z
I addfrom_tfin it.But it have an error(the version oftransformersis the lastest one)TypeError: ('Keyword argument not understood:', 'from_tf')It seem like parameterfrom_tfjust use for pytorch,and tensorflow just have parameterfrom_pt.docs are different in pytorch and tensorflow, tf version ofdocsjust havefrom_pt.
2021-02-23T12:51:37Z
Ah now I understand the source of yourOSError: it seems thatfrom_pretrainedexpects your model to be serialised astf_model.h5instead of the TensorFlow checkpoint format. You can see this in the source code here:transformers/modeling_tf_utils.py at master · huggingface/transformers · GitHubIf you’re happy to use the PyTorch API, it seems that itmightbe possible to load.ckptfiles withfrom_pretrained:transformers/modeling_utils.py at master · huggingface/transformers · GitHubIn that case you could try something likeconfig = BertConfig.from_pretrained("path/to/your/bert/directory")
model = BertModel.from_pretrained("path/to/bert_model.ckpt.index", config=config, from_tf=True)
2021-02-23T13:03:21Z
Maybe I  should use pytorch load.ckptand save the model.Then load with tensorflow?I think I can use albert insteadThanks a lot for your patience help. Happy Chinese New Year!
2021-02-23T13:09:12Z
Sniper:Maybe I should use pytorch load.ckptand save the model.Then load with tensorflow?Yes that might work - good luck and happy chinese new year to you too!