TypeError: compute_loss() got an unexpected keyword argument 'labels'
Hi there,I’ve been trying to further pre-train BERT on my own data using the example scriptrun_mlm.py.The code I’m using to train is the following:python3 transformers/examples/tensorflow/language-modeling/run_mlm.py \
--model_name_or_path bert-base-cased \
--validation_split_percentage 20 \
--line_by_line \
--learning_rate 2e-5 \
--do_train \
--do_eval \
--per_device_train_batch_size 128 \
--per_device_eval_batch_size 256 \
--num_train_epochs 4 \
--output_dir output \ 
--train_file text.txtThe full output can be seenhere.In particular, I’m getting an errorTypeError: compute_loss() got an unexpected keyword argument 'labels'and I’m not really sure how to fix it.I also ran the same code in colab using the same parameters, data, etc and everything works as expected and training runs without issues.OS VersionUbuntu 20.04.4 LTSTensorflow version2.8.0Python version3.8.10Any help is much appreciated.Thank you!
2022-03-22T14:48:27Z
cc@Rocketknight1and@joaogante
2022-03-22T16:37:32Z
Hi@silkski, we changed what some of thecompute_lossfunctions do internally in a relatively recent version ofTransformers. As a result, if you’re using an older version, the example scripts for the current version won’t work for you. Try updating on your local machine (e.g.pip install --upgrade transformers) and let us know if that fixes it!
2022-03-22T19:02:35Z
Complementing@Rocketknight1’s answer – TF/Keras 2.8 defined a newcompute_lossfunction, which clashed with one of our internal functions. Because of that, updatingtransformersshould fix it
2022-03-24T11:28:03Z
Thank you so much@Rocketknight1and@joaogante. Upgrading transformers did the trick.