How to show the learning rate during training
Hi everyoneI would like to know if it is possible to include thelearning ratevalue as part of the information presented during the training.The columnsAccuracy,F1,PrecisionandRecallwere added after setting a customcompute_metricsfunction.  And I would like to have theLearning Rateas well.image775×193 22.4 KBIs it possible to add it there?Thanks in advance
2022-01-22T16:29:56Z
Hi Milyiyo,I am curious how did you add accuracy, F1 Precision and Recall?
2022-01-22T19:36:36Z
Hi@Amalq,I used this:from sklearn.metrics import precision_recall_fscore_support, accuracy_score


def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='weighted')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }That function was provided to the Trainer like this:from transformers import Trainer

trainer = Trainer(model=model,
                  args=training_args,
                  compute_metrics=compute_metrics,
                  tokenizer=tokenizer,
                  eval_dataset=dataset['validation'],
                  train_dataset=dataset['train'],
                  data_collator=data_collator)I hope this can help
2022-01-22T19:38:25Z
Thank you so much@milyiyo, that is really helpful.
2022-01-23T09:04:56Z
Hi Alberto, yes it is possible to includelearning ratein the evaluation logs!Fortunately, thelog()method of the Trainer class is one of the methods that you can “subclass” to inject custom behaviour:TrainerSo, all you have to do is create your own Trainer subclass and override thelog()method like so:class MyTrainer(Trainer):
    def log(self, logs: Dict[str, float]) -> None:
        logs["learning_rate"] = self._get_learning_rate()
        super().log(logs)

trainer = MyTrainer(...)
trainer.train()You should now see the learning rate in the eval logs.Hope that helps, let me know if any questions.CheersHeiko
2022-01-23T17:01:55Z
Thanks@marshmellow77That solved my problem, look the new column:image785×201 22.6 KBThanks a lot for your help
2022-05-17T15:15:20Z
marshmellow77:def log(self, logs: Dict[str, float]) -> None:
        logs["learning_rate"] = self._get_learning_rate()
        super().log(logs)Hi@marshmellow77,I am trying to use the same logic to log the learning rates at each epoch. But, I am getting the following  error:AttributeError: ‘LMCL_Trainer’ object has no attribute ‘_get_learning_rate’LMCL_Trainer is the name of the subclass extended from Trainer. Providing the class below →class LMCL_Trainer(Trainer):
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        
    
    def log(self, logs: Dict[str, float]) -> None:
        logs["learning_rate"] = self._get_learning_rate()
        super().log(logs)I have the transformers version 3.3.0 installed.
2022-05-18T10:37:51Z
Hi@Vighnesh, the transformers library is currently at version 4.19. Lots has changed since v3.3, I recommend upgrading to the latest version to see if this solution works for you.CheersHeiko
2022-05-19T06:07:04Z
Thanks a lot. Yes the latest version has a lot of changes/improvements.The solution worked with transformers version 4.19
2023-09-26T13:44:10Z
HiI’ve tried the suggested solution with transformers version: 4.31.0, but for some reason the learning rate is not plotted as part of the evaluation logs.Any ideas why? (I do see that it is saved in the log history)
2023-11-14T17:21:48Z
@marshmellow77Hello. I want to use your solution to infer the learning rate. I am currently using Python 3.10. I get the following error:---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[36], line 1
----> 1 class MyTrainer(Trainer):
      2     def log(self, logs: Dict[str, float]) -> None:
      3         logs["learning_rate"] = self._get_learning_rate()

Cell In[36], line 2, in MyTrainer()
      1 class MyTrainer(Trainer):
----> 2     def log(self, logs: Dict[str, float]) -> None:
      3         logs["learning_rate"] = self._get_learning_rate()
      4         print(logs["learning_rate"])

NameError: name 'Dict' is not definedWhich specific module do you need to import it from?
2024-01-02T13:36:34Z
Hi@artyomboyko,Did you solve the error?Maybe was missing something like:from typing import Dict
2024-01-02T15:33:19Z
@milyiyoHello and Happy New year! Yes, you can find solutionhere.