Logs of training and validation loss
Hi, I made this post to see if anyone knows how can I save in the logs the results of my training and validation loss.I’m using this code:*training_args = TrainingArguments(*
*    output_dir='./results',          # output directory*
*    num_train_epochs=3,              # total number of training epochs*
*    per_device_train_batch_size=16,  # batch size per device during training*
*    per_device_eval_batch_size=16,   # batch size for evaluation*
*    warmup_steps=50,                 # number of warmup steps for learning rate scheduler*
*    weight_decay=0.01,               # strength of weight decay*
*    logging_dir='./logs',            # directory for storing logs*
*    logging_steps=20,*
*    evaluation_strategy="steps"*
*)*

*trainer = Trainer(*
*    model=model,                         # the instantiated 🤗 Transformers model to be trained*
*    args=training_args,                  # training arguments, defined above*
*    train_dataset=train_dataset,         # training dataset*
*    eval_dataset=val_dataset             # evaluation dataset*
*)*And I thought that using logging_dir and logging_steps would achieve that but in such logs all I see is this:*output_dir ^A"^X*
*^Toverwrite_output_dir ^B"^L*
*^Hdo_train ^B"^K*
*^Gdo_eval ^A"^N*

*do_predict ^B"^\*
*^Xevaluate_during_training ^B"^W*
*^Sevaluation_strategy ^A"^X*
*^Tprediction_loss_only ^B"^_*
*^[per_device_train_batch_size ^C"^^*
*^Zper_device_eval_batch_size ^C"^\*
*^Xper_gpu_train_batch_size ^A"^[*
*^Wper_gpu_eval_batch_size ^A"^_*
*^[gradient_accumulation_steps ^C"^[*
*^Weval_accumulation_steps ^A"^Q*
*^Mlearning_rate ^C"^P*
*^Lweight_decay ^C"^N*And it goes on like that.Any ideas will be welcome.My system instalation:-transformersversion: 3.4.0- Platform: Linux-3.10.0-1127.13.1.el7.x86_64-x86_64-with-centos-7.8.2003-Core- Python version: 3.6.8- PyTorch version (GPU?): 1.6.0 (False)- Tensorflow version (GPU?): not installed (NA)- Using GPU in script?: No- Using distributed or parallel set-up in script?: No
2020-12-17T11:57:49Z
Hi!I was also recently trying to save my loss values at eachlogging_stepsinto a .txt file.There might be a parameter I am unaware of, but meanwhile I pulled from git the latest version of the transformer library and slightly modified the trainer.py to include indef log(self, logs: Dict[str, float]) -> None: the following lines to save my logs into a .txt file:# TODO PRINT ADDED BY XXX
logSave = open('lossoutput.txt', 'a')
logSave.write(str(output) + '\n')
logSave.close()Happy to hear if there is a less ‘cowboy’ way to do this, one that would not require modifying trainer.py
2020-12-17T14:03:03Z
You could also subclassTrainerand override thelogmethod to do this (which is less cowboy-y).@lysandreis the logger master and might know a more clever way to directly redirect the logs from our logger.
2020-12-18T13:54:43Z
The things I’m thinking off are way more cowboy than what you’re doing@aberquand! I think@sgugger’s solution is the cleanest.You could redirect all the logs to a text file and then filter them out, but your approach here sounds better.
2021-10-04T09:51:06Z
I know this is late as hell, but I will leave this here for future reference and if anyone comes across this post. I think that even less cowboy way would be to use callback:class LogCallback(transformers.TrainerCallback):
    def on_evaluate(self, args, state, control, **kwargs):
        # calculate loss here

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    compute_metrics=compute_metrics,
    callbacks=[LogCallback],
)Another even less cowboy way (without implementing anything) is that when you use those logging_steps args etc. You can access those logs after training is complete:trainer.state.log_historyYou should have metrics and losses from all steps over training. Hope this will help someone in future.
2023-03-28T08:11:27Z
karkawal:trainer.state.log_historyThis is the way!
2023-04-03T02:52:28Z
karkawal:trainer.state.log_historyI have issues with the code that you provided.import matplotlib.pyplot as pltimport numpy as npfrom transformers import TrainerCallbackclass LogCallback(TrainerCallback):definit(self, state):self.state = statedef on_step_end(self, args, state, control, **kwargs):
    if state.global_step % 10 == 0:
        for name, value in state.log_history[-1].items():
            print(f"{name} at step {state.global_step}: {value}")training_args = TrainingArguments(output_dir=“gpt_model”,overwrite_output_dir=True,learning_rate=7e-5,weight_decay=0.01,num_train_epochs=350,logging_steps=50,save_total_limit=2,per_device_train_batch_size=3,#gradient_accumulation_steps=4,save_steps=10_000,evaluation_strategy=‘no’,#load_best_model_at_end=True)trainer = Trainer(model=mpm_model,args=training_args,data_collator=data_collator,train_dataset=mpm_dataset[‘train’],#compute_metrics=compute_metrics,callbacks=[LogCallback],)train_output = trainer.train()log_callback = LogCallback(state)I want to store the logs to plot the loss curves, how can I modify my code to fix this issue?I am not using the validation data as my entire data is used for training due to the small size of the dataset.
2023-04-03T02:56:30Z
Can you please help me with this issue. I want to store the log history of my training model so that I want to plot the loss curves.Here is my code that I tried to use and here I am just using the entire data as training data, I did not split my dataset due to its small size.Code:import matplotlib.pyplot as pltimport numpy as npfrom transformers import TrainerCallbackclass LogCallback(TrainerCallback):definit(self, state):self.state = statedef on_step_end(self, args, state, control, **kwargs):
    if state.global_step % 10 == 0:
        for name, value in state.log_history[-1].items():
            print(f"{name} at step {state.global_step}: {value}")training_args = TrainingArguments(output_dir=“gpt_model”,overwrite_output_dir=True,learning_rate=7e-5,weight_decay=0.01,num_train_epochs=350,logging_steps=50,save_total_limit=2,per_device_train_batch_size=3,#gradient_accumulation_steps=4,save_steps=10_000,evaluation_strategy=‘no’,#load_best_model_at_end=True)trainer = Trainer(model=mpm_model,args=training_args,data_collator=data_collator,train_dataset=mpm_dataset[‘train’],#compute_metrics=compute_metrics,callbacks=[LogCallback],)train_output = trainer.train()log_callback = LogCallback(state)Error:TypeError                                 Traceback (most recent call last)Cell In[31], line 3213                 print(f"{name} at step {state.global_step}: {value}")17 training_args = TrainingArguments(18     output_dir=“gpt_model”,19     overwrite_output_dir=True,(…)29     #load_best_model_at_end=True30 )—> 32 trainer = Trainer(33     model=mpm_model,34     args=training_args,35     data_collator=data_collator,36     train_dataset=mpm_dataset[‘train’],37     #compute_metrics=compute_metrics,38     callbacks=[LogCallback],39 )41 train_output = trainer.train()42 log_callback = LogCallback(state)File /scratch/kkonatha/kk/lib/python3.9/site-packages/transformers/trainer.py:519, in Trainer.init(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)517 default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to)518 callbacks = default_callbacks if callbacks is None else default_callbacks + callbacks→ 519 self.callback_handler = CallbackHandler(520     callbacks, self.model, self.tokenizer, self.optimizer, self.lr_scheduler521 )522 self.add_callback(PrinterCallback if self.args.disable_tqdm else DEFAULT_PROGRESS_CALLBACK)524 # Will be set to True byself._setup_loggers()on first call toself.log().File /scratch/kkonatha/kk/lib/python3.9/site-packages/transformers/trainer_callback.py:296, in CallbackHandler.init(self, callbacks, model, tokenizer, optimizer, lr_scheduler)294 self.callbacks =295 for cb in callbacks:→ 296     self.add_callback(cb)297 self.model = model298 self.tokenizer = tokenizerFile /scratch/kkonatha/kk/lib/python3.9/site-packages/transformers/trainer_callback.py:313, in CallbackHandler.add_callback(self, callback)312 def add_callback(self, callback):→ 313     cb = callback() if isinstance(callback, type) else callback314     cb_class = callback if isinstance(callback, type) else callback.class315     if cb_class in [c.classfor c in self.callbacks]:TypeError:init() missing 1 required positional argument: ‘state’
2023-05-18T07:35:55Z
Hi@perch,@sgugger, I hope you are well. I get this logs from trainer in my folder, How I can use them? open them? I wana to see training loss and validation loss.  many thanksCapture7705×90 4.22 KB
2023-05-18T22:57:14Z
HI@workpiece, I hope you are well. sorry, did you fix your problem to save the logs ? how did you visualized them? many thnaks for your help. this sis my argument:training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=3, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,logging_steps=5000,
                                per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()