How to read the logs created by hugging face trainer?
args = TrainingArguments(
    output_dir='./results'
    evaluation_strategy="epoch",
    save_strategy="epoch",
    logging_dir='./logs',
    label_names=["labels"],
)How to read the training and validation losses?I am getting logs like this /content/logs/events.out.tfevents.1677075882.2041f82e97b0.219.0
2023-02-23T10:30:36Z
Thelogging_diris whereTensorboardfiles are stored. Since you’re not specifying--logging_strategy/--logging_steps, the Trainer is logging every 500 steps by default. You can visualize the data in a web browser with the following command:tensorboard --logdir content/logsThis will output some text in the terminal, which should contain a localhost address (something like “http://localhost:6006/”). Copy&paste that to a web browser (or do ctrl+cllick) and you should see a bunch of tensorboard plots.
2023-05-19T02:08:44Z
Hi@mapama247, sorry, should I write these command in the terminal or in the python environment?
2023-05-19T03:17:57Z
Hi@mapama247, I would appreciate if you could let me know how I can see my tensors my system is linux. my training code is as follow: and Results_Path=‘/home/nlpproject/’training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=15, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,logging_steps=5000,
                                per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-19T03:55:31Z
hi@illstart, I hop eu are fine. sorry, I wana visualize  the logs training and validation loss in trainer. would you please tell me how did u do that?training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=15, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,logging_steps=5000,
                                per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)

Trainer(model=model, args=training_args,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-19T19:30:07Z
In the terminal, as long as you have tensorboard installed.
2023-05-19T19:32:27Z
Do you see some files (with weird names) in/home/nlpproject/? In that case, open a terminal and do:tensorboard --logdir /home/nlpproject/
2023-05-20T01:04:04Z
@mapama247, many many thanks for your reply. Sorry, dusing training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell me how I can save the best model , my code is as followtraining_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-21T21:26:09Z
Ah okay… but this is a different problem that has nothing to do with tensorboardThe reason why you see that the checkpoints disappear is that you have the argumentsave_total_limit=1, which limits the number of saved checkpoints to 1. Just remove that or increase the number.
2023-05-22T01:28:02Z
Hi@mapama247, sorry, do you know how I can save the model for each epoch? regardless of it is the best model or not? I want to save model after each epoch. if I change teh steps to epoch it won’t save any checkpoints at the end. I want to save model after each epoch.this is my codetraining_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=15,evaluation_strategy="steps", logging_strategy="steps",save_strategy="steps",save_steps=500,seed=42,load_best_model_at_end=True,logging_steps=500,
report_to="tensorboard",per_device_train_batch_size=2,eval_steps=500,save_total_limit=2,per_device_eval_batch_size=2,
warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)
2023-05-26T14:02:11Z
Removesave_total_limit=2, setsave_strategy,evaluation_strategyandlogging_strategyto “epoch”, and removesave_steps=500,eval_steps=500andlogging_steps=500. This way you should get 15 folders with a different checkpoint each (one per epoch) and the best checkpoint directly on your output_dir.
2023-05-28T01:07:21Z
@mapama247many thanks for your reply. sorry, have u been use multiple GPU by using trainer API? I use it but the results are very strange in comparison with using 1 gpu. indeed I didn’t change anything in my code for using 1 GPU and let trainer use all gpu but results are very strange. can you please help me with that?
2023-07-11T20:39:00Z
I ran into the same issue with multi-gpu training. Were you able to resolve this?
2024-04-26T12:29:08Z
Simply getting the logs of the trainer object, you could usetrainer.state.log_historyTo get detailed logs of everything hf does under the hood though:is to disable the huggingface default logger and add your own custom python logger that writes to a file or writes to stdout. (What worked for me)after you disable the default handler, do remember tologging.enable_propogation()on the huggingface logger to be able to add your own logger for the “transformers” library.Checkout the definitions for enabling and disabling the default loggers and enabling propogation here:github.comhuggingface/transformers/blob/20081c743ee2ce31d178f2182c7466c3313adcd2/src/transformers/utils/logging.py#L233def disable_default_handler() -> None:"""Disable the default handler of the HuggingFace Transformers's root logger."""_configure_library_root_logger()assert _default_handler is not None_get_library_root_logger().removeHandler(_default_handler)def enable_default_handler() -> None:"""Enable the default handler of the HuggingFace Transformers's root logger."""_configure_library_root_logger()assert _default_handler is not None_get_library_root_logger().addHandler(_default_handler)def add_handler(handler: logging.Handler) -> None:"""adds a handler to the HuggingFace Transformers's root logger."""