What are eval/runtime, eval/steps_per_second, eval/samples_per_second graphs?
I train a BERT model and use WandB for monitoring.I can see following graphs and I am trying to understand those.eval/runtimeeval/steps_per_secondeval/samples_per_secondAlso have train/global_step graph.I would like to what are these and differents of each.
2023-08-31T18:36:41Z
Here is my understanding. All these graphs are related to the time it takes to evaluate on the evaluation_dataset that you pass to the Trainer.eval\runtime- Total time it takes to evaluate (in seconds)eval/steps_per_second- Avg time to evaluate each batcheval/samples_per_second- Avg time to evaluate each sampleYou can confirm this by plugging in your values to this example below.eval\runtime = 60len(eval_dataset) = 200per_device_eval_batch_size = 10Then the values for the other graphs are as follows.eval/samples_per_second = eval\runtime / len(eval_dataset)eval/steps_per_second = eval\runtime / (len(eval_dataset) / per_device_eval_batch_size)Still looking for concrete evidence on this, but the calculation works for my use case. I will update this answer if I find the relevant code/documentation!
2023-09-07T17:23:17Z
I think this is all correct, but I found something for training speed metrics that seems counter-intuitive that I thought would be worth sharing for anyone finding this discussion like myself. Someone can correct me if I misunderstood:For eval, it is all based on the time for each evaluation phase (end of epoch or at step interval), so that you get several values over the course of a training run, whenever you stop to evaluate. For training, you get a single value that considers the total runtime oftrainer.train()including evaluation phases in between, since it is onlycalled a the end of training.Personally I am interested in measuring training samples per second in the same way as in evaluation: including only the phases where the trainer is actually trying to take training steps, and perhaps getting many values per run as a result that could later be averaged, etc. This way if I am trying to improve my training speed and compare to others, I can focus on whats going on in the training steps. I may be in the minority on this though and the idea thattrain_runtimespans the time spent intrainer.train()is sensible. The undesirable part is you can’t usetrain/total_flosand thetrain/train_runtimeto compute a ballpark estimate of the TFLOPS you achieved in training steps, sincetrain/total_flosseems to only include (estimated) operations in training steps but train_runtime includes non-training phases.Anyway, I think it is worth documenting somewhere, and would love to know if anyone has a good solution for measuring training TFLOPs for comparison with what they could theoretically be getting from their GPU(s), or if my desired approach doesn’t actually work.