More Informative Performance Callback Data - NemoMegatron GPT
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale. Hi!

Any update on this issue? I'm interested at least on logging `tokens_per_sec` to be able to compare performance between different configs.

Thanks,

Toni
Made some updates to the Docker image to support Pytorch NGC 23.10 so that i can consume the performance updates in NCCL 2.19.3. I am running some benchmarks with a prepared NemoMegatron 7B w/ Wikipedia and was wondering how i get a more representative performance callback data.
For example
```
Epoch 0: :   3%|▎         | 9000/301500 [1:42:42<55:37:46, v_num=0-56, reduced_train_loss=nan.0, global_step=9e+3, consumed_samples=6.3e+5, train_step_timing in s=0.615, val_loss=7.040]
Epoch 0, global step 9000: 'val_loss' reached 7.04238 (best 7.04238), saving model to '/data/nemo_megatron/nemo_experiments/megatron_gpt/2023-11-29_23-30-56/checkpoints/megatron_gpt--val_loss=7.04-step=9000-consumed_samples=630000.0.ckpt' as top 3
```
Wondering how i can get MFU or tokens/sec from this. In the legacy Megatron-LM implementation it had forward-compute/backward-compute and collective timings reported per step.