Consider refactoring CTC greedy decoding
@artbataev this may be related to your observation of TDT being faster than CTC in some cases. If I recall correctly, max does work on GPU so I don't know why I forced it onto CPU. Maybe the slice? Either way, if it fixes the slowdown let's move the .CPU() to after max calculation.

It might have been that the logprobs can be super long for 12 hours so I wanted to preserve memory, in which case the cost of speed is not under consideration but I suppose we can make it a premature optimization and select whether to move the tensor to CPU before or after depending on length of the tensor.  It is pretty common to run long audio CTC - NFA does it constantly. But on average, it is more common to run short audio batches  > It might have been that the logprobs can be super long for 12 hours so I wanted to preserve memory

The tensor pointed to by "prediction" already existed on GPU. It is just being copied to CPU in this code. We should not expect any memory increase from my suggestion. In fact, we will see a CPU memory decrease. That's true. Can you see if moving it onto CPU at the end before return is sufficient to remove this bottleneck then let's do it in your other pr 
**Is your feature request related to a problem? Please describe.**

As part of my work on #8673 , I have been profiling to make sure we have low latency and high throughput. I have observed that this line can sometimes take https://github.com/NVIDIA/NeMo/blob/57444ae9b63afc5419a16948f7d032a1a9f7dc7f/nemo/collections/asr/parts/submodules/ctc_greedy_decoding.py#L218 several milliseconds. To be concrete, running at batch size 10, the model is taking 167 milliseconds, while decoding is taking 29 milliseconds, with almost all of the time spent on this max() operation. This is documented below.

<img width="1244" alt="image" src="https://github.com/NVIDIA/NeMo/assets/4767568/623c66d5-57b9-4d9d-8e35-62cfad51a702">

It is not acceptable to have model inference be bottlenecked on non-GPU work like this.

I uploaded a gzipped version of the nsight systems file if you would like to view it.

[report25.nsys-rep.gz](https://github.com/NVIDIA/NeMo/files/14950389/report25.nsys-rep.gz)

The most obvious way to handle this situation is to do the "max()" operation on the GPU. I previously did a refactor that sped up CTC decoding by improving copy from GPU to CPU https://github.com/NVIDIA/NeMo/pull/8521, but it seems that is better to copy only after the max() operation is done. I did not do this in my previous PR, because (1) I didn't think it would be a concern and (2) that change is much more intrusive. _greedy_decode_logprobs is built with running on only a single element at a time rather than a batch. Maybe vmap can save us...

BTW, why is this only sometimes slow? I'm not certain, but it probably has to do with the CPU cache. It's working directly on pinned memory. The cache line may be getting evicted only sometimes?