Draft: Cuda graphs greedy rnnt inference
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. This is now at parity with the original blank_as_pad greedy RNN-T decoding implementation. Currently, I can decode librispeech test clean in 1 min, 39 seconds, while the original implementation runs in 2 min, 21 seconds, at batch size 16. This may seem disappointing, but it is clear that there is still CPU overhead being counted because loading data off disk is not fast enough. In addition, this result is using float32, not float16.

That is to say that the WER output here is 0:

https://github.com/NVIDIA/NeMo/pull/7976/files#diff-e87be798c4c213fbdf438595c4c06898e1dabea0757aaa60334331cbf9f44cb1R83

However, this still needs to be cleaned up. It has a lot of cruft, including an attempt to use TensorRT from before my realization that the problem was CPU overhead, not slowness in the GPU implementation. This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. This PR was closed because it has been inactive for 7 days since being marked as stale.
# What does this PR do ?

Greedy RNNT inference currently sees the GPU idle 90% of the time because of CPU overheads. This experiments with using cuda graphs to do greedy inference, so that we can decoder at much higher throughputs. Very WIP. I began to work on this because I discovered that using TensorRT was not helpful at all, because it speeds up such a small portion of the code right now.

**Collection**: ASR

# Changelog 
- Will have to do after cleaning up

Not ready for serious review.