Deploying to Riva with self-trained ASR model, but unable to infer audio 
**Describe the bug**

I used NeMo to train a Mandarin ASR model (QuartzNet), and then I converted it to .rmir format through the tutorial.
I deployed .rmir to riva (I only have this model enabled), but the client cannot successfully call the ASR model on the server side, there will be the following error
![error](https://user-images.githubusercontent.com/56921951/188802743-ba9db1e3-d518-45df-8704-80b7954c571a.png)


**Steps/Code to reproduce bug**

- build environment
docker pull nvcr.io/nvidia/riva/riva-speech:2.4.0-servicemaker
sudo docker run --gpus all -it --rm -v /NeMo/nemo_experiments/QuartzNet15x5/rmir_fold:/servicemaker-dev -v /NeMo/nemo_experiments/QuartzNet15x5:/data --entrypoint="/bin/bash" nvcr.io/nvidia/riva/riva-speech:2.4.0-servicemaker

- convert to .riva file
nemo2riva --out Nemo_CTC.riva QuartzNet15x5.nemo

- convert to .rmir file
riva-build speech_recognition -f /servicemaker-dev/NeMo_CTC.rmir /data/2022-08-19_07-43-11/Nemo_CTC.riva --decoder_type=greedy

- start riva-speech
ngc registry resource download-version nvidia/riva/riva_quickstart:2.4.0
cd riva_quickstart_v2.4.0
bash riva_init.sh
bash riva_start.sh
**get a result like this =>**
**c1106038@DGX-Station-62:~/NeMo/riva_quickstart_v2.4.0$ sudo bash riva_start.sh
Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
2cfde38fd430b4ce346b125850848e187cc399422c59470f5eaef05bc00ea22b
Waiting for Riva server to load all models...retrying in 10 seconds
Riva server is ready...**

(In order to use a custom rmir file, only modify riva_model_loc and use_existing_rmirs=true in config.sh)
1. riva_model_loc="/home/c1106038/NeMo/nemo_experiments/QuartzNet15x5/rmir_fold"
2. use_existing_rmirs=true

- start client
bash riva_start_client.sh
(In order to use GPU modify the riva_start_client.sh as follows)
![image](https://user-images.githubusercontent.com/56921951/188805307-511da5b5-c8c0-43ba-9bc0-57c1eb5b977b.png)
**get a result like this =>**
![修改後](https://user-images.githubusercontent.com/56921951/188804990-d812dfd3-2287-4613-8c22-89b491bbc6b1.png)

- Execute riva_asr_client
riva_asr_client --audio_file=/test/morning_conference_000001.wav -language_code=zh-CN
**get a Error=>**
![error (1)](https://user-images.githubusercontent.com/56921951/188806188-a118efc9-5dad-4821-baee-943604bf21af.png)


**Expected behavior**

Custom model inference audio available on Riva.

**Environment overview (please complete the following information)**

- DGX V100 GPU
- Method of NeMo install:
docker pull nvcr.io/nvidia/nemo:22.05
docker run --gpus all -it --name Nemo_container -v NeMo:/workspace --shm-size=8g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/snd nvcr.io/nvidia/nemo:22.05

**Environment details**
OS version: Ubuntu 20.04
PyTorch version: 1.12.0a0+8a1a93a
Python version: 3.8.13
