"all_probs" is not functional in text-generation-server
**Describe the bug**

Use the following script to make a request will result in crashing in NeMo text-generation-server:

```
import json
import requests

port_num = 5555
headers = {"Content-Type": "application/json"}

def request_data(data):
    resp = requests.put('http://localhost:{}/generate'.format(port_num),
                        data=json.dumps(data),
                        headers=headers)
    resp = resp.json()
    if "sentences" not in resp:
        return f"Error: {resp}"
    return resp['sentences']

data = {
    "sentences": ["Tell me an interesting fact about space travel."],
    "tokens_to_generate": 32,
    "temperature": 1.00,
    "add_BOS": True,
    "top_k": 0,
    "top_p": 0.7,
    "greedy": False,
    "all_probs": True, # <-- problematic
    "repetition_penalty": 1.0,
    "min_tokens_to_generate": 0,
    "end_strings": [], # stop sequence
}

for sentence in request_data(data):
    print(sentence)

```

**Steps/Code to reproduce bug**

1. Launch a service (see environment below for the command).
2. Run the above script to make a request.

**Expected behavior**

The proper response as shown in the NeMo API document.

**Environment overview (please complete the following information)**

1. Download a checkpoint from HuggingFace and put it under `/data`.
2. Launch a docker

```
MODEL=/data/nemo-megatron-gpt-1.3B/nemo_gpt1.3B_fp16.nemo
TP_SIZE=1
VOLUME=/data

docker run --rm -it --gpus all --shm-size=20g --ipc=host \
    --ulimit memlock=-1 --ulimit stack=67108864 --network=host \
    -v $VOLUME:/data \
    nvcr.io/ea-bignlp/nemofw-training:23.04.1-py3:latest \
    python3 /opt/NeMo/examples/nlp/language_modeling/megatron_gpt_eval.py \
    gpt_model_file=$MODEL server=True \
    tensor_model_parallel_size=$TP_SIZE \
    trainer.devices=$TP_SIZE \
    port=5555
```

**Additional context**

The server side crashes at the line https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/text_generation_utils.py#L430:

```
  File "/opt/NeMo/nemo/collections/nlp/modules/common/text_generation_utils.py", line 429, in synced_generate
    torch.distributed.broadcast(full_logits, src, group)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 1422, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 1526, in broadcast
    _check_single_tensor(tensor, "tensor")
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 568, in _check_single_tensor
    raise RuntimeError(
RuntimeError: Invalid function argument. Expected parameter `tensor` to be of type torch.Tensor.
```

This line is weird because it attempts to broadcast `full_logits`, but this variable seems not properly defined:

```
    for tokens, lengths, output_logits, full_logits in batch_token_iterator:
        context_length += 1
```

cc @yidong72 @michalivne @ericharper 