Speaker diarization example produces negative offsets
**Describe the bug**

Using [speaker_diarize.py](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_recognition/speaker_diarize.py ) results in a negative offset for a file in a manifest, which leads to an error in dataloader:
```

  File "speaker_diarize.py", line 45, in main
    sd_model.diarize()
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 341, in diarize
    self._run_vad(manifest_vad_input)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 193, in _run_vad
    for i, test_batch in enumerate(tqdm(self._vad_model.test_dataloader())):
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_label.py", line 476, in vad_frame_seq_collate_fn
    return _vad_frame_seq_collate_fn(self, batch)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_label.py", line 245, in _vad_frame_seq_collate_fn
    audio_signal = torch.stack(audio_signal)
RuntimeError: stack expects a non-empty TensorList
```

Resulting manifest: 
```
{"audio_filepath": "test.wav", "duration": 2.2756875, "label": "infer", "text": "_", "offset": -0.15}
```
**Steps/Code to reproduce bug**

Command:
python speaker_diarize.py   diarizer.paths2audio_files=files.txt   diarizer.vad.model_path="vad_telephony_marblenet"   diarizer.vad.threshold=0.7   diarizer.speaker_embeddings.model_path="speakerverification_speakernet"   diarizer.out_dir="demo"

Audio file with manifest:
[reproduce.zip](https://github.com/NVIDIA/NeMo/files/6799603/reproduce.zip)


**Expected behavior**
No negative offsets in manifests.


**Environment overview (please complete the following information)**

Nemo version 1.1.0
Pytorch version 1.9.0
