[DIARIZATION] is clustering needed when number of speakers is known?
Hello,

I'm using the diarization model and already know the number of speakers. I thought that the clustering algorithm was only needed to determine the number of speakers, but it seems that it is still being run if when I use the following settings:

```
meta = {
    'audio_filepath': an4_audio,
    'offset': 0,
    'duration': None,
    'label': 'infer',
    'text': '-',
    'num_speakers': 2,
    'rttm_filepath': None,
    'uem_filepath': None
}
```
and

`config.diarizer.clustering.parameters.oracle_num_speakers = True`

Is that the expected behavior? My main issue is that while the other parts of the model can be run on the GPU, the  clustering is still run on the CPU so the overall inference is quite long.