Wrong Confidence for Speaker Identification
We made a finetune for the speaker identification for the Portuguese language, the best result we got was using these parameters:
Epochs 2
Batch size 32
Optimizer 'SGD'
Lr 0.011823477534453206
Scale 60
Margin 0.07095099595312822

But when testing with our test dataset, we found that some audios that are clearly different are giving a very high confidence rate. Here are some examples:
85% confidence rate
[85.zip](https://github.com/NVIDIA/NeMo/files/11545713/85.zip)

86% confidence rate
[86.zip](https://github.com/NVIDIA/NeMo/files/11545714/86.zip)

90% confidence rate
[90.zip](https://github.com/NVIDIA/NeMo/files/11545708/90.zip)

92% confidence rate
[92.zip](https://github.com/NVIDIA/NeMo/files/11545709/92.zip)

94% confidence rate
[94.zip](https://github.com/NVIDIA/NeMo/files/11545710/94.zip)

In another scenario, we validate whether they are in fact the same people. And in some cases it affected a low confidence rate, here are some examples:
59% confidence rate 
[59.zip](https://github.com/NVIDIA/NeMo/files/11545711/59.zip)

64% confidence rate 
[64.zip](https://github.com/NVIDIA/NeMo/files/11545712/64.zip)


Below is our model used for both scenarios:
["model"](https://drive.google.com/file/d/1CHyLBXmd_NoC3mg8Q8d_0gIsZdRWw8nd/view?usp=share_link)

Do these problems above make sense? Are there any parameter tweaks, or tweaks to the audios that can be done to fix this issue? 
Thanks in advance