Expression detection in Speaker Diarization 
While working with speaker diarization task I found that if input audio have expressions (like anger, shouting, whispering, exciting etc.) then model is recognise as another speaker.

So, if these expressions also recognise than it will be good.