On Training Targets and Activation Functions for Deep Representation
  Learning in Text-Dependent Speaker Verification
  Deep representation learning has gained significant momentum in advancing
text-dependent speaker verification (TD-SV) systems. When designing deep neural
networks (DNN) for extracting bottleneck features, key considerations include
training targets, activation functions, and loss functions. In this paper, we
systematically study the impact of these choices on the performance of TD-SV.
For training targets, we consider speaker identity, time-contrastive learning
(TCL) and auto-regressive prediction coding with the first being supervised and
the last two being self-supervised. Furthermore, we study a range of loss
functions when speaker identity is used as the training target. With regard to
activation functions, we study the widely used sigmoid function, rectified
linear unit (ReLU), and Gaussian error linear unit (GELU). We experimentally
show that GELU is able to reduce the error rates of TD-SV significantly
compared to sigmoid, irrespective of training target. Among the three training
targets, TCL performs the best. Among the various loss functions, cross
entropy, joint-softmax and focal loss functions outperform the others. Finally,
score-level fusion of different systems is also able to reduce the error rates.
Experiments are conducted on the RedDots 2016 challenge database for TD-SV
using short utterances. For the speaker classifications, the well-known
Gaussian mixture model-universal background model (GMM-UBM) and i-vector
techniques are used.
