Dynamic Recognition of Speakers for Consent Management by Contrastive
  Embedding Replay
  Voice assistants record sound and can overhear conversations. Thus, a consent
management mechanism is desirable such that users can express their wish to be
recorded or not. Consent management can be implemented using speaker
recognition; users that do not give consent enrol their voice and all further
recordings of these users is subsequently not processed. Building speaker
recognition based consent management is challenging due to the dynamic nature
of the problem, required scalability for large number of speakers, and need for
fast speaker recognition with high accuracy. This paper describes a speaker
recognition based consent management system addressing the aforementioned
challenges. A fully supervised batch contrastive learning is applied to learn
the underlying speaker equivariance inductive bias during the training on the
set of speakers noting recording dissent. Speakers that do not provide consent
are grouped in buckets which are trained continuously. The embeddings are
contrastively learned for speakers in their buckets during training and act
later as a replay buffer for classification. The buckets are progressively
registered during training and a novel multi-strided random sampling of the
contrastive embedding replay buffer is proposed. Buckets are contrastively
trained for a few steps only in each iteration and replayed for classification
progressively leading to fast convergence. An algorithm for fast and dynamic
registration and removal of speakers in buckets is described. The evaluation
results show that the proposed approach provides the desired fast and dynamic
solution for consent management and outperforms existing approaches in terms of
convergence speed and adaptive capabilities as well as verification performance
during inference.
