{'fraction_non_alphanumeric': 0.02174758288187874, 'fraction_numerical': 0.004510225770496902, 'mean_word_length': 4.9436142350947465, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Dynamic Recognition of Speakers for Consent Management by Contrastive Embedding Replay

Voice assistants record sound and can overhear conversations. Thus, a consent management mechanism is desirable such that users can express their wish to be recorded or not. Consent management can be implemented using speaker recognition; users that do not give consent enrol their voice and all further recordings of these users is subsequently not processed. Building speaker recognition based consent management is challenging due to the dynamic nature of the problem, required scalability for large number of speakers, and need for fast speaker recognition with high accuracy. This paper describes a speaker recognition based consent management system addressing the aforementioned challenges. A fully supervised batch contrastive learning is applied to learn the underlying speaker equivariance inductive bias during the training on the set of speakers noting recording dissent. Speakers that do not provide consent are grouped in buckets which are trained continuously. The embeddings are contrastively learned for speakers in their buckets during training and act later as a replay buffer for classification. The buckets are progressively registered during training and a novel multi-strided random sampling of the contrastive embedding replay buffer is proposed. Buckets are contrastively trained for a few steps only in each iteration and replayed for classification progressively leading to fast convergence. An algorithm for fast and dynamic registration and removal of speakers in buckets is described. The evaluation results show that the proposed approach provides the desired fast and dynamic solution for consent management and outperforms existing approaches in terms of convergence speed and adaptive capabilities as well as verification performance during inference.


Index Terms
Consent management, voice assistant systems, bucket training, contrastive embedding replay, dynamic registrations.

I. INTRODUCTION
Many recent internet of things (IoT) applications such as smart homes, smart transport systems or smart healthcare rely on voice assistants as primary user interface. This is due to the fact that end-users prefer to communicate with IoT devices more naturally, using voice commands rather than classical interfaces such as a touch screen [1]. Consequently, consent management is now becoming a concern. For example, the recent European Union legislation, general data protection regulation (GDPR), requires all parties' consent for personal data collection. In the context of voice assistant systems, providing this feature is essential to protect users from being recorded without giving consent. If not giving consent, users should at least be able to communicate dissent such that their voice is not recorded. Implementing such a consent/dissent management system for voice assistants is challenging. The existence of voice assistant systems to nearby users may initially not be evident. Also, there is no obvious interface to articulate consent or dissent.
Recent initial ideas to implement consent management can be divided in two broad categories: 1) Consent management without voice assistant system support 2) Consent management with voice assistant system support The first category assumes that the operators of a voice assistant eco system do not support the implementation of consent management while the second approach assumes collaboration of a provider (e.g., Amazon in the case of the Echo voice assistant).
In the first category, Denial of Service approaches have been proposed; the voice assistant is prevented to collect voice samples by a non-consenting party. Specifically, an acoustic jamming device can be used to prevent all voice assistant systems in the vicinity of a user to record [2].
While such approach is possible, it is difficult to implement reliably in practical settings.
In the second category more options are available. One approach is to add information to the acoustic channel that can subsequently be detected by a back-end. A sound signal (i.e., a tag) is embedded in the audio stream (via a speaker) that can be used by the voice assistant's backend for consent management [3]. This approach faces challenges, in particular when consent of multiple users should be handled, requiring collision management of tag signals.
A second approach in this category is the use of speaker recognition for consent management.
Despite the rich literature for speaker recognition, the direct use of such approaches in the context of consent management is not practical. For the practical use of speaker recognition for consent management the following design requirements, must be met: 1) Fast training and dynamic inference 2) Modular design that can be distributed for different sets of speakers 3) Dynamic registration of new users that do not provide consent 4) Dynamic removal of users that now provide consent and previously did object (if required) 5) Front-end feature extraction and back-end classification should be possible In this paper we present a novel speaker recognition solution that addresses the aforementioned design requirements 1 to 4. In particular, the modular nature of the proposed method is beneficial for front-end only deployment by distributing the training to multiple agents, i.e., each agent is responsible for the consent management of a subset of users trained in the front-end. This way, samples from users who do not provide consent are not sent to the back-end, enhancing privacy preservation. It is worth mentioning that point 5 enables the consent management system to be separately deployed between front-end and back-end. In this case, it is possible to train the system with differential privacy in an end-to-end manner, and deploy the trained feature extraction and classification to the front-end and back-end, respectively [4], [5]. Moreover, it is of critical importance to remove private attributes (e.g., gender) from trained features during inference. Considering the aforementioned privacy preservations in the case of front-end feature extraction and back-end classification (i.e., point 5) are left for future research.

A. The Main Contributions
Existing classical methods for speaker recognition require training from scratch when a single speaker has to be added or removed from the pool of recognized speakers. Such training from scratch is not efficient and, thus, these existing methods are unsuitable for a consent management scenario where speakers dynamically join and leave. This work proposes a novel approach that enables such dynamic addition and removal of speakers using a modular design which ensures fast convergence and accurate speaker recognition.
The specific contributions are: 1) A training process based on the contrastive embeddings as a way to learn speaker equivariance inductive bias is proposed. The proposed approach is efficient in terms of convergence 4 speed and accurate prediction of speakers that do not provide consent. This is mainly due to learning the underlying speaker equivariance inductive biases using contrastive embeddings as feature extractions during training. In particular, a group of speakers form a bucket with the corresponding contrastive embeddings continuously used as a replay buffer for classification. Moreover, due to the modular nature of the proposed method it can be easily scaled as a distributed system.
2) A multi-strided random sampling of the contrastive embedding replay buffer is proposed.
The proposed sampling strategy starts with the large number of utterances from the initial buckets to fill up the memory size. Then, sparsely sampling the buckets of speakers to preserve enough memory for the buckets seen so far. This leads to memory efficiency, diversity in utterance selection, progressive difficulty during training, and avoiding the parameters' shifts to the buckets of speakers with more samples.
3) A dynamic algorithm for registering/removing speakers in different buckets is proposed.
The proposed approach registers new speakers according to L2 pairwise distance of new speakers from the prototypes of the previously trained contrastive embeddings of speakers per bucket. The proposed method takes into account the updated states of buckets after registering new speakers for selecting the optimal buckets using a dynamic programming approach. The dynamic programming solution with the linear maximum time complexity on the order of total number of new speaker registrations is designed such that each time new speakers are registered in different buckets and do not share the same bucket. This is due to the fact that updating the state of a bucket may affect the optimality of the bucket in terms of the Euclidean distance for the subsequent registrations in the same bucket in each iteration. Finally, it is observed that only using a portion of the utterances of old speakers leads to a great performance in terms of average overall accuracy. This is critical for preserving the privacy of the old speakers by removing the unnecessary utterances in the back-end.

II. DISTINCTIONS BETWEEN PROPOSED METHOD AND THE METHODS FROM LITERATURE
We briefly address some key differences between the proposed method and other methods from the literature including: fast learning (e.g., few-shot learning) solutions used for speaker recognition, continual learning, and contrastive learning in the context of speaker verification.
Few-Shot Learning: It is worth mentioning that the proposed approach deviates from fewshot learning methods in several aspects. First and foremost, the main goal of few-shot learning methods is to generalize on the classes, never seen during the training mode, with similar features using only a few examples/shots during the inference mode [6]- [8]. In other words, the samples with similar features to those during the training are classified using a few shots during the inference mode. Moreover, the meta learning methods fail if the distribution of the unseen speakers are different from those during the training that tends to happen when increasing the number of unseen test speakers [9]. However, in the context of consent management for voice assistant systems, it is not required to generalize to the voice samples of different individuals. This is due to the fact that the system requires to identify only the speakers that do not provide consent each time without generalizing to the speakers that are not among those for consent management. In other words, such a generalization actually hurts the consent management as a privacy measure. This is due to the fact that there is a chance for generalizing to speakers that are already giving consent according to the samples from the speakers that do not.
The other limiting factors for few-shot learning within the context of consent management are summarized as follows.
1) Limited capacity of feature extraction network to achieve meaningful prototypes to classify the queries according to the distance metric per episode. This leads to the requirement for an additional regularization term for the entire speakers during each episode that is considered to be a limiting factor in terms of scalability.
2) The size of the support set to extract such features as accurate as possible is often limited.
3) The convergence speed for few-shot learning methods during the training is generally slow.
Considering the aforementioned reasons, for consent management it is critical to identify speakers that do not provide consent as fast and accurate as possible. This is to avoid collecting privacy sensitive information while training. Moreover, it is of critical importance to propose efficient adaptive strategies for registering/removing speakers as fast and accurate as possible.
Continual Learning: The main goal of continual learning approaches is to train the network for different sets of tasks sequentially such that each time the network only has access to the data for the current task. In other words, the algorithm is not provided with the entire data for all the tasks at the same time. Current continual learning methods can be divided to three main categories: regularization based methods, parameter isolation approaches, and replay methods [10]. The regularization methods limit the ability to classify based on the tasks seen so far as 6 they preserve per-task prediction accuracy. Parameter isolation based approaches require known task IDs during the inference.
However, none of these conditions is necessarily the case for consent management applications.
In particular, it is necessary to classify the speakers that do not provide consent so far dynamically. Moreover, the task or bucket IDs during the inference mode is not necessarily known and needs to be predicted. Consequently, the only promising type of continual learning approaches that may be useful in the context of consent management is based on replay buffer methods.
The generative replay based solutions require a separate generative model to create a buffer for each class [11]. Despite a relatively good performance for simple classification tasks, applying such generative models that truly represent the underlying features of voice samples is a challenge. Another type of solutions within this category is based on task-specific weight realizations maintained in the memory [12]. In this approach the parameters of a target network are learned using a hyper-network for each specific task. The main challenges in this type of solutions are related to the task difficulty and relatively simple target network architectures required for classifications.
There are some key differences between the proposed method and replay based continual learning methods. First and foremost, in the context of continual learning it is assumed that the entire data for each task is provided sequentially and the network is fully trained for the current task using the replay buffer of the previous tasks to avoid catastrophic forgetting. This is not necessarily the case for the consent management systems as generally only a small portion of dataset for each bucket of speakers may be provided during each iteration and the system needs to efficiently use a small portion of dataset for each task during each iteration. In other words, rather than full training of the network for each sequential task the proposed method just trains the contrastive features of each bucket of speakers only a few steps into the training according to partially available data for each sequential task. Subsequently, for training the current bucket of speakers the contrastive features partially trained from the previous buckets are used as embedding replay buffer for training the classifier iteratively. This way, the classifier does not wait for completion of contrastive feature training of each bucket of speakers to obtain samples from embedding buffer for the current bucket. Consequently, the parameter update of the classifier is more flexible during each progressive registration circumventing the issues with decoupled representation and classification [13]. Furthermore, the classifier architecture is not modified after each progressive bucket/task registration and the final layer is set to the entire number of speakers for consent management. Finally, as the proposed sampling strategy aims to uniformly sample from the buckets seen so far according to the contrastive embedding buffer, the task recency problem due to the imbalance samples from the new and old classes is not an issue. Consequently, the sequential process continues until the entire buckets of speakers are covered for each iteration and converge to the optimal parameters for classification at the end of training iterations.
Moreover, in the context of consent management for voice assistant systems, it is required to identify speakers and the corresponding buckets that dynamically do not provide consent with a very high accuracy in the test time. In other words, any performance drop in terms of prediction accuracy on the previously learned tasks is not desirable as it is the case in most of replay based continual learning approaches specially for on-line class-incremental setting. In particular, unlike the classical on-line class incremental learning, it is of critical importance to predict not only the speakers during the test mode but also the bucket index. This is due to the fact that the process for registration of new speakers to the optimal old buckets or removing speakers from the buckets happens during the test/inference mode. So, one needs to predict the bucket index as well as the corresponding speakers during the test time.
Finally, storing the buffer in the input space, that is the case in the replay based methods, is often very costly and memory-intensive. Additionally, sampling the buffer in the input space requires contrastive training of the current batch together with the replayed buffer samples from the previous bucket resulting slow convergence during the training [14]. This is not desirable for fast recognition of speakers and slows down the training. To alleviate the aforementioned problems in classical replay based continual learning, the proposed method stores the buffer samples in the embedding space that has significantly lower dimensions.
Contrastive Learning: The concept of contrastive learning has gained popularity for speech verification applications [15]. In particular, a generalized end-to-end loss is proposed for speaker verification such that at each iteration the entire utterances for each speaker is used in the batch for the training. Additionally, it is assumed that the dataset contains the same number of utterances per speaker that is not necessarily the case in practice. Consequently, it is argued that using the entire utterances of all the speakers in the batch for the training requires less number of positive and negative tuples compared to the tuple based end-to-end approach. However, in the case of consent management to obtain efficient and dynamic contrastive training, it is impossible to use the entire utterances of the entire speakers in each batch. This is due to the large number of utterances per speaker and large total number of speakers resulting significantly large batch size that is costly and memory-intensive. Consequently, tuple based contrastive training is sub-optimal for dynamic and efficient consent management in the voice assistant systems.

III. THE METHOD
In this section, first the overall framework for the training of the speakers for consent management is explained in an algorithmic way. Then, a novel mechanism for dynamic registration of new speakers is proposed. Finally, a method for removing the speakers registered previously in the buckets is developed.

A. Training
The framework for the entire training process is described by an agent interacting with groups of speakers, i.e., buckets. Each agent is responsible for training a windowed stream of buckets in a modular manner. This way, it is possible to distribute the training process among different agents. Fig. 1 shows the overall pictorial viewpoint of the proposed training with contrastive embedding replay.
The proposed method starts the training with the selection of buckets of speakers that do not provide consent, i.e., a windowed bucket stream, for each agent. Subsequently, each bucket of speakers is fed to a supervised contrastive learning framework where feature extraction is achieved by running the supervised contrastive learning for each bucket individually only a few steps into the training 1 . The individual supervised contrastive loss L being the total number of buckets seen so far, is defined as follows.
where I b denotes all the speakers' utterances in the batch for the bucket b during training, and sup,s,u is defined as follows.
where A s,b (u) and P s,b (u) are defined as in which I s,b and P s,b denote all the utterances of other speakerss = s in the bucket b, and all the utterances of speaker s in the bucket b, respectively. The operation \{u} excludes the anchor utterance u from the corresponding set, and |P s,b (u)| denotes the corresponding cardinality of the set of utterances for speaker s in bucket b excluding anchor utterance u. The parameter τ is a positive scalar denoting the temperature. The embedding terms in (2) are obtained as where Emb θ e,b (.) and Proj θ proj,b (.) denote the embedding network and the projection head, respectively. The notation Enc θ b (.) is used for the encoder containing the embedding network followed by the projection head. The projection head is implemented using the attention pooling layer to obtain the embeddings for speaker s in bucket b, z s,b , with the parameter set θ b = 1 Note that contrastive feature extraction in each run of the training process requires only a few steps, e.g., epochs cont = 5. In other words, the proposed method does not wait for the full convergence of the supervised contrastive learning for each bucket during each run of the training process. Subsequently, the contrastive embedding buffer is sampled according to a progressive multistrided random sampling algorithm that will be explained next. Finally, the classifier is trained using the samples provided by the progressive multi-strided random embedding buffer sampling algorithm. In other words, the contrastive training provides an inductive bias for speaker classification during training as the main task. Presenting the contrastive inductive bias to the main classification task during training results efficient use of data and fast convergence as will be discussed in the simulations. Load a random shard of dataset for the bucket b with n utt utterances per speaker. 5 Train feature extraction encoder Enc θ b (.), initialized by the corresponding checkpoints if available or random initialization otherwise, for epochs cont contrastively. 6 Use Enc θ b (.) to encode the shard of dataset for the bucket b and store the result in the embedding buffer to replay for the subsequent bucket. 7 Use the proposed progressive multi-strided random buffer sampling Algorithm. 2 to obtain the contrastive embedding buffer D maxmem,buff and the corresponding labels y maxmem,buff . 8 Train the classifier Cls φ (.) using {D maxmem,buff , y maxmem,buff } for epochs cls . Stack the selected utterances and the corresponding labels.
14 end 15 Permute selected utterances and the corresponding labels for training. 16 end classifier are returned as the outputs of the algorithm.
The proposed method for progressive multi-strided random sampling of the contrastive embedding buffer is provided in the Algorithm. 2. For each bucket, a given number of utterances 13 are selected randomly with a given stride 2 . The selected utterances and corresponding labels are appended to their corresponding buffers, this is shown in steps 3 & 4. The number of selected utterances is compared to the maximum allowed memory size in step 5. If the selected utterances of speakers is less than the maximum allowed memory, then the permuted utterances and the corresponding labels are returned in step 15. Otherwise, the utterances per speaker for each bucket registered so far are incrementally added until the maximum allowed memory is filled, this is shown in steps 5-14. The entire process 3-15 is repeated until the number of buckets seen so far B have been processed. Finally, the contrastive embedding buffer and the corresponding labels with the max mem size for the buckets seen so far are returned as the outputs of the algorithm.

B. Dynamic Management of Speakers' Consent
The process for adaptive registering of new speaker(s) to the previously registered bucket(s) of speaker(s) and/or unregistering speaker(s) from the bucket(s) is provided in the Algorithm. 3.
For adaptive registration of new speakers, first the prototypes for speakers previously registered in each bucket is computed in the inference mode in step 5 as follows.
where c s,b denotes the prototype of speaker s in bucket b, P s,b denotes the set of held-out utterances during the inference for the speaker s in bucket b with the cardinality of | P s,b |, and z (i) s,b is the corresponding embedding following the same definition as in (5). Subsequently, the L2 pairwise distance of the encoded features of n reg -th new speaker in the bucket b, z sn reg ,b , from the prototypes c s,b is calculated for the held-out utterances of the new speaker 3 for ∀n reg , ∀b, and ∀s in step 6 as follows.
2 It is assumed that the contrastive embedding buffer is organized such that the utterances of the same speaker are grouped together, that is a common assumption in the context of contrastive learning where samples of the same category (in this case speaker) are grouped together for the training purposes, and each speaker loads nutt utterances per iteration during the training using the custom data loader. Also, without loss of generality and for simplifying the analysis it is assumed that each bucket contains the same number of speakers, i.e., NB = N/B when N is divisible by B.  Compute (7) for n reg -th new speaker using the updated Enc θ b (.). Form b * reg according to steps 3-9, and obtain the longest unique optimal buckets l * and corresponding speakers r reg based on the Algorithm. 4.

12
Update Enc θ b * (.) for l * , and obtain the updated embedding. 13 Update parameters of Cls φ (.) using the updated embeddings from the previous step.
14 Update r reg obtained in step 11 to exclude registered speakers for the next round. Consequently, the corresponding label of the bucket including the speaker's prototype with the shortest L2 pairwise distance from the new registered speaker is returned for ∀n reg in step where b * nreg represents the optimal bucket index for the registration of n reg -th new speaker. It is worth noting that the main reason for registering the new speakers in the buckets with the shortest L2 pairwise distance is to efficiently allocate the Euclidean space for each bucket after new registrations. In other words, registering new speakers into the buckets with shortest L2 pairwise distance requires less Euclidean space. Consequently, it is possible to register more new speakers in the disjoint updated feature space of the buckets. This property is essential for bucket prediction that requires disjoint buckets in the feature space. According to the above explanations, the buckets with the shortest L2 pairwise distance from the new speakers are referred to as optimal buckets in this paper. It is worth mentioning that the maximum number of in which P sn reg ,b denotes the set of held-out utterances during the inference for the new speaker s nreg . Consequently, the index of optimal buckets for all new speaker registrations obtained according to (6)-(8) forms the following vector.
As the number of buckets is usually smaller than the number of new speaker registrations, there are at least two new speakers registered in the same bucket during each iteration. However, the registration of a new speaker in a bucket changes the contrastive feature state of that bucket such that it may no longer be the optimal bucket for registering the subsequent new speaker.
Moreover, new speakers in the subsequent round may select different optimal buckets, according to the shortest L2 pairwise distance, after registrations of new speakers in the current round.
Consequently, a dynamic programming approach of decision type is designed in Algorithm.
4 to make sure in each round a subset of new speakers are registered in the sequence of longest optimal unique buckets l * and do not share the same bucket, i.e., step 11. Next, the new speakers are registered in the buckets with unique optimal index in each iteration in step 12, and the corresponding parameters of contrastive embedding encoder are updated for these buckets. Finally, the parameters of classifier are updated in step 13. Once the registration process is completed for each round the corresponding index(es) of new speaker(s) registered in the optimal bucket(s) is/are removed from the list in (10) for the next iteration. This is shown in step 3 by excluding the list of registered new speakers r reg from the original list of new speakers n reg for the next iteration. The entire process is repeated until all the new speakers are registered, i.e., r reg = n reg or equivalently n reg \{r reg } = b * reg = ∅. The dynamic programming Algorithm. 4 obtains the longest unique sequence of the optimal buckets for registering new speakers in each iteration of the Algorithm. 3. It receives the full list of optimal buckets in (10) as the input for the entire list of new speakers with N reg elements in the first iteration of Algorithm. 3. Then, the longest unique sequence of the optimal buckets is achieved according to the recursive call as follows.
In (11), the optimal bucket at index i, b * i , is added only if it does not already exist in the set of new speakers in the interval [0, i). By increasing the index i, the entire list of optimal buckets is covered starting from the base case in the bottom-up way. It is worth mentioning that the proposed dynamic programming Algorithm. 4 to obtain the longest unique sequence of optimal buckets per registration only requires the linear time complexity of O(N reg ) for the worst case, i.e., the first round of iteration in Algorithm. 3 in which the registration set r reg is still empty. This is due to the fact that the sequence of optimal buckets b * reg is progressively reduced after each registration, i.e., step 3 in Algorithm. 3.
For unregistering the speaker(s) from the bucket(s) adaptively, first select the bucket(s) to unregister speaker(s) from. Then, repeat the steps similar to 12 and 13 in the Algorithm. 3 for the reduced set, i.e., excluding the unregistered speaker(s), of speakers in the bucket until convergence. It is worth mentioning that in this paper the time interval in which each bucket of speakers are providing to the back-end for consent management is not considered for simplicity of the analysis. This 4 can be addressed by additionally using a time stamp to label each bucket.
Consequently, the speakers that do not provide consent for a given time interval are grouped in the buckets with the corresponding time stamps stored as a decision tree in the back-end. In other words the problem boils down to a decomposable search algorithm that is known to be a fully retro-active data structure via decision trees [16]. This is due to the fact that registration

IV. EXPERIMENTS
The goal of the simulations is to answer the following questions: 1) Can the proposed method enable a fast training of dynamically registered speakers?
2) Can the proposed method adaptively register new speakers efficiently?
3) Can the proposed method adaptively remove the speakers efficiently? 4) Can the proposed method predict the bucket and corresponding speakers adaptively? 5) Can the proposed method provide a good verification performance during the testing phase?
All the experiments were run on a single NVIDIA GeForce RTX 2070 GPU. The code for the simulations will be made available.

A. Dataset
The LibriSpeech dataset is used for all the results [17]. Different subsets of the aforementioned dataset is used for training and validation/testing. In particular, 40 speakers are randomly selected from the set of speakers with lower word error rate denoted by "clean" in the LibriSpeech dataset

C. Baseline
We compare the proposed method with the baseline for consent management in terms of: accuracy, total elapsed time for convergence in the training phase, and verification performance in the testing phase. As the baseline without the contrastive feature extraction performs poorly in terms of accuracy and total elapsed time, the adapted baseline (i.e., adapted from scratch) is used to include contrastive feature extraction of all speakers at once.  In other words, all the speakers in the buckets are predicted with almost 100% accuracy after around 40 epochs, while the adapted from scratch training requires at least 500 epochs to achieve the same performance. The total elapsed time for training using the proposed method is around 4 minutes, while the adapted from scratch requires at least 25 minutes to converge. This shows the importance of bucketing the speakers during the training and progressive bucket registration to achieve a much faster convergence.

D. Results
The performance of the same network architecture as a result of classical training of all the N = 40 speakers from scratch without contrastive feature extraction strategy is shown with the black dashed lines. It is observed that even after 1000 epochs the validation accuracy is on the order of 60% and the validation loss is around 1.5. In other words, even after around 30 minutes the accuracy is on the order of 60%. This shows that the contrastive feature extraction significantly helps the classification and results fast convergence with a high accuracy compared to training from scratch without the contrastive features.  This is to avoid keeping the utterances of old speakers in the back-end unnecessarily. It is worth mentioning that using less than 50% of the old utterances degrades the overall validation accuracy.
In other words, there is a compromise between the number of old utterances used during the new registrations (i.e., preserving the privacy of old users) and the overall accuracy. Consequently,   registrations.  Table. I. This is due to the fact that the subsequent round is delayed by the maximum elapsed time of registering a speaker in an agent in the current round. Moreover, due to the non-contiguous speaker registration in each round, it is possible to register noncontiguous speakers in parallel. As a result the delay in registering non-contiguous speakers in each round is due to the maximum elapsed time for registering a given speaker in the previous  Finally, it is observed that fully modular training of randomly initialized classifier on the fully trained contrastive embeddings after the dynamic registrations of all the new speakers in the optimal buckets requires more iterations. In particular, the best values for average validation accuracy are achieved after around 11 minutes only for classification of fully trained contrastive embeddings for Agent 3. This is mainly due to the sub-optimal initialization of the classifier, compared to the proposed method with adaptive initializations in each round. It is worth mentioning that fully modular training using a single linear classification layer by absorbing the non-linearity as a part of the cross-entropy loss, results a similar performance in terms of validation accuracy for fully modular training. In other words, the validation accuracy is close to the optimal values obtained by linking the linear classification layer to a kernel machine with finite dimensional reproducing kernel Hilbert space for fully modular training [20].
In conclusion, dynamic registrations in different rounds are quickly keeping up with the new registrations for different agents. This is due to the updated parameters of contrastive encoder in the buckets for each agent according to the new registrations. In other words, adaptive feature representations as a result of dynamic updates in each round of registrations leads to a faster convergence with high accuracy for different rounds. This is extremely useful for the fast registration of the new speakers using the old features without training on the entire dataset, including the new speakers, from scratch that requires much longer time as reported above.  is a desirable behavior for bucket inference. For the case that different number of speakers are unregistered from the bucket, the performance remains as good as the case that all the speakers in the bucket are available as shown in second and third rows in Fig. 10. In other words, the bucket inference and consequently speaker prediction are not affected by the number of speakers in the bucket during the testing phase. This leads to the conclusion that bucket inference for variable number of speakers in the buckets is possible using the proposed approach due to Consequently, the proposed approach can efficiently predict the speakers and the corresponding buckets during the test mode for different number of speakers per bucket with a very high accuracy. Table. II reports the verification performance in terms of maximum equal error rate (EER) (%) for all the different trials during the testing phase for different agents. This is denoted by the column "Proposed" representing the verification performance of the proposed method for the observed buckets with totally different held-out utterances during the test time. Also, the verification performance of the corresponding adapted from scratch method, denoted by the column "Adapted", is reported for comparison purposes. To analyze the verification performance of the proposed method in the testing phase, the entire test samples are used for each agent.
It is observed that the proposed classifier trained dynamically, using progressive samples from all the buckets, performs better than the adapted approach during the test time for almost all the agents. According to the results, the proposed trained classifier outperforms the adapted from scratch classifier for all the agents. In particular, for more challenging scenarios in which the adapted approach provides higher EER, the proposed method provides significantly smaller values. The main reason is due to the dynamic training of the proposed classifier after each bucket registration. This provides the classifier with an easier task by registering less number of speakers each time during the training. In other words, the classifier is trained to discriminate optimally between buckets of speakers progressively each time during the iterations. This leads to faster convergence and more generalization capabilities during the inference in terms of verification for the proposed method. On the contrary, the adapted from scratch training requires much more iterations to discriminate between the entire speakers at once during the training. This leads to less generalization capabilities during the test time, more vulnerabilities to over-fitting, and hence worse performance compared to the proposed method in terms of EER in the testing mode.