{'fraction_non_alphanumeric': 0.0360485589843388, 'fraction_numerical': 0.01005467519228987, 'mean_word_length': 4.92613948380011, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Speaker Diarization as a Fully Online Learning Problem in MiniVox

We proposed a novel machine learning framework to conduct real-time multi-speaker diarization and recognition without prior registration and pretraining in a fully online learning setting. Our contributions are two-fold. First, we proposed a new benchmark to evaluate the rarely studied fully online speaker diarization problem. We built upon existing datasets of real world utterances to automatically curate MiniVox, an experimental environment which generates infinite configurations of continuous multi-speaker speech stream. Second, we considered the practical problem of online learning with episodically revealed rewards and introduced a solution based on semi-supervised and self-supervised learning methods. Additionally, we provided a workable web-based recognition system which interactively handles the cold start problem of new user's addition by transferring representations of old arms to new ones with an extendable contextual bandit. We demonstrated that our proposed method obtained robust performance in the online MiniVox framework.


INTRODUCTION
Speaker recognition involves two essential steps: registration and identification [2]. In laboratory setting, the stateof-the-art approaches usually emphasize the registration step with deep networks [3] trained on large-scale speaker profile dataset [4]. However, in real life, requiring all users to complete voiceprint registration before a multi-speaker teleconference is hardly a preferable way of system deployment. Dealing with this challenge, speaker diarization is the task to partition an audio stream into homogeneous segments according to the speaker identity [5]. Recent advancements have enabled (1) contrastive audio embedding extractions such as Mel Frequency Cepstral Coefficients (MFCC) [6], i-vectors [7] and d-vectors [8]; (2) effective clustering modules such as Gaussian mixture models (GMM) [9], mean shift [10], Kmeans and spectral clustering [8] and supervised Bayesian non-parametric methods [11,12]; and (3) reasonable resegmentation modules such as Viterbi and factor analysis subspace [13]. In this work, we proposed a new paradigm to consider the speaker diarization as a fully online learning problem of the speaker recognition task: it combines the embedding extraction, clustering and resegmentation into the same problem as an online decision making problem.
Why is this online learning problem different? The state-of-the-art speaker diarization systems usually require large datasets to train their audio extraction embeddings and clustering modules, especially the ones with deep neural networks and Bayesian nonparametric models. In many real-world applications in developing countries, however, the training set can be limited and hard to collect. Since these modules are pretrained, applying them to out-of-distribution environments can be problematic. For instance, an intelligent system trained with American elder speaker data might find it hard to generalize to a Japanese children diarization task because both the acoustic and contrastive features are different. To tackle this problem, we want the system to learn continually. To push this problem to the extreme, we are interested in a fully online learning setting, where not only the examples are available one by one, the agent receives no pretraining from any training set before deployment, and learns to detect speaker identity on the fly through reward feedbacks. To the best of our knowledge, this work is the first to consider diarization as a fully online learning problem. Through this work, we aim to understand the extent to which diarization can be solved as merely an online learning problem and whether traditional online learning algorithms (e.g. contextual bandits [14,15,16]) can be a practical solution.
What is a preferable online speaker diarization system? A preferable AI engine for such a realistic speaker recognition and diarization system should (1) not require user registrations before its deployment, (2) allow new user to be registered into the system real-time, (3) transfer voiceprint information from old users to new ones, (4) be up running without pretraining on large amount of data in advance. While attractive, assumption (4) introduced an additional caveat that the labeling of the user profiles happens purely on the fly, trading off models pretrained on big data with the user directly in- teracting with the system by correcting the agent as labels. To tackle these challenges, we formulated this problem into an interactive learning model with cold-start arms and episodically revealed rewards (users can either reveal no feedback, approving by not intervening, or correcting the agent).
Why do we need a new benchmark? Traditional dataset in the speaker diarization task are limited: CALL-HOME American English [17] and NIST RT-03 English CTS [18] contained limited number of utterances recorded under controlled conditions. For online learning experiments, a learn-from-scratch agent usually needs a large length of data stream to reach a comparable result. Large scale speaker recognition dataset like VoxCeleb [4,19] and Speakers in the Wild (SITW) [20] contained thousands of speaker utterances recorded in various challenging multi-speaker acoustic environments, but they are usually only used to pretrain diarization embeddings. In this work, we proposed a new benchmark called MiniVox, which can transform any large scale speaker identification dataset into infinitely long online audio streams with various configurations.
To the best of our knowledge, this is the first approach to apply the Bandit problem to the speaker diarization task. We built upon the Linear Upper Confidence Bound algorithm (LinUCB) [21] and proposed a semi-supervised learning variant to account for the fact that the rewards are entirely missing in many episodes. For each episode without feedbacks, we applied a self-supervision process to assign a pseudo-action upon which the reward mapping is updated. Finally, we generated new arms by transferring learned arm parameters to similar profiles given user feedbacks.

BACKGROUND: THE BANDIT PROBLEM
In the online learning setting, the data become available in a sequential order and later used to update the best predictor for future data or reward associated with the data features. In many case, the reward feedback is the only source where the online learning agent can effectively learn from the sequential past experience. This problem is especially important in the field of sequential decision making where the agent must choose the best possible action to perform at each step to maximize the cumulative reward over time. One key challenge is to obtain an optimal trade-off between the exploration of new actions and the exploitation of the possible reward mapping from known actions. This framework is usually formulated Algorithm 1 Online Learning with Episodic Rewards 1: for t = 1,2,3,· · · , T do 2: (x(t), r(t)) is drawn according to Px,r 3: Context x(t) is revealed to the player 4: Player chooses an action at = πt(x(t)) 5: Feedback ra t ,t(t) for the arm at is episodically revealed 6: Player updates its policy πt 7: end for as the Bandit problem where each arm of the bandit corresponds to an unknown (but usually fixed) reward probability distribution [22], and the agent selects an arm to play at each round, receives a reward feedback and updates accordingly. An especially useful variant of Bandit is the Contextual Bandit, where at each step, the agent observes an N -dimensional context, or feature vector before selecting an action. Theoretically, the ultimate goal of the Contextual Bandit is to learn the relationship between the rewards and the context vectors so as to make better decisions given the context [23].

THE FULLY ONLINE LEARNING PROBLEM
Algorithm 1 presents at a high-level our problem setting of our interactive learning system for speaker diarization, where r a,t (t) ∈ [0, 1] is the reward of action a at time t, and r(t) ∈ [0, 1] N denotes a vector of rewards for all arms at time t. P x,r denotes a joint probability distribution over (x, r), and π : C → A denotes a policy. Unlike traditional setting, in step 5 we have the rewards revealed in an episodic fashion (i.e. sometimes there are feedbacks of rewards being 0 or 1, sometimes there are no feedbacks of any kind). We consider our setting online semi-supervised learning [24], where agents continually learn from both labeled and unlabeled data.

Contextual Bandits with Extendable Arms
In an ideal online learning scenario without oracle, we start with a single arm, and when new labels arrive new arms are then generated accordingly. This problem is loosely modelled by the bandits with infinitely many arms [25]. For our specific application of speaker registration process, we applied the arm expansion process outlined in Figure 1: starting from a single arm (for the "new" action), if a feedback confirms a new addition, a new arm is initialized and stored (more details on the handling of growing arms can be found in section 4.4).

Episodically Rewarded LinUCB
We proposed Background Episodically Rewarded LinUCB (BerlinUCB [26]), a semi-supervised and self-supervised online contextual bandit which updates the context representations and reward mapping separately given the state of the feedbacks being present or missing. As in Algorithm 2, the steps 1 through 12 of BerlinUCB are the same as the standard LinUCB algorithm [21], and in case of a missing reward, we introduced the steps 13 through 20 as the alternative strategy. We assume that: (1) when there are feedbacks available, the feedbacks are genuine, assigned by the oracle, and (2) when the feedbacks are missing (not revealed by the background), it is either due to the fact that the action is preferred (no intervention required by the oracle, i.e. with an implied default rewards), or that the oracle didn't have a chance to respond or intervene (i.e. with unknown rewards). Especially in the Step 15, when there is no feedbacks, we assign the context x t to a class a (an action arm) with the self-supervision given the previous labelled context history (section 4.3). Since we don't have the actual label for this context, we only update the reward mapping parameter b a and leave the covariance matrix A a untouched. This additional usage of unlabelled data (or unrevealed feedback) is especially important in our model.

Self-Supervision and Semi-Supervision Modules
We construct our self-supervision modules given the cluster assumption of the semi-supervision problem: the points within the same cluster are more likely to share a label. As shown in many work in modern speaker diarization, clustering algorithms like GMM [9] and spectral clustering [8] are especially powerful unsupervised modules, especially in their offline versions. Their online variants, however, often perform poorly [12]. Nonetheless, we chose three popular clustering algorithms as the self-supervision: GMM, Kmeans and K-nearest neighbors (KNN), all in their online versions.

Complete Engine for Online Speaker Diarization
To adapt our BerlinUCB algorithm to the specific application of speaker recognition, we firdst define our actions. There are three major classes of actions: an arm "New" to denote that a new speaker is detected, an arm "No Speaker" to denote that no one is speaking, and N different arms "User n" to denote that user n is speaking. Table 1 presents the reward assignment given four types of feedbacks. Note that we if the background revealed the feedbacks then 10: Observe feedback ra t ,t

13:
elif the background revealed NO feedbacks then 14: if use self-supervision feedback 15: r = [at == predict(xt)] % clustering modules 16: ba t ← ba t + r xt 17: elif % ignore self-supervision signals 18: end if 20: end if 21: end for assume that when the agent correctly identifies the speaker (or no speaker), the user (as the feedback dispenser) should send no feedbacks to the system by doing nothing. In another word, in an ideal scenario when the agent does a perfect job by correctly identifying the speaker all the time, we are not necessary to be around to correct it anymore (i.e. truly feedback free). As we pointed out earlier, this could be a challenge earlier on, because other than implicitly approving the agent's choice, receiving no feedbacks could also mean the feedbacks are not revealed properly (e.g. the human oracle took a break). Furthermore, we noted that when "No Speaker" and "User n" arms are correctly identified, there is no feedback from us the human oracle (meaning that these arms would never have learned from a single positive reward if we don't use the "None" feedback iterations at all!). The semi-supervision by self-supervision step is exactly tailored for a scenario like this, where the lack of revealed positive re-

Feedback types (+, +) (+, -) (-, +)
None New r = 1 r = 0 -Alg. 2 Step 13 No Speaker -r = 0 r = 0 User n -r = 0 r = 0 Table 1: Possible algorithm routes given no feedbacks, or a feedback telling the agent that the correct label is a * . (+, +) means that the agent guessed it right by choosing the right arm; (+, -) means that the agent chose this arm incorrectly, since the correct one is another arm; (-, +) means that the agent didn't choose this arm, while it turned out to be the correct one. "-" marks scenarios not applicable.
ward for "No Speaker" and "User n" arms is compensated by the additional training of the reward mapping b at if context x t is assigned to the right arm.
To tackle the cold start problem, the agent grows it arms in the following fashion: the agent starts with two arms, "No Speaker" and "New"; if it is actually a new speaker speaking, we have the following three conditions: (1) if "New" is chosen, the user approves this arm by giving it a positive reward (i.e. clicking on it) and the agent initializes a new arm called "User N " and update N = N + 1 (where N is the number of registered speakers at the moment); (2) if "No Speaker" is chosen, the user disapproves this arm by giving it a zero reward and clicking on the "New" instead), while the agent initializes a new arm; (3) if one of the user arms is chosen (e.g. "User 5" is chosen while in fact a new person is speaking), the agent copies the wrong user arm's parameters to initialize the new arm, since the voiceprint of the mistaken one might be beneficial to initialize the new user profile. In this way, we can transfer what has been learned for a similar context representations to the new arm. 2

BENCHMARK DESCRIPTION: MINIVOX
MiniVox is an automatic framework to transform any speakerlabelled dataset into continuous speech datastream with episodically revealed label feedbacks. Since our online learning problem setting assumes learning the voiceprints without any previous training data at all, MiniVox's flexibility in length and configuration is especially important. As outlined in Figure 2, MiniVox has a straightforward data stream generation pipeline: given a pool of single-speaker-annotated utterances, randomly concatenate multiple pieces with a chosen number of speakers and a desired length. The reward stream is then sparsified with a parameter p as the percentage of time a feedback is revealed.
There are two scenarios that we can evaluate in MiniVox: if we assume there is an oracle, the online learning model is given the fixed number of the speakers in the stream; if we assume there is no oracle, the online learning model will start from zero speaker and then gradually discover and register new speakers for future identification and diarization.

Experimental Setup and Metrics
We applied MiniVox on VoxCeleb [4] to generate three data streams with 5, 10 and 20 speakers to simulate real-world conversations. We extracted two types of features (more details in section 6.2) and evaluated it in two scenarios (with or without oracle). The reward streams are sparsified given a revealing probability of 0.5, 0.1 and 0.01. In summary, we evaluated our models in a combinatorial total of 3 speaker numbers × 3 reward revealing probabilities × 2 feature types × 2 test scenarios = 36 online learning environments. The online learning timescale range from ∼12000 to ∼60000 timeframes, with a frame shift of 10 ms. For notation of a specific MiniVox, in this paper we would denote "MiniVox C5-MFCC-60k" as a MiniVox environment with 5 speakers ranging 60k time frames using MFCC as features.
To evaluate performance in the above MiniVox environments, we reported Diarization Error Rates (DER), the standard introduced in the NIST Rich Transcription 2009 (RT-09). In addition, as a common metric in online learning literature, we also recorded the cumulative reward: at each frame, if the agent correctly predicts a given speaker, the reward is counted as +1 (regardless of whether the agent observes it or not).
We compared 5 agents: The baseline, LinUCB is the contextual bandit with extendable arms proposed in section 4.1.
BerlinUCB is our standard contextual bandit model designed for sparse feedbacks without the self-supervision modules. To test the effect of self-supervision, we introduced three clustering modules in BerlinUCB (Alg 2, Step 15) denoted B-Kmeans, B-KNN (with K=5), and B-GMM, whose clustering modules are randomly initialized and updated online.

Feature Embeddings: MFCC and Neural Networks
We utilized two feature embeddings for our evaluation: MFCC [6] and a Convolutional Neural Network (CNN). We utilized the same CNN architecture as the VGG-M [27] used in VolCeleb evaluation [4]. It takes the spectrogram of an utterance as the input, and generate a feature vector of 1024 in layer fc8 (table 4 in [4] for details about this CNN).
Why don't we use more complicated embeddings? Although more complicated embedding extraction modules such as i-vectors [7] or d-vectors [8] can improve diarization, they require extensive pretraining on big datasets, which is contradictory to our problem setting and beyond our scope.
Why do we still include a pretrained CNN embedding? Indeed, if our end goal is to let the system learn from scratch without pretraining, why do we consider it in our evaluation? The CNN model was trained for speaker verification task in VoxCeleb and we are curious about the relationship between a learned representation and our online learning agents. Despite this note, we are most interested in the performance given MFCC, because we aim to push the system to the extreme of not having pretraining of any type before deployment. (e, f) the self-supervision is the best; (g, h) LinUCB is the best.

Results
Given MFCC features without pretraining, our online learning agent demonstrated a robust performance (Figure 3a,b,c,d): in most cases, it significantly outperformed the baseline. 3 Learning without Oracle (Table 2). In both the MFCC and CNN MiniVox environments, we observed that Berlin-UCB and its variants outperform the baseline most of the time. The discrepancy of performance of the MFCC and CNN environments can be explained by the innate difficulties of the two tasks: while the CNN embeddings are already well separated because they were pretrained with contrastive loss [4], in MFCC environments our online learning models need to learn from scratch both how to cluster and how to map the reward with the features, while maintaining a good balance between exploitation and exploration.
Learning with Oracle (Table 3). Given the number of speakers, the online clustering modules appears to be more effective. However, the behaviors vary: we observed that B-GMM performed the poorest in the oracle-free environments, but performed the best in some environments with oracle; we also noted that despite the consistent best model in many oracle-free environments, the standard BerlinUCB was surpassed by the baseline and its self-supervised variants in a few MFCC cases with oracle; in certain challenging cases where the reward is sparsely revealed (p=0.01 or 0.1), the selfsupervised variants improve the performance of BerlinUCB. Is self-supervision useful? To our surprise, our benchmark results suggested that for most cases, the proposed self-supervision modules didn't improve upon our proposed contextual bandit model. Only in specific conditions (e.g. MiniVox C20-MFCC-60k p=0.1 with Oracle), the selfsupervised contextual bandits outperformed both the standard BerlinUCB and the baseline. Further investigation into the reward curve revealed more complicated interactions between the self-supervision modules with the online learning modules (the contextual bandit): as shown in Figure 3e,f, B-GMM and B-KNN built upon the effective reward mapping from their BerlinUCB backbone, and benefited from the unlabelled data points to yield a fairly good performance.

CONCLUSION AND OUTLOOK
We considered the novel problem of online learning speaker diarization. We formulated the practical task as an interactive system that episodically receives sparse bandit feedback from users. During unlabelled episodes, we proposed to learn from pseudo-feedback generated by self-supervised modules enabled by clustering. We provided a benchmark to evaluate this task, and demonstrated an empirical merit of the proposed methods over standard online learning algorithm. Ongoing work include extending the online learning framework in both extraction and clustering modules, branch management (e.g. routing [28]) and self-supervision with graph methods.