{'fraction_non_alphanumeric': 0.030653950953678476, 'fraction_numerical': 0.009744106148560597, 'mean_word_length': 4.655778894472362, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving speaker turn embedding by crossmodal transfer learning from face embedding

Learning speaker turn embeddings has shown considerable improvement in situations where conventional speaker modeling approaches fail. However, this improvement is relatively limited when compared to the gain observed in face embedding learning, which has been proven very successful for face verification and clustering tasks. Assuming that face and voices from the same identities share some latent properties (like age, gender, ethnicity), we propose three transfer learning approaches to leverage the knowledge from the face domain (learned from thousands of images and identities) for tasks in the speaker domain. These approaches, namely target embedding transfer, relative distance transfer, and clustering structure transfer, utilize the structure of the source face embedding space at different granularities to regularize the target speaker turn embedding space as optimizing terms. Our methods are evaluated on two public broadcast corpora and yield promising advances over competitive baselines in verification and audio clustering tasks, especially when dealing with short speaker utterances. The analysis of the results also gives insight into characteristics of the embedding spaces and shows their potential applications.


Introduction
As the daily production of broadcast TV and internet content is growing quickly everyday, it is an essential task to make large multimedia corpora easily accessible through search and indexing. Therefore, research effort has been devoted to unsupervised segmentation of videos into homogeneous segments according to person identity, one of which is speaker diarization, i.e. segmenting an audio stream according to the identity of the speaker. It allows search engines to answer the question "who speaks when?" and to create rich transcription of "who speaks what?", which is very useful for multimedia documents structuring and indexing.
In the literature, state-of-the-art Gaussian-based speaker diarization methods have been shown to be successful in various types of content such as radio or TV broadcast news, telephone conversation and meetings [23,19,26]. In these contents, the speech signal is mostly prepared speech and clean audio, the number of speakers is limited, and the duration of speaker turn is more than 2 seconds on average. When these conditions are not valid, in particular the assumption of speaker turn duration, the quality of speaker diarization deteriorates [29]. As shown in TV series or movies, state-of-the-art approaches do not perform well [8,4] when there are many speakers (from 28 to 48 speakers), or speaker turns are spontaneous and short (1.6 seconds on average in the Game of Thrones TV series).
To alleviate these shortcomings of speaker diarization, researches have been proposed along two fronts: better methods to learn speaker turn embeddings or utilizing the multimodal nature of video content. The recent work on speaker turn embedding using triplet loss shows certain improvements [5]. Other multimodal related works focus on late fusion of two streams by propagating labels [2,6] or high level information such as distances or overlapping duration [11,28]. pared to the gain obtained when using such methods in learning face embeddings. To explain this disparity between modalities, one can point to the clear difference in amounts of training data, as there are hundreds of thousands images from thousands identities in any standard face dataset. The limited size of speech data is very challenging to overcome because we cannot use Internet search engines to collect speech segments similarly to face images in [25,33]. Moreover, manual labeling speech segments is much more costly. To mitigate the need of massive dataset, we take advantage of pretrained face embeddings by relying on the multimodal nature of person diarization.
Although transfer learning is widely applied in other topics [34,22], transferring between acoustic and visual domains has mainly been applied to the task of speech recognition [24], in which the two streams are highly correlated. On the other hand, with respect to identity, because there is not a definite one-one inference from a face to a voice, it is still an open question of how to apply transfer learning between a face embedding and a speaker embedding. To answer this question, we start with an observation. Although one cannot find the exact voice of a person given only a face, however, if given a small set of potential candidates, it is possible to pick a voice which is more likely to come from the given face than other voices. For example, when most candidates are male voices then it is more likely to find the correct one if the voice is female. Thus, there are latent attributes which are shared between the two modalities. Rather than relying on multimodal data with explicit shared labels such as genders, ages, or accent and ethnicity, we want to discover the latent commonalities from the source domain, a face embedding, and transfer to the target domain, a speaker turn embedding. Therefore, we hypothesize that by enforcing the speaker turn embedding to have the same geometric properties with the face embedding with respect to identity, we can improve the performance of the speaker turn embedding.
Because from one space, there are different properties to be used as constrains to be enforced on the other space, we propose 3 different strategies aiming at different granularity for transferring: • Target embedding transfer: We are given the identity correspondences between the 2 modalities. Hence, given the 2 inputs from the same identity, one can force the desired embedded features of the speaker turn to be close to embedded features of the face. Minimizing the disparity between the 2 embedding spaces with respect to identity will act as a regularizing term for optimizing the speaker turn embedding.
• Relative distance transfer: One can argue that exact similar location in the embedding spaces is hard to achieve given the fuzzy relationship between the 2 modalities. It may be sufficient to only enforce relative order between identities, thus assuming that 2 people who look more similar will have more similar voices. • Clustering structure transfer: This approach focus on discovering shared commonalities between the 2 embedding spaces such as age, gender, or ethnicity. If a group of people share common facial traits, we expect their voices to also share common acoustic features. In particular, the shared common traits in our case is expressed as belonging to the same cluster of identities in the face embedding space. Experiments conducted on 2 public datasets REPERE and ETAPE show significant improvement over the competitive baselines, especially when dealing with short utterances. Our contributions are also supported by crossmodal retrieval experiments and the visualization of our intuition.
The rest of the paper is organized as follows. Sec. 2 reviews other works related to ours, Sec. 3 introduces triplet loss and the motivation of our work, Sec. 4 describes our transfer methods in details. Sec. 5 presents and discusses the experimental results, while Sec. 6 concludes the paper.

Related Work
Below we discuss prior works on audio-visual person recognition and transfer learning which share similarities with our proposed methods.
As person analysis tasks in multimedia content such as diarization or recognition are multimodal by nature, significant effort has been devoted to using one modality to improve another. Several works exploit labels from the modality that has superior performance to correct the other modality. In TV news, as detecting speaker changes produces less false alarm rate and less noise than detecting and clustering faces, speaker diarization hypothesis is used to constrain face clustering, i.e. talking faces with different voice labels should not have the same name [2]. Meanwhile in [6], because face clustering outperforms speaker diarization in TV series, labels of face clusters are propagated to the corresponding speaker turns. Another approach is to perform clustering jointly in the audio-visual domain. [28] linearly combines the acoustic distance and the face representation distance of speaking tracks to perform graph-based optimization; while [11] formulates the joint clustering problem in a CRF framework with the acoustic distance and the face representation distance as pair-wise potential functions. Beside late fusion of labels, early fusion of features proposed in [18,27] is only suitable for supervised tasks; and because their datasets are limited with 6 identities, the case is not conclusive. Note that the aforementioned works focus on aggregating two streams of information whereas we emphasize on the transfer of knowledge from one embedding space to another. By applying recent advances in embedding learning, with deep networks for face [25,30] and speaker turn [5] our goal is not only to improve the target task (as speaker turn embedding in our case) but also provide a unified way for multimodal combination.
Each of our three learning approaches draw inspiration from a different line of research. First, we can point to coupled matching of image-text or heterogeneous recognition [20,17,21] or harmonic embedding [30] as related background for our target embedding transfer. Since it is arguable that audiovisual identities contain less correlated information, our method uses the one-one correspondence as a regularization term rather than as an optimal goal. Second, as the learning targets is an Euclidean embedding space in both modalities, relative distance transfer is inspired by metric imitation [9] or multi-tasks metric learning [3]. In our work, the triangular relationship is transferred across modalities instead of neighbourhood structure or across tasks of the same modality. Finally, though co-clustering information and cluster correspondence inference have been used in transfer learning on traditional tasks of text mining [34,22], we are first to expand that concept into exploiting clustering structure of person identities for crossmodal learning.

Triplet loss and motivation
Given a labeled training set of {(x i , y i )}, in which x i ∈ R D , y i ∈ {1, 2, .., K}, we define an embedding as f (x) ∈ R d , which maps an instance x into a ddimensional Euclidean space. Additionally, this embedding is constrained to live on the d-dimensional hypersphere, i.e. ||f (x)|| 2 = 1. Within the hypersphere, the distance between 2 projected instances is simply the Euclidean distance: In this new embedding space, we want the intra-class distances d(f (x i ), f (x j )), ∀x i , x j /y i = y j to be minimized and the inter-class distances d(f (x i ), f (x j )), ∀x i , x j /y i = y j to be maximized. A major advantage of embedding learning is that the projection f is class independent. At test time, we can expect examples from a different class, or identity, to appear and still satisfy the embedding goals. This makes embedding learning suitable for verification and clustering tasks.
To achieve such embedding, one method is to learn the projection that optimizes the triplet loss in the embedding space. A triplet consists of 3 data points: (x a , x p , x n ) such that y a = y p and y a = y n and thus, we would like the 2 points (x a , x p ) to be close together and the 2 points (x a , x n ) to be further away by a margin α in the embedding space 1 . Formally, a triplet must satisfy: (2) where T is the set of all possible triplets of the training set, and α is the margin enforced between the positive and negative pairs. Subsequently, we define the loss to be minimized as:  In spite of its advantages, the triplet loss training is empirical and depends on the training data, the initialization, and triplet sampling methods. For a certain set of training samples, there can be an exponential number of possible solutions that yield the same training loss. One approach to guarantee good performance is to make sure that the training data come from the same distribution of the test data (as in [25]). Another solution for the projection to work in more general unseen cases may be to gather a massive training dataset with more training data (which is the case of FaceNet which was trained with 100-200 millions images of 8 millions of identities [30]). Although it is possible to gather such a large scale dataset for visual information, it is less the case for acoustic data. This explains why speaker turn embedding TristouNet only gains slight improvement over Gaussian-based methods [5]. To alleviate the data concern, we tackle the problem of embedding learning from the multimodal point of view. By using a superior face embedding network that was trained on a face dataset with the same identities as in the acoustic dataset, we can regularize the speaker embedding space and thus guide the training process to a better minima.

Crossmodal transfer learning
In audio-visual (or multimodal data in general) settings, data contain 2 corresponding streams If the learning process is applied independently to each modality, we can learn 2 projections f A and f V into 2 embedding spaces R d A and R d V following their own respective losses: and in which L A and L V are defined from the general embedding loss Eq. 3 to speaker turn embedding and face embedding. As shown in the experiments, f V can already achieve significantly lower than the counterpart in acoustic domain, therefore our goal is to transfer the knowledge from face embedding to the speaker turn embedding. Hence, we assume that f V is already trained with Eq. 6 using the corresponding face dataset (as well as optional external data). Using f V , an auxiliary term L V →A (f A ) is defined to regularize the relationship between voices and faces from the same identity in addition to the loss function used to train speaker turn embedding in Eq. 3. Formally, the final loss function can be written as: The transfer loss L V →A (f A ) depends on what type of knowledge is transferred across modalities. λ is a constant hyper-parameter chosen through experiments specifically for each transfer type. In the following sections, different types of L V →A (f A ) will be described in details.

Target embedding transfer
Assuming that f A projects x A i into the same hypersphere as f V (x V j ), one can observe that by enforcing f A (x A i ) to be in close proximity of f V (x V j ) when y i = y j , f A could achieve a similar training loss as f V . In that case, the regularizing term in Eq. 7 can be defined as the disparity between crossmodal instances of the same identity: The goal of Eq. 8 is to minimize intra-class distances by binding embedded speaker turns and embedded faces within the same class similarly to coupled multimodal projection methods [20,21]. In this work, we extend this goal further by adopting the multimodal triplet paradigm to jointly minimize intra-class distances and maximize inter-class distances.
Multimodal triplet loss. In addition to minimizing the audio triplet loss of Eq. 5, we also want two embedded instances to be close if they come from the same identity, regardless of the modality they comes from, and to be far from embedded instances of all other identities in both modalities as well. Concretely, the regularizing term is thus defined as the triplet loss over multimodal triplets: where m • is the modality associated with the sample x m• • , and the loss l is adapted from Eq. 4 by using the embedding appropriate to each sample modality. The set T tar denotes all useful and valid cross-modal triplets, i.e. with the positive sample to be of the same identity of the anchor (y a = y p ), and the negative sample to be from another identity (y a = y n ); and with (m a , m p , m n ) ∈ Q A,V , the set of valid modalities (all combinations except (V, V, V ), (V, V, A), and (A, A, A) already considered in the primary loss of Eq. 5). For instance, if (m a , m p , m n ) = (A, V, V ), the loss will foster the decrease of the intra-class distance between f A (x A a ) and f V (x V p ) while increasing the inter-class distance between x A a and x V n . The strategy to collect the set T tar at each epoch of the training is described in Alg. 1.
Using Eq. 9 as regularizing term in L(f A ), one can effectively use the embedded faces as targets to learn a speaker turn embedding. Note that this is similar in spirit to the neural network distillation [16], using one Algorithm 1 Target embedding transfer triplet set.
.N 2: T tar = ∅ 3: for ∀(a, p, n)/y a = y p ∧ y a = y n do 4: for m a , m p , m n ∈ {Q A,V } do 5: if d a,p + α > d a,n then 8: T tar = T tar ∪ (a, p, n) 9: Output T tar embedding as a teacher for the other. Moreover, the two modalities can be combined straightforwardly as their embedding spaces can be viewed as one harmonic space [30].

Relative distance transfer
The correspondence between faces and voices is not a definitive one-to-one, i.e. it is not trivial to precisely select the face corresponding to a voice one has heard. Therefore target embedding transfer might not generalize well even when achieving low training error. Instead of the exact locations, the relative distance transfer approach works at a lower granularity and aims to mimic the discriminative power (i.e. the notion of being close or far) of the face embedding space. Thus, it does not directly transfer the embeddings individual instances but the relative distances between their identities.
Before computing relative distances, let us define the mean face representation M y of a person and the distance between identities within the face embedding space according to: where X y is the set of visual samples with identity y. The goal is then to collect in the set T rel all audio triplets (a, p, n) with arbitrary identities where the sample p has an identity which is closer to the identity of the anchor sample a than the identity of the sample n, as defined in the face embedding. In other words, if within the face embedding space the relative distances among the 3 identities of the triplet (a, p, n) follows: then this relative condition must hold in the speaker turn embedding space as well: )) (12) Then, at each epoch, Eq. 11 and 12 can be used to collect the set T rel , as shown in Alg. 2, and the regu-larizing transfer loss L V →A (f A ) can then be defined as the average sum of the standard triplet loss over this set. In theory, relative distance transfer can achieve the same training error as with target embedding transfer, but leave more freedom to the relaxation of the exact location of the embedded features.
Algorithm 2 Relative distance transfer triplet set.
if d a,p + α > d a,n then 8: T rel = T rel ∪ (a, p, n) 9: Output T rel

Clustering structure transfer
The common idea of the 2 previous transferring methods is that people with similar faces should have similar voices. Thus they aim at putting constrains based on the distances among individual instances in the face embedding space. In clustering structure transfer, the central idea does not focus on pair of identities. but rather, we hypothesize that commonalities between 2 modalities can be discovered amongst groups of identities. For example, people within a similar age group are more likely to be close together in the face embedding space, and we also expect them to have more similar voices in comparison to other groups.
Based on this hypothesis, we propose to regularize the target speaker turn embedding space to have the same clustering structure with the source face embedding space. To achieve that, we first discover groups in the face embedding space by performing a K-Means clustering on the set of mean identity representations {M V yi } obtained as in Sec. 4.2. If we denote by C the number of clusters, the resulting cluster mapping function is defined as: Secondly, to define the regularizing term L V →A (f A ), we simply consider the set of cluster labels c yi attached to each audio sample (x A i , y i ) as the second label, and define accordingly a triplet loss relying on this second label (i.e by considering the instances (x A i , c yi )). In this way, one can guide the acoustic instances of identities from the same cluster to be close together, thus preserving the source clustering structure. How to collect the set of triplet T str to be used for the regularizing term at each epoch is detailed in Alg.3.

9: Output T str
This group structure can be expected to generalize for new identities because even though a person is unknown, he/she belongs to a certain group which share similarities in the face and voice domains. In our work, we only apply K-Means once on the mean facial representations. However, as people usually belong to multiple non-exclusive common groups, each with a different attribute, it would be interesting in further works to aggregate multiple clustering partitions with different initial seeds or with different number of clusters. As the space can be hierarchically structured, one other possibility could be to apply hierarchical clustering to obtain these multiple partitions.

Experiments
We first describe the datasets and evaluation protocols before discussing the implementation details and the results. For the reproducibility, our annotations, pretrained models, and auxiliary scripts will be made publicly available.

Datasets
REPERE [12]. We use this standard dataset to collect people tracks with corresponding voice-face information. It features programs including news, debates, and talk shows from two French TV channels, LCP and BFMTV, along with annotations available through the REPERE challenge. The annotations consist of the timestamps when a person appears and talks. By intersecting the talking and appearing information, we can obtain all segments with face and voice from the same identity. As REPERE only contains sparse reference bounding box annotation, automatic face tracks are aligned with reference bounding boxes to get the full face tracks. This collection process is followed by manual examination for correctness and consistency and to remove short tracks (less than 18 frames ≈ 0.72s). The resulting data is split into training and test sets. Statistics are shown in Tab. 1. ETAPE [13]. This standard dataset contains 29 hours of TV broadcast. In this paper, we only consider the development set to compare with state-of-the-art methods. Specifically, we use similar settings for the "same/different" audio experiments than in [5]. From this development set, 5130 1-second segments of 58 identities are extracted. Because 15 identities appear in the REPERE training set, we remove them and retain 3746 segments of 43 identities.

Experimental protocols and metrics
Same/different experiments. Given a set of segments, distances between all pairs are computed. One can then decide if a pair of instances has the same identity if their (embedded) distance is below a threshold. We can then report the equal error rate (EER), i.e. the value when the false negative rate and the false positive rate become equal as we vary the threshold. Clustering experiments. From a set of all audio (or video) segments, a standard hierarchical clustering is applied using the distance between cluster means in the embedded space as merging criteria. Then, each time 2 clusters are merged, we compute 3 metrics on the clustering set: • Weighted cluster purity (WCP) [31]: For a given set of clusters C = {c}, each cluster c has a weight of n c , which is the number of segments within that cluster. At initialization, we start from N segments with weight 1 each. The purity purity c of a cluster c is the fraction of the largest number of segments from the same identities to the total number of segments in the cluster n c . WCP is calculated as: • Weighted cluster entropy (WCE): A drawback from WCP is that it does not distinguish the errors. For instance, a cluster with 80% purity, 20% error due to 5 different identities is more severe than if it is only due to 2 identities. To characterize this point, we thus compute the entropy of a cluster, from which WCE is calculated as: • Operator clicks index (OCI-k) [14]: This is the total number of clicks required to label all clusters. If a cluster is 100% pure, only 1 click is required. Otherwise, besides 1 click to annotate segments of the dominant class, then 1 extra click is needed to correct each erroneous track of a different class. For a cluster c of n c speaker segments, the cluster cost is formally defined as: where n c i denotes the number of segments from identity i in the cluster. The cluster clicks are then added to produce the overall OCI-k performance measure. This metric simultaneously combines the number of clusters and cluster quality in one number to represent the manual effort in practical applications.

Implementation details
Face embedding. Our face model is based on ResNet-34 [15] trained on CASIA-WebFaces [33]. We follow the procedure of [25] as follows: • A DPM face detector [10] is run to extract a tight bounding box around each face. No further preprocessing is performed except for randomly flipping training images. • ResNet-34 is first trained to predict 10,575 identities by minimizing cross entropy criteria. Then the last layer is removed and the weights are frozen. • The last embedding layer with a dimension of d = 128 is learned using Eq. 6 and the face tracks of the REPERE training set.
Speaker turn embedding. Our implementation of TristouNet consists of a bidirectional LSTM with the hidden size of 32. It is followed by an average pooling of the hidden state over the different time steps of the audio sequence, followed by 2 fully connected layers of size 64 and 128 respectively. As input acoustic features to the LSTM, 13 Mel-Frequency Cepstral Coefficients (MFCC) are extracted with energy and their first and second derivatives.
Optimization. All embedding networks are trained using a fixed α = 0.2 and the RMSProp optimizer [32] with a 10 −3 learning rate. From each mini-batch, both hard and soft negative triplets are used for learning.

Face embedding
We conducted this experiment to choose the best (more accurate) face embedding to transfer to the audio domain amongst the following candidates: • VGG-Face(dim=4096): We use the model from [25], which was pretrained using 2.6 millions faces of 2622 identities. • Rn34-FC(dim=512): ResNet-34 trained using the CASIA-WebFaces and using the activation of the last layer before the softmax identity classification as face features. • Rn34-Emb(dim=128): The embedding layer is learned using the trained face tracks of the REPERE dataset. From the REPERE test set, 6000 pairs of tracks (3000 negative, 3000 positive) are selected for benchmarking the embeddings using the same/different experimental setting. We compare using the EER and the AUC of the ROC curve. From Tab. 2, we can see that the RestNet34 slightly outperforms VGG-Face, and that further using a triplet loss learned using the face tracks of the REPERE data helps improving the results. Thus in the following experiments, Rn34-Emb is chosen as embedding to transfer to the audio domain.

REPERE -Clustering experiment
We applied the audio (or video) hierarchical clustering to the 629 audio-visual test tracks of REPERE. Results are presented in Fig. 2. Face clustering with Rn34-Emb clearly outperforms all speaker turn based methods. At the beginning, Div. first merges longer audio segments with enough data so it achieves higher purity. However, as small segments get progressively merged, the performance of BIC and Div. quickly deteriorate due to the lack of good voice statistics.
Our transferring methods surpass TristouNet in both metrics, especially in the middle stages, when the distances between clusters becomes more confusing. This shows that the knowledge from the face embedding helps distinguishing confusing pairs of clusters. The gap in WCE also means that our embedding is also more consistent with respect to the intercluster distances. We should note that in WCP and WCE, segments count as one unit and are not weighted  Tab. 3 reports the number of clicks to label and correct the clustering results. Our target embedding transfer reduces the OCI-k by 30 from the closest competitor in both the best case and with the ideal number of clusters. This in practice can decrease the effort of human annotation by 10 − 12%. Other transferring methods also show improvement of 7-10%.

ETAPE -Same/different experiment
From the ETAPE development set, 3746 segments of 43 identities are extracted. From these segments, all possible pairs are used for testing and the EER is reported in Tab.4. All of our networks with transferred knowledge outperform the baselines. With short segments of 1 second, BIC and Div. do not have enough data to fit the Gaussian models well, therefore they perform poorly. By transferring from visual embedding, we can improve TristouNet with a relative improvement of 6% of EER. We should remark that in [5], the original Tris-touNet achieved 17.3% and 14.4% when being trained and tested on 1s sequences and 2s sequences respectively. However, it is important to note that our models are trained on a smaller dataset (4.5h vs. 13.8h of ETAPE data in [5]) and from an independent training set (REPERE vs. ETAPE). Using our transfer learning methods, the speaker turn embedding model could be easily trained by combining different dataset, i.e. combining REPERE and ETAPE training sets. Comparison of transfer methods. Though the difference is small, target embedding shows an advantage in both the REPERE clustering experiments and in the ETAPE experiment. It seems that as the level of granularity decreases, the performance decreases. It could be interesting in future work to combine these different transfer method to see whether any further gain could be obtained.

Parameter sensitivity
In all our transfer learning settings, we need to choose one hyper parameter λ, and the number of clusters for structure transfer setting. Hence, we perform benchmarking with different values of λ and report results in Fig. 3. In Fig. 3-(a) and (b), we can observe that except for relative distance transfer, the rest are quite insensitive to this hyper parameter λ. Each of them has a different optimal value, which is due to the difference in the nature of each method. One possible explanation for the case of relative distance transfer when λ ≥ 2 is that there is no proximity constrains on the location of the embedded features, thus instability is not bounded and can increase at test time. Fig.3-(c) shows how structure transfer performs under different granularity. Further analysis in the characteristics of clusters is presented in next subsection.

Further multimodal analysis
Each transfer method is different in nature and can be exploited differently. Below, we analyze target transfer and structure transfer. Cross modal retrieval. One interesting potential of target embedding transfer is the ability to connect a voice to a face of the same identity. To explore this aspect, we formulate a retrieval experiment: given 1 instance of the source embedding domain (voice or face), its distances to the embedding of 1 correct identities and 9 distractors in the enrolled domain are computed and ranked accordingly. There are 4 different settings depending on the within or cross domain retrieval: audio-audio, visual-visual, audio-visual, and visual-audio. Fig. 4-(a) shows the average precision of 980 different runs when choosing from the top 1 to 10 ranked results (Prec@K). Although the cross modal retrieval settings cannot compete with their single modality counterparts, they perform better than random chance and show consistency between the face embedding and speaker turn embedding. This proves that the two modalities cannot be coupled as in coupled matching learning but can be used as a regularizer of one another. Shared clusters across modalities. Fig. 4-(b) visualizes 4 clusters which share the most common identities across the 2 modalities, when using the face embedding and the speaker embedding with structure transfer. One can observe 2 distinct characteristics among the clusters which are automatically captured: gender and age. It is noteworthy that these characteristics are discovered without any supervision.

Conclusion
Inspired by state-of-the-art machine learning techniques, we have proposed three different approaches to transfer knowledge from a source face embedding to a target speaker turn embedding. Each of our approaches explore different properties of the embedding spaces at different granularity. The results show that our methods improved speaker turn embedding in the tasks of verification and clustering. This is particularly significant in cases of short utterances, an important situation that can be found in many dialog cases, e.g. TV series, debates, or in multi-party humanrobot interactions where backchannels and short answers/utterances are very frequent. The embedding spaces can also provide potential discovery of latent characteristics and a unified crossmodal combination. Another advantage of the transfer learning approaches is that each modality can be trained independently with their respective data, thus allowing future extension using advance learning techniques or more available data.
In the future, experiments with more complicated tasks such as person diarization or large scale indexing can be performed to explore the possibilities of each proposal. Also, working with other corpora in different languages is an interesting direction.