{'fraction_non_alphanumeric': 0.03608138179506199, 'fraction_numerical': 0.016742608879940658, 'mean_word_length': 5.032278683285394, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robust Latent Representations via Cross-Modal Translation and Alignment

Multi-modal learning relates information across observation modalities of the same physical phenomenon to leverage complementary information. Most multi-modal machine learning methods require that all the modalities used for training are also available for testing. This is a limitation when the signals from some modalities are unavailable or are severely degraded by noise. To address this limitation, we aim to improve the testing performance of uni-modal systems using multiple modalities during training only. The proposed multi-modal training framework uses cross-modal translation and correlation-based latent space alignment to improve the representations of the weaker modalities. The translation from the weaker to the stronger modality generates a multi-modal intermediate encoding that is representative of both modalities. This encoding is then correlated with the stronger modality representations in a shared latent space. We validate the proposed approach on the AVEC 2016 dataset for continuous emotion recognition and show the effectiveness of the approach that achieves state-of-the-art (uni-modal) performance for weaker modalities.


INTRODUCTION
The term modality refers to the particular form in which something exists or is experienced or expressed [1]. Most physical phenomena we experience consists of multiple modalities; for example, we can see, hear and touch the rain; objects around us may have their own characteristic shape, sound and smell. The information to completely explain an event is often unevenly spread across the multiple modalities that capture this event. Multi-modal machine learning utilises multiple modalities to model or explain an event; whereas uni-modal or mono-modal machine learning uses only one of these modalities [2].
The uni-modal performance of individual modalities on any task associated with an event might be significantly different [3]; modalities whose individual performances are comparatively better (worse) are referred to as stronger (weaker) modalities [4]. Thus, the techniques that can effectively combine both the supplementary and complementary information provided by these multiple modalities, called multi-modal fusion methods, provide improved performance compared to uni-modal methods [5][6] [7]. However, in general, most multi-modal fusion techniques demand the simultaneous presence of all the modalities that were used during the model training phase, for the testing phase as well [1]. This requirement becomes a severe limitation in case one or more sensors are missing or severely corrupted by noise during testing, unless such situations are explicitly handled within the modelling framework [8]. Thus, it is inevitable to improve the testing performance of individual modalities by utilising other modalities during training [3][9] [10]. In particular, since the individual modalities corresponding to the same physical phenomenon might not perform equally well on the downstream task, it is logical to improve the uni-modal testing performance of the weaker modalities by utilising the stronger modalities during the training stage.
To this end, we propose Stronger Enhancing Weaker (SEW), a framework for improving the testing performance of weaker modality by exploiting the stronger modality during the training phase only. SEW is a supervised neural network framework for knowledge transfer from a stronger to a weaker modality. During training, the stronger modality serves as auxiliary or guiding modality that helps to create weaker-modality representations that are more discriminative, compared to the representations obtained using uni-modal training, for the classification or regression task. We achieve this by using a combination of weaker-to-stronger modality translation and feature alignment with respect to stronger modality representations. This solution is based on the intuition that inter-modal translation can create intermediate representations that capture joint information between both the modalities. Explicit alignment between the intermediate and the stronger modality representations further encourages the framework to discover components of weaker modality that are maximally correlated with the stronger modality. It is to be noted that, with the SEW framework based training, the presence of stronger modality is no longer required at the testing phase. We show the effectiveness of our method on the AVEC 2016 audiovisual continuous emotion recognition dataset where SEW improves the uni-modal performance of weaker modalities.

RELATED WORK
Most works on multi-modal training for uni-modal performance enhancement are for applications where the different modalities are essentially different types of images. For example, multi-modal training using RGB and depth images can be used to improve the unimodal performances for the task of hand gesture recognition [3]. This is done by forcing the modality-specific parts of the network to learn a common correlation matrix for their in-depth feature maps. Depth images were also used to improve the test-time performance of RGB on action recognition task using an adversarial loss based feature alignment [10]. However, it is to be noted that the modalities considered in these works are same-sized images and the uni-modal networks share the same architecture. This prevents their direct applicability to distinct modalities like audio, video and text, where the modalities differ from each other in terms of the characteristics of the features and dimensionality. Nevertheless, of late, a few works have been proposed in this realm [9][11] [12]. A sequence-to-sequence network with cyclic translation between multiple modalities can be used to obtain an intermediate representation which is robust to missing modalities during testing for sentiment analysis [11] . A Multi-modal co-learning framework has been shown to improve the uni-modal performance of text modality via training using audio, video and text modalities [12]. However, these methods have been shown to primarily benefit the uni-modal performance of text modality, which is the stronger modality for the task and the data sets considered. In contrast, we aim to explicitly improve the weaker modality using the stronger modality during training. A joint audiovisual training and cross-modal triplet loss can be used to develop a face/speech emotion recognition system using multi-modal training [9] . An inherent caveat in such a system can be the degradation in performance of the stronger modality caused by the weaker modality as seen from the test results in [9].

PROPOSED METHOD
In this section, we describe in detail our proposed Stronger Enhancing Weaker (SEW), a supervised neural network framework, which uses both stronger and weaker modality representations during training to improve the testing performance of weaker modality (Figure 1). Our framework is built on two important concepts, namely inter-modality translation and feature alignment. These concepts are implemented via four main modules: an inter-modal translator, an intra-modal auto-encoder, a feature alignment module and a taskspecific regressor or classifier. These modules are described next.
The inter-modal translator contains an encoder, WE and a decoder, SD1. The translator takes the weaker modality features, MW, as input and produces the stronger modality features,MSW, as output. The encoder of inter-modal translator, thus creates intermediate representations, msw, that capture joint information between both modalities. This is achieved by using a translation loss, Ltr, between the true, MS, and predicted,MSW, features of the stronger modality: WE is encouraged to discover components of weaker modality that are inclined towards the stronger modality by increasing the alignment between msw and the stronger modality representations. For this purpose, the stronger modality features have to be projected into the same latent space as msw. An intra-modal autoencoder is thus used to create stronger modality representations, mss, of the same dimensionality as that of the inter-modal translator representations,msw, by using an auto-encoding loss, Lae, between true, MS, and predicted,MS, features: Similarly to [11], we use Mean-Square-Error (MSE) as Ltr and Lae for modality reconstructions. A feature alignment loss, L al , ensures that the intermediate representations of the inter-modal translator are maximally aligned to the stronger modality representations: Following [13][14], we use Canonical Correlation Analysis (CCA), described as follows, for feature alignment such that L al = -CCA. CCA for deep neural networks, also known as Deep CCA or DCCA, is a method to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated [15]. For a training set of size, p, Ms ∈ R p×d 1 and Mw ∈ R p×d 2 are the input matrices corresponding to the stronger and weaker modalities respectively. mss ∈ R p×d and msw ∈ R p×d are the representations obtained by nonlinear transformations introduced by the neural network layers in the encoders SE and WE respectively. Note that SE and WE have brought the individual modalities with dimensions d1 and d2 into a common latent dimension d.
If θes and θew denote the vectors of all parameters of SE and WE respectively, then the goal of DCCA is to jointly learn parameters for both the views such that correlation (ρ) between mss and msw is as high as possible, i.e., Ifmss andmsw be the mean-centred versions of mss and msw respectively, then the total correlation of the top-K components of mss and msw is the sum of the top-K singular values of the matrix, T = Σ −1/2 s ΣswΣ −1/2 w , in which the self (Σs, Σw) and cross covariance (Σsw) matrices are given by where r1 > 0 and r2 > 0 are the regularisation constants. The gradient of correlation thus obtained on the training data is used to find (θ es , θ ew ). Finally, the task specific regressor or classification module, which takes the inter-modal translator representations as input, ensures the discriminative ability of the resulting latent space. This module is trained using a prediction loss, Lpr, that takes the true, T l , and predicted task labels, P l , as input: The total training loss combines the four components: where α, β and γ are weighting hyper-parameters. After training, all the components except the encoder, WE, and the regressor, R, are removed, so that the stronger modality is not required at the testing (deployment) phase.

VALIDATION
In this section, we compare the performance of SEW with other unimodal methods [16][17] [18][19] and a state-of-the-art cross-modal knowledge transfer method [9]. We describe the dataset, evaluation metrics, architecture and training details, and present an ablation study, which quantifies the contributions of different parts of SEW.

Dataset and Evaluation Measures
We use RECOLA, the AVEC 2016 emotion recognition dataset [16], which contains audiovisual recordings of spontaneous and natural interactions from 27 French-speaking participants. Continuous dimensional emotion annotations (in the range [-1,1]) in terms of both arousal and valence are provided with a constant frame rate of 40 ms for the first five minutes of each recording, by averaging annotations from all annotators and also taking the inter-evaluator agreement into consideration [16]. The dataset is equally divided into three sets, by balancing the gender, age, and mother-tongue of the participants with each part consisting of nine unique recordings, resulting in 67.5 k segments in total for each part (training, development and test). Since, the test labels are not publicly available, we report the results on the development set. We have used the same audio and video features as the AVEC 2015 and 2016 baselines [16] for fair comparison with previous literature. These are, 88-D extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) features extracted using openSMILE, LGBP-TOP based 168-D video-appearance features and 49 facial landmarks based 632-D video-geometric features. It is to be noted that the dataset provides separate features for both arousal and valence. This dataset is ideal for our objective, since the unimodal performances of audio and video (both geometric and appearance) features vary considerably for both arousal and valence as reported in the previous literature [9] as well as verified by our experiments (see Table  1). Similar to the AVEC 2016 challenge, we use Concordance Correlation Coefficient (CCC) (eq. 10) as the primary evaluation metric.
where x and y are the true and predicted labels respectively. µx, µy, σx, σy and σxy refer to their means, variances and covariance respectively. We also evaluate the binary classification results by segregating the true and predicted annotations into negative [-1,0] and positive (0,1] classes.

Experiments
In order to identify the stronger and weaker modalities, we first assessed the unimodal performances of audio, video-geometric and video-appearance features for arousal and valence using a regressor similar to [9]. It consists of 4 single time-step GRU-RNN layers, each made up of 120 neurons, followed by a linear layer and trained using MSE loss. The unimodal results thus obtained are shown in Table 1. For arousal, the performance of audio surpasses both video-geometric and video-appearance features. For valence, the video-geometric features out-perform audio and video-appearance features. Thus, we have 5 cases for cross-modal knowledge transfer from stronger to weaker modalities, namely, video-geo(+audio) and video-app(+audio) for arousal and audio(+video-geo), videoapp(+audio) and video-app(+video-geo) for valence, where the modality in parenthesis indicates the stronger modality.

Architecture and Training
Because the proposed method combines multi-modal data with different characteristics, it was necessary to adjust various architectural parameters according to the characteristics of the given modalities rather than solving the problem using one generic model. Hence, the encoders and decoders of both inter-modal translator and intramodal auto-encoder of all the 5 different multi-modal combinations vary from each other. We performed an exhaustive neural architecture search over the number of encoder linear layers, the number of decoder linear layers and latent space dimension. Specifically, the encoder and decoder for each modality differ in terms of the number of linear layers and the number of neurons in each layer. Since the provided video-appearance features were already refined using PCA, we did not reduce the dimensionality further and used a single linear layer of size 168 for both its encoder and decoder.    [17] 0.502 0.512 0.519 0.529 MTL (PU) [18] 0.508 0.502 0.506 0.468 DDAT (RE) [19] 0.544 0.539 0.508 0.528 DDAT (PU) [19] 0.513 0.518 0.498 0.514 EmoBed [9] 0.527 0.549* 0.521 0.564* SEW 0.565* 0.544 0.552* 0.554 kept 32. The number of CCA components, K, was kept 10 in all the experiments. The contribution from all loss components were kept equal by using α = β = γ = 1 in eq. 9. Table 2 reports the results using full SEW framework as well as after ablating individual components from it. The bottom row of Table  2 provides the uni-modal results for the weaker modalities for ease of comparison with the SEW results. Comparing the last 2 rows, we can see that the SEW-(CCA&SD1) results are close to the unimodal results for the weaker modality. This is as expected since SEW-(CCA&SD1) contains only the WE and regressor with no interaction with the stronger modality. In all the 5 cases, SEW was able to improve the results from the uni-modal and SEW-(CCA&SD1) models both in terms of CCC and binary accuracy. For arousal video-geo(+audio) and video-app(+audio), removing the CCA based alignment causes a drop of 0.053 and 0.036 respectively in CCC and 2.9% and 4.1% respectively in binary accuracy. The corresponding numbers for valence audio(+video-geo) are 0.056 and 2.5% respectively. These observations prove the significance of the CCA based distribution alignment in the SEW framework. For valence videoapp(+audio) and video-app(+video-geo), removing the decoder of inter-modal translator causes a drop of 0.040 and 0.044 respectively in CCC and 4.4% and 4.8% respectively in binary accuracy, which proves the effectiveness of weaker to stronger modality translation.

Results
In Table 3, we compare the best uni-modal results of SEW with 4 most relevant works from the previous literature in terms of CCC. [16][17] [18] [19] are different uni-modal methods whereas [9] is a cross-modal training method. [16] provides the baseline results on RECOLA dataset for the AVEC 2016 challenge. The uni-modal baseline method used an SVM based classifier on the individual features. From Table 3, it can be seen that the SEW framework significantly outperforms the baseline uni-modal results for all the weaker modalities considered. Our method is also able to improve the unimodal results for all the cases from [19], which uses difficulty awareness based training framework and [17][18] which uses multi-task learning. SEW performs better than EmoBed [9] for arousal videogeo(+audio) and valence audio(+video-geo) by a margin of 0.038 and 0.031 respectively in CCC. For arousal video-app(+audio), the performance of SEW and EmoBed are very close (0.544 and 0.549 respectively). However, for the valence video-app features, EmoBed performs comparatively better than SEW. Nevertheless, comparison between the values in the top and bottom rows of Table 2 shows that SEW framework has been able to improve the uni-modal performances of the weaker modalities. Specifically, to the best of our knowledge, the highest results to date in literature on the uni-modal performances of arousal video-geometric features and valence audio features have been achieved using the SEW framework, thus proving the effectiveness of the method.

CONCLUSION
It is evident from the abundant previous literature on multi-modal learning that there exists gaps between the uni-modal performances of different modalities corresponding to the same task. Thus, it is intuitive to improve the performance of the weaker performing modalities by using the stronger modalities during training. Our proposed method, Stronger Enhancing Weaker (SEW), provides a framework for cross-modal knowledge transfer from stronger to weaker modality. Performance of SEW on the RECOLA dataset for the task of continuous emotion recognition proves that it is able to improve the uni-modal performance of weaker modality using the stronger modality during training. Further work involves, extending this framework for multi-modal sequential data.