{'fraction_non_alphanumeric': 0.030702915264491246, 'fraction_numerical': 0.01603206412825651, 'mean_word_length': 4.828118113706479, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Understanding the Relation between Gestures and Language

In this paper, we explore the relation between gestures and language. Using a multimodal dataset, consisting of Ted talks where the language is aligned with the gestures made by the speakers, we adapt a semi-supervised multimodal model to learn gesture embeddings. We show that gestures are predictive of the native language of the speaker, and that gesture embeddings further improve language prediction result. In addition, gesture embeddings might contain some linguistic information, as we show by probing embeddings for psycholinguistic categories. Finally, we analyze the words that lead to the most expressive gestures and find that function words drive the expressiveness of gestures.


Introduction
Gestures are often referred to as "non-verbal language" and extensive studies in psychology, sociology, and anthropology have demonstrated the important role they play in communication (McNeill, 1992;Iverson and Goldin-Meadow, 1998;Alibali et al., 2000). While language and gesture can occur independently, people often use them together to communicate, suggesting that multimodality plays an important role in understanding gestures. In this work, we consider human gestures together with their corresponding utterances. We jointly learn gesture and word embeddings, and attempt to predict psycholinguistic categories and the language of the speaker from their gesture embeddings.
Even for humans it is very challenging to predict words from gestures alone (or vice versa), due to the many-to-many relationship between words and gestures. Therefore, instead of directly predicting one modality from the other (Desai and Johnson, 2021), we use contrastive pre-training learning to learn a joint embedding space that aligns both modalities (Kiros et al., 2014;Tian et al., 2020;Radford et al., 2021). This allows our model to learn an association between language and gestures, despite a large amount of uncertainty inherent in the task. The main contributions of this work are as follows: • First, we explore a multimodal approach to learn gesture embeddings through contrastive learning. Through validation experiments relying on these embeddings, we demonstrate that there is an association between gestures and languages representations. • Second, we probe gesture embeddings for various psychological and linguistic categories and show that gestures can be predictive of several categories with better-than-random accuracy. We find that function words, such as pronouns, preposition or modal verbs, can be predicted from the gestures. We also show that gesture embeddings can be used to predict discourse markers. • Third, we show that it is possible to predict the language of a speaker from our learned gestures embeddings. Our findings indicate that the difference in gestures across the languages may be driven by the function words. • Finally, we conduct several analyses to better understand the learned gesture representations.

Related Work
Semi-supervised Multimodal Learning. Our work builds on the idea of multimodal learning, where a model is trained to represent several modalities in a shared embedding space (Chen et al., 2019;Li et al., 2019;Lu et al., 2019;Tan and Bansal, 2019). In particular, we focus on semisupervised multimodal learning (Yuan et al., 2021;Wu et al., 2021;Zhai et al., 2021), which is effective and useful training strategy for settings where obtaining labeled training data is laborious or prohibitive. We base our model on CLIP (Radford et al., 2021), which uses a large amount of (multi-modal) unlabeled data combined with efficient pretraining objective leading to strong zero-shot performance in both language and vision tasks. Pose Estimation and Gesture Understanding. While most of the recent multimodal work has focused on modalities such as vision (Morency et al., 2007), language, or speech (Levine et al., 2009;Ginosar et al., 2019), there are also studies that highlight the importance of gestures for various aspects of human activities; see Kelly et al. (2008) for an overview. Work in this space has addressed, among other tasks, gesture recognition (Zhang et al., 2020), gesture generation (Kucherenko et al., 2020b;Ferstl et al., 2021;Yoon et al., 2020;Kucherenko et al., 2020a;Alexanderson et al., 2020), gestureto-gesture generation (Tang et al., 2019). All of these tasks, while close to our problem space, are not directly applicable for gesture representation.
Another line of work focuses on the temporal alignment and interaction of speech and gestures. Loehr (2007), shows that speech and gestures occur synchronously. (Rieser, 2015) shows that utterances and gestures are not always synchronous and suggests using λ-π calculus to model them. (Saint-Amand et al., 2013) provide a comprehensive study of the alignment of speech and gestures in a constraint-based grammar, while (Lücking et al., 2013) show that gestures follow the language but the opposite does not hold.
An important question is how to obtain labels that describe gesture. We use pseudo ground-truth pose estimates from OPENPOSE (Cao et al., 2019). While even state-of-the-art gesture recognition systems can be noisy, this noise is significantly reduced on videos such as TED talks (Yoon et al., 2018) given that there is only one speaker and have good light conditions. For a comprehensive overview of recent progress in the field of pose estimation see Munea et al. (2020).

Data
Our primary source of data is the YouTube Gesture Dataset (Yoon et al., 2018). The dataset consists of over 1,500 TED talk videos of English speakers addressing various topics like science, medicine, society, and others. The camera is usually in front of the speaker, so the gestures are visible. The dataset contains precomputed key points for the head, neck, shoulders, elbows, and wrists, with each pose represented as a 16-dimension vec- tor, with x and y coordinates for each key point. The dataset includes subtitles, auto-generated by YouTube and aligned by gentle 1 .We also use an additional dataset that we compiled ourselves, consisting of 600 videos of Spanish speakers. We process videos from the TEDx channel using the playlist "TEDx talks en Español." Subtitles for Spanish data are auto-generated and aligned by YouTube, so we just download the appropriate subtitle file. Figure  1 shows an example of gestures aligned with the corresponding language. We split each video into several clips using PySceneDetect, 2 which detects changes in the camera angle during the talk and splits a single video into several sub-clips. This is necessary so that the pose movements remain continuous during the short clips, even if the camera angle is changed.

Input Representation
One of the key considerations for our approach is how to represent gestures and language as input to our model. We start by dividing each clip into a series of gestures using the sliding window approach of Yoon et al. (2018). We cut every clip into 1 second sequences of frames; with 15 frames per second, our input becomes 15 frames. Detailed per-millisecond word alignments are available in our datasets. We align each one second interval with the corresponding phrase. Our method requires that gestures and language are timely-aligned within some interval of t seconds, where we use t=1 second. It does not require them to be aligned exactly, as the pose encoder and gesture encoder use different positional embeddings. One limitation of such an approach is that longer gestures are truncated, and very short gestures are collapsed together. We experimented with several lengths of the sliding window and found out that the choice of the time interval does not affect the overall performance.
Another possibility would be to split the utterances by word and take all the corresponding frames that fall within the given time interval. For instance, we can take the word 'hello' from the subtitles, and get all the corresponding frames while the word is pronounced. This way we guarantee that there is no overlap between the gestures, and a single word corresponds to a single series of gestures. However, previous results in the literature McNeill (2005) indicate that there are different gesture phases, and they are not necessarily timely aligned with the words. In such a case this approach would be limited. We experimented with such a setting as well, but the resulting performance is only marginally better than random. Figure 2 shows the overview of our approach. After preprocessing, poses and phrases are passed through two separate encoders. We use a transformer architecture to separately encode the text and the poses. The pose encoder model very closely follows the CLIP's base image encoder: it is a 12layer 768-wide model with 12 attention heads. We have to adjust the width from 512 to 768 to match the size of the text model, which is necessary for cosine similarity. The pose encoder is randomly initialized and takes as input a tensor of size (15,16) where 15 is the number of frames in a 1 sec. clip and 16 is the joint dimension. This input gets transformed to (15, 768) with the fully connected layer and is passed directly to the attention block, bypassing the input embedding layer. This is possible because the pose is already represented as a vector, and does not have to be embedded. We use the last frame as an end-of-sentence token for the prediction. On top of the transformer, there is a 768 × 768 projection layer. We use the multilingual XLM-RoBERTa (Conneau et al., 2019) as a pre-trained encoder for language with another projection layer on top of the encoder.

Approach
After encoding the pose and language into vectors of fixed length, they are normalized and the dot product is taken separately for each modality. The Multi-class N-pair loss (Sohn, 2016) objective is used to learn the match between the poses and the corresponding utterances in a single batch. We selected a batch of size two to make the training task easier. While contrastive learning benefits from large batch sizes (Newcombe, 2018), we found that in our case the higher the batch size, the harder it is for the model to learn. We tried batch sizes 8, 16, and 32, and the results were worse. We hypothesize this is due to a large amount of noise in the data. We use AdamW with a learning rate of 1e − 5 as an optimizer, and cosine schedule as a learning rate schedule.
One possible concern for our approach is the size of the training dataset. CLIP uses more than 350 million image-text pairs, while our dataset in- Figure 2: Overview of our approach: we train joint pose and language encoders on training data and produce embeddings. We show we can predict the native language of the speaker and several linguistic categories from the gesture embeddings alone. cludes only 180,000 1-second clips. To mitigate this, we first conduct an experiment where we show that the proposed pose encoder can predict whether the source of motion was left or right hand. Second, we use a pre-trained text encoder, namely XLMRoberta, as a text encoder. This is in contrast to CLIP, where the authors train the model from scratch. This way we only train a pose encoder, while the text encoder is only fine-tuned. Third, contrary to images that are represented as a 336x336 matrix, poses have much lower dimensionality, namely, our input has a dimension of 15x16. Our intuition is that these factors combined can substantially reduce the required amount of training data.

Alignment Validation
To make sure that the model is capable of learning gesture-language alignments, we conduct two simple experiments. We aim to verify whether a pose encoder can associate many similar (but with a substantial degree of variety) gestures with a single word/phrase, given the limited amount of training data and proposed model architecture. In other words, before learning a many-to-many mapping (many possible gestures can correspond to the same word, and many possible words can correspond to the same gesture), we want to verify that one-to-many mapping is even possible.
In the first validation experiment, from our dataset, we select only the poses where the source of motion is either the left or the right hand. We do this by calculating the circular variance (Pedregosa et al., 2011) of the angles between joints on the right side and the left side. Only the clips where the circular variance is above 750 on one side and less than 100 on the other side are selected. For these clips, we artificially insert the words 'left' or 'right' at random position in the existing utterance, depending on which side has is a high variance. This process resulted in 12,287 pose sequences with right-hand movement and 11,846 with lefthand movements.
We set the batch size equal to two and include only one left and one right pose in each batch so that it can be matched with the corresponding text in only one correct way. The resulting accuracy on the validation set is 99.93% and 100% for poses and language respectively. This experiment suggests that learning gesture-text alignments is possible when the gestures have sufficient expressiveness.
In the second validation experiment, given an input gesture and its embedding, we use a simple cosine similarity measure to find the closest text embedding. For each (gesture, text) pair from the validation set, we pick another random pair and    calculate the cosine similarity between each gesture and text, so we end up with 4 similarity scores, 2 for each gesture (or correspondingly 2 for each text). For each gesture, we take the text with the highest score as a prediction and compare the text we find using the similarity against the correct text paired with the gesture in the data. Figure 3 shows an example.
On the validation dataset, this experiment leads to 65.4% accuracy. When reversed, i.e., starting with a text embedding, we find the most similar gesture embedding according to a cosine similarity and compare it against the gold standard, we achieve 64.9% accuracy. This performance significantly higher than the random baseline indicates that the learned representations contain useful information.

Experiments and Results
To evaluate the strength of the connection between gestures and language, we perform two types of experiments. First, we perform experiments within one single language only, i.e. how gestures and language interact in either English or Spanish. The second type is cross-language, how gestures are different among English and Spanish speakers.

Single Language Experiments
We aim to predict a psycholinguistic category of utterance from the gesture embeddings obtained with the model described in Section 4. The motivation for this type of analysis is that humans might be able to understand that the person is, for instance, angry from the pose alone. We used Linguistic Inquiry and Word Count (LIWC) lexicon  and General Inquirer categories (Stone et al., 1966) to map the utterances to their categories. Namely, if the text contained any word from LIWC or General Inquirer category, we considered the whole utterance to belong to this category, i.e. label y=1, and y=0 otherwise. Some utterances can have more than one category. In total, we run 146 binary classification problems (65 LIWC categories and 81 General Inquirer categories). 4 We compare our results with a majority baseline, as well as a raw pose baseline, where we fit the logistic classifier on the vector of joints (15 frames with a 16-dimensional vector on each, flattened into the 240-dimensional vector). We use 30-fold 5 cross-validation on the validation dataset, stratified by the language variable, and grouped by the video id (i.e., several clips from the same video should only be in either the training data or the validation data, to exclude the possibility of data contamination). At each training and test fold, we additionally sub-sample English clips to make the number of English and Spanish clips equal. Therefore the accuracy of the majority baseline is always 50%. For the prediction, we use the same model as Conneau and Kiela (2018), a logistic classifier with the default parameters. To verify that the accuracy of one model is larger, we performed a one-sided Wilcoxon paired signed-rank test (Wilcoxon, 1945) on the accuracy scores from cross-validation. We decided to use this test based on the results from Demšar (2006). The null hypothesis is that the accuracies of two classifiers are the same, and the alternative hypothesis is that the embeddings/raw model is larger than the raw/majority.
We rerun our experiment with 30 different random seeds. Table 2 shows the LIWC categories where the embeddings model significantly outperforms the baseline (majority) model for a 90% of the 30 runs. 6 Interestingly, the resulting categories belong to function words. This finding indicates that gestures accompanying function words may have a more apparent visual appearance, compared to the other words. This finding extends previous work from psychology pointing to the importance of function words in communication (Chung and Pennebaker, 2007). Table 3 shows the same type of analysis for Spanish language. There is only one category overall, prepositions, also part of the function words. Table 4 presents the results for the General Inquirer categories. In addition to the pronouns, active verbs show a strong connection with gestures.
Another type of analysis we conducted is predicting the use of discourse markers in speech from the gestures. We use a list of words from Discsense (Sileo et al., 2020). Table 5 shows the results. The embedding model is significantly better than both the raw poses model and the majority model, suggesting that joint language-vision learning is beneficial for this task. We also attempt to predict Valence-Arousal-Dominance states from (Moham-mad, 2018), but neither raw poses nor embedding model could predict better than the majority baseline.
These results are in line with the findings reported in Lücking et al. (2013), where authors conducted the experiments using a dataset with manually annotated alignments between gestures and phrases, and found that prepositional phrases are associated with gestures as well.
An important observation from these results is that in many cases a classifier relying only on the raw poses significantly outperforms the majority baseline, suggesting that gestures by themselves contain information about language. Additionally, this finding is further supported by the improvements obtained with the gesture embeddings, which show that the joint learning of gestures and language is beneficial.

Experiments with English and Spanish
If gestures are indeed closely related to the corresponding language, we hypothesize that we should be able to predict the language of a speaker (e.g., English or Spanish) from the gestures alone. Table  7 shows the language prediction results using the gesture embeddings. We use the identical crossvalidation with the sub-sampling scheme as described in Section 5.1.
To further investigate which gestures lead to better-than-random accuracy on the language prediction task, we use the LIWC lexicon to identify the word categories that have the highest improvement with respect to the majority baseline. This analysis can be interpreted as follows: "While a person is using word category X, the gestures of an English speaker can be more easily distinguished from those of a Spanish speaker." Table 6 shows the accuracy of the language prediction task split by the LIWC categories. We select the poses that have the highest probability to be predicted correctly, and extract their corresponding utterances. From these utterances, we extract the LIWC category for the words, and calculate the accuracy separately for each word category. There are eleven categories where the embedding model outperforms the raw poses model and the raw poses model is better than the majority.
Additionally, we identify the words with the highest mutual information between the occurrence of the word in the utterance and the probability to be predicted correctly.  Accuracy Majority 50.0 (± 0.0) Raw Poses 51.7 (± 3.04) Embeddings 52.7 (± 3.01) Table 5: Accuracy on the discourse marker prediction task for raw poses and gesture embeddings. Some examples of discourse markers include: actually, anyway, so.
unigrams (top row) and bigrams (bottom row) with the highest mutual information. These words can be interpreted as the most expressive, as the corresponding gestures are more clearly distinctive from the other gestures. Once again, it appears that the majority of the expressive unigrams and bigrams represent function words, which further supports the strength of the connection between these groups of words and gestures. One potential concern is to confirm that there is not any unseen bias (e.g., channel information) that is helping the system to recognize the language of the speaker. We manually inspected a random sample of 100 poses and could not see any differences, such as data artifacts from pose extraction, or any other data processing issues. In addition, we conducted several analyses: • We analyzed the distribution of the joints' coordinates for the poses that were matched correctly versus incorrect ones. Similarly, we also analyzed the distribution of the joints' coordinates between English and Spanish videos. Maybe some joints are at the very specific position for English/Spanish videos that it makes very easy for the model to distinguish? We could not see any direct difference.
• We analyzed the coefficients of the logistic classifier for the embeddings model. We looked at the magnitude of the coefficients, i.e., whether some features drive the prediction. The motivation for this analysis is the following: if the logistic classifier relies on only a small number of features, instead of the learned representation as a whole, the gesture representation might be suboptimal.
• We fit the model on a single joint only. The motivation is the following: can we predict the language of the speaker from the neck (e.g., nose, shoulder) alone. Figure 4 shows the performance with the standard error bars (using 10 fold cross-validation). While the results are still worse than our proposed model, sometimes even one simple joint can lead to very strong performance.

Conclusions and Lessons Learned
In this paper, we explored the relation between gestures and language. Using a CLIP-style joint embedding model for gestures and language, applied on a bilingual multimodal dataset consisting of TED talks in English and Spanish, we report several findings: First, we found that gestures can be used to infer the corresponding language and conversely that language can be used to infer the corresponding gesture. Our proposed model can predict the matching between language and gesture with 65.4% accuracy, compared to the random 50.0% baseline.  Table 6: LIWC categories that drive better-than-random accuracy on the language prediction task. Accuracy Majority 50.0 (± 0.0) Raw Poses 60.1 (± 14.28) Embeddings 63.8 (± 11.00) Table 7: Accuracy on the language prediction task for raw poses and gesture embeddings Unigrams si, ade, y, asking, mas, here, life, po, medicine, ti Bigrams ade mas, from the, so that, and it, is to, y con, if your, we just, we can, we 've Table 8: Top 10 unigrams and bigrams in English and Spanish, with the highest mutual information between the occurrence of the ngram in the utterance and the probability to be predicted correct.
Second, we showed that it is possible to predict several social or psycholinguistic word categories from the gestures with better than random probability. Through extensive probing of gesture embeddings for LIWC and General Inquirer linguistic categories, we were able to identify the categories where gesture embeddings significantly outperform random baselines: the majority of these categories consist of function words, which is a finding that aligns with previous social science findings. We report the results separately for English and Spanish.
In a similar vein, we showed that gestures can be also predictive of discourse markers. Our results indicate that gesture embeddings contain useful information about discourse structure, outperforming both majority and joint-only baselines.
Finally, we reported that gestures by themselves are predictive of the native language of the speaker, and that gesture embeddings further improve this result. Through several analyses, we found that function words are most strongly associated with gestures, which aligns with theories of language evolution that posit that function words are closely connected to the body.
There are several limitations of this work. First, this work focuses on hands and head gestures only, ignoring whole body movements and facial expressions. We also assume that two active hands/arms perform a single gesture, while it is also possible to have two separate gestures for two hands. The gestures are also specific for public presentations.
For future work, we plan to include hand gestures in our dataset. We also consider compiling an 'in-the-wild' gestures dataset, to extend our findings to more forms of communication, and expand beyond the gestures and language for TED talks.

Acknowledgement
This material is based in part upon work supported by the John Templeton Foundation (grant #61156). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the John Templeton Foundation. We thank Hanwen Miao for his data analyses and suggestions on an earlier iteration of this project, and our colleagues Laura Biester, Dojune Min, Ashkan Kazemi, Jonathan Kummerfeld, Naihao Deng, Simin Fan, and Siqi Shen for reviewing an initial version of the manuscript and for all the feedback they provided.   52.9 (6.9) 50.1 (6.5) 1610.5 your, thy, thou Table 11: Accuracy on the General Inquirer category prediction task. The accuracy of the majority baseline is 50%. We show the categories where the accuracy of the embeddings model is significantly larger than the majority model at a 0.05 significance level. We highlight in bold the categories where the embeddings model is significantly larger than raw poses model. ± denotes standard deviation