{'fraction_non_alphanumeric': 0.03112783064099727, 'fraction_numerical': 0.013609205576454968, 'mean_word_length': 4.83745963401507, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics

We propose a novel learned deep prior of body motion for 3D hand shape synthesis and estimation in the domain of conversational gestures. Our model builds upon the insight that body motion and hand gestures are strongly correlated in non-verbal communication settings. We formulate the learning of this prior as a prediction task of 3D hand shape over time given body motion input alone. Trained with 3D pose estimations obtained from a large-scale dataset of internet videos, our hand prediction model produces convincing 3D hand gestures given only the 3D motion of the speaker's arms as input. We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation. We demonstrate that our method outperforms previous state-of-the-art approaches and can generalize beyond the monologue-based training data to multi-person conversations. Video results are available at http://people.eecs.berkeley.edu/~evonne_ng/projects/body2hands/.


Introduction
When we communicate, we convey nonverbal signals with our body and hands [33]. In particular, subtle nuances can be conveyed by performing specific conversational hand gestures, as the human hand is richly expressive with many degrees of joint freedom. This primordial form of communication is deeply ingrained in human nature. From early infancy, human babies pay extra attention to their own and others' hands [38] and subsequently learn to convey their needs via hand and finger gestures long before they speak. Endowing machines with the ability to perceive and use conversational hand gestures is therefore an essential step toward teaching them to effectively interact with humans.
However, learning the intricacies of conversational hand gestures requires vast amounts of data. While previous approaches attempted rule-based [3] and data-driven [18] methods, a learning based method from large swaths of data would allow for both modeling the fine-grained details of hand motion as well as generalization beyond the training set. Unfortunately, there are many challenges in capturing conversational hand gestures in realistic settings. These include the elaborate motions of fingers, the relatively small size of the hand with respect to the body, and frequent self occlusions. Such challenges make capturing the motion of human hands difficult, even for industry-level multi-camera, optical-marker-based motion capture systems [13]. Embodied 3D hand motion capture datasets in realistic conversational scenarios are therefore extremely rare.
In this paper, we build upon the insight that body motion and hand gestures are strongly coupled during speech [18]. By learning this correlation, we can build a reliable prior for hand gestures conditioned only on the observation of body motion. This approach allows us to take advantage of readily available body motion data from current motion capture systems as well as single-view image-  [19,42]. Specifically, we formulate a 3D hand-gesture prediction problem from 3D arm motion input, and demonstrate that body-hand correlations can be learned from a large-scale publicly-available monologue video dataset.
Leveraging the learned body-motion-to-hand correlation, we present two applications: First, we propose a learned approach to realistic conversational hand gesture synthesis from body-only input (See Fig. 1 for an example). Second, we use the learned correlation as a body-motion prior for singleview 3D hand pose estimation. While body priors for pose estimation have been classically considered in a non-learning, general setting [40], recent 3D hand trackers [49,42,48] have overlooked them. Our novelty is in proposing a learned deep body prior for the domain-specific setting of conversational gestures. Focusing on a single domain of motion, such as nonverbal communication, allows us to learn a stronger prior than in the general setting.
We demonstrate that perceptually-realistic hand gestures can be synthesized using our model by taking only body motion as input without observing the hands. Moreover, we show the potential of using body motion cues as a strong prior for single-view 3D hand pose estimation. This prior is especially effective in scenarios where the captured appearance of the hands in video is degraded due to occlusions, motion blur, or low resolution. We demonstrate that our proposed model generalizes beyond the training set of monologue data to other speakers as well as to multi-person settings.

Related Work
Conversational gestures Early work on generating plausible gestures for conversational agents [4,3,23,25,21] derive arm and hand motion via manually defined rules drawn from a set of annotated motion segments. Recent approaches learn person-specific gestures from speech [41,26,27,10,7], text [5], or both [32,44] without requiring hand-specified rules. However, these approaches use lab-recorded audio, text, and motion capture sequences from constrained environments [26,27,7], which limit the variety of captured gestures, or rely on simplifying assumptions for motion generation [6,27,32], which cannot be generalized to in-the-wild video analysis, like ours.
More recently, [10] uses a GAN to learn a person-specific mapping from speech to 2D upper-body keypoints from in-the-wild videos. In contrast, our method focuses on the link between a speaker's body and their hands. Furthermore, we use a 3D representation, which allows us to standardize across skeletons and hence, leverage a vast dataset of in-the-wild videos containing various speakers and settings. This enables learning gestural patterns in both person-specific and population-level domains.
3D hand synthesis Prior physics-based 3D hand synthesis approaches [28,36,46] generate motion patterns based on kinematic and task-specific constraints. These methods are effective when taskspecific and external contact constraints are well-defined, but cannot be immediately applied in-thewild. Recent data-driven approaches [30,18,34,39] define objective functions using a combination of cost terms, such as smoothness of motion [18] or transition likelihood between segments [30], to compose a trajectory of gesture sequences from a predefined collection of segmented gesture phases. Whereas the expressiveness of these methods is limited by the sequences contained in the collection, our approach directly regresses articulated 3D hand poses, allowing it to generalize to novel gestures and avoid the need for a predefined library.
Probabilistic generative models [26,34,24] have also been popular in hand synthesis. Using a temporal neural network trained on an inverse kinematic loss function, [24] predicts hand poses from acoustic features and upper-body joint angles and velocities captured via mo-cap. While [24], requires accurate upper-body mo-cap data, our approach trains only on in-the-wild data, which affords a richer variety of conversational situations compared to those of lab-constrained datasets. Our network takes a 3D body pose sequence as input. The body pose encoder learns inter-joint relationships, while the UNet summarizes the sequence into a body dynamics representation. Finally, the hand decoder learns a mapping from body dynamics to hands. The output is a predicted corresponding gestural hand pose sequence. L1 regression to the ground truth hand poses provides a training signal, while an adversarial discriminator ensures the predicted motion is realistic.
Single image 3D human pose estimation Classically, 3D human pose estimation relied on information such as motion stereo [9]. More recent work have exploited human pose priors [19,43] via data-driven approaches or explored placing constraints on joint angles or bone length [47,8] to recover accurate body mesh representations from a single in-the wild image, but they often disregard hands [19,22]. Our approach leverages the success of these methods to infer back the missing hands.
Current methods in 3D hand pose estimation often focus solely on priors from the hand itself by using shape priors in deep convolutional networks to constrain the hand geometry [2] and 3D articulation of joints [49], or by using a structured inverse kinematics representation [48] to regress joint locations or mesh parameters from a single RGB image [45,14]. In contrast, our method leverages contextual information by explicitly capitalizing on the readily available body dynamics of the speaker. In addition to implicitly using hand shape as a prior, we emphasize the significance of using learned human body dynamics as a prior. More importantly, while these methods rely on clean hand images as input, our method can hallucinate appropriate hand poses despite heavily obstructed views-or even, no view-of the hands by reasoning about movements from other visible upper body parts.

Our approach
Our main insight builds upon the idea that a speaker's hand gestures are highly correlated with their body motion [18]. This enables us to leverage body motion as a strong prior for synthesizing and reconstructing hand motion. To learn this prior in a data-driven way, we formulate a predictive task: given the body poses of a speaker, the goal is to predict their corresponding hand poses.
In this section, we first formulate our prediction problem and introduce our model, which is based on a 1D fully convolutional network architecture with an adversarial discriminator. We then apply our model to real world applications -hand synthesis and estimation -for which we build and train on a large scale, in-the-wild 3D motion dataset.

Problem Definition
The objective of our model, G, is to predict a sequence of 3D hand poses H from a corresponding sequence of 3D body poses B: , is defined by the 6 3D joints of a speaker's arms (wrists, elbows, and shoulders), where each joint is represented by a 3D axis-angle representation in a fixed kinematic structure. Similarly, the 3D hand pose at time t, h t ∈ R 126 , is defined by 21 3D joint angles for each hand (20 finger joints and 1 global wrist orientation, as shown in Fig. 1(c)) via an axis-angle representation in a fixed kinematic structure (126 = 2 × 21 × 3). Note that our model only considers 3D motion cues of the body and hands. In practice, such data is obtained by reconstructing the 3D body and hand poses from a parametric human model [29,37,35,16], and removing the shape variation and camera parameters. In contrast to 2D representations of gesture motion [10], our normalized 3D representation is invariant to changes in body shape and camera pose, allowing our model to generalize to unseen humans and scenes.
An Encoder-Decoder Model for G We use a fully convolutional 1-D encoder-decoder network for our hand gesture prediction model G. The network consists of a body encoder, a UNet dynamics encoder and a hand decoder, as overviewed in Fig. 2. The body encoder implicitly learns inter-joint relationships from an input sequence of T 3D body joint rotations B ∈ R T ×18 and outputs a body embedding φ B ∈ R T ×P , where P is the body embedding size. To learn a dynamics embedding summarizing the speaker's body poses for the whole sequence, we feed the body embedding through a UNet architecture with a bottleneck that allows the network to pool information from past and future contexts, and skip connections that allow high-frequency temporal information to flow through, capturing fast motions. The UNet's temporal bottleneck of size T encodes the body dynamics. The output of the UNet (in R T ×D , where D is the dynamics embedding size), is then fed as input to the hand decoder, which learns to regress to the ground truth hand posesĤ ∈ R T ×126 via an L1 loss: Learning Realistic Motion Dynamics To avoid blending together different modes of motion, we take inspiration from [10] and introduce an adversarial discriminator [11], D, conditioned on the temporal deltas of the predicted hand pose sequence. The discriminator ensures we produce life-like hand motion as output and facilitates learning realistic gesture dynamics. We use ∆ to denote a function that takes as input a sequence of hand poses and outputs the difference between consecutive poses (e.g.
The discriminator D maximizes the following objective while the generator G minimizes it: Thus, the discriminator learns to classify real or fake motion dynamics, inherently nudging the generator to produce more realistic speaker hand movements. Our full objective is thus: Our hand gesture prediction model implicitly learns the interplay between a speaker's body and their hands, which can be used as a strong prior to produce realistic communicative hand gestures. We introduce two such applications: hand synthesis and estimation.
Synthesizing 3D Hand Pose from Body Pose With only 3D body pose annotations as input, our model (defined in Eq. 1) hallucinates a realistic hand gesture sequence. This model can be particularly useful in synthesizing realistic hand gestures for body-only data, where hand motion capture is unavailable. For example, body-only mocap data (e.g., CMU Mocap [1] and KIT [31]) or existing 3D body pose estimation methods [19,22] that output torso and limb poses only.

3D Hand Pose Estimation with Body Pose Priors
We can extend our model to hand pose estimation by using an additional hand image when such input is available: where I h is a series of hand images cropped around the left and right hand regions of the input video. Dropping the last fully connected layer of a ResNet-50 model pretrained on ImageNet, we treat the rest of the ResNet-50 as a fixed feature extractor for I h . The resulting appearance-based feature is passed through a linear layer, which outputs the image embedding φ I ∈ R T ×Q , where Q is the image embedding size. The image embedding is then concatenated with the body embedding φ B and fed through the remaining unchanged UNet and hand decoder pipeline (See Fig. 2).
While existing single-view 3D hand pose estimation approaches [49,48,45,14] rely only on hand images as input, our method also leverages the inter-correlation between body pose and hand gestures. Thus, our prior based on body motion provides an additional cue for hand gesture estimation to overcome challenges caused by fundamental depth ambiguity, frequent self-occlusion, and severe motion blur. Furthermore, we consider the temporal aspect of the input, allowing our method to produce smoother, more realistic hand sequences.

Experiments
We evaluate our learned body prior on two hand gesture applications. First, we evaluate the quality of our 3D hand synthesis method by performing a perceptual user study on results from in-the-wild videos. Second, we quantitatively compare our body prior model in single-view hand pose estimation from video against state-of-the art hand tracking methods [42,48] on a ground-truth dataset captured in the Panoptic Studio [15]. In all experiments, our method is trained and tested in a person-agnostic setting: we train and test on many different speakers at once. To test the generalizability of our method, we ensure that speakers appearing in the test clips do not appear in the training set. We use 3D body pose estimates from MTC [42] as input to our model during testing.

Datasets
To study common body and hand gesture dynamics, we produce 3D body and hand annotations to accompany the large scale, in-the-wild monologue dataset presented in [10]. For evaluation, we utilize a smaller multi-person conversation dataset with clean 3D body and hand ground truths captured in a multi-view setting via the Panoptic Dome [15].
In-the-wild 3D Body and Hand Gesture Dataset Learning a body pose prior for hand gesture prediction requires a large-scale dataset capturing natural 3D body and finger movements that occur simultaneously with speech. However, such motion capture datasets are extremely rare due to limitations in capturing hands. We thus leverage a state-of-the art monocular motion capture algorithm, MTC [42], to reconstruct 3D body and hands from a large scale collection of public monologue videos [10]. MTC estimates 3D body and hands from a 3D parametric human model (Adam), defined by shape parameters and 62 3D joint rotation pose parameters (22 joints for body and 20 joint for each hand). From MTC's output, we use the pose parameters corresponding to the arms and fingers as pseudo ground-truth to train our models, Eq 1 and Eq 5. Although the reconstruction outputs from MTC contain failure cases with noticeable motion jitters and artifacts, we found them adequate for training our model when used on video with sufficient resolution. To cover a broad range of gesture styles, we annotate 81 hours of in-the-wild videos for 8 gesturing speakers covering a wide range of topics from a variety of different settings (e.g. television shows, lectures, religious sermons).
Panoptic Studio Dataset Given the lack of clean ground truth for the in-the-wild capture, we utilize an additional Panoptic Studio dataset presented in [17] for evaluation purposes. This dataset consists of 120 sequences (about 2 hours) of individuals naturally conversing in a negotiation game scenario. To handle severe occlusions among other individuals in the scene, the dataset uses a multiview approach (with 31 HD cameras) to reconstruct 3D body and hand keypoint locations. We apply an additional parametric 3D human model reconstruction algorithm [16] to convert the keypoints to a 3D angle representation, matching the 3D human model form (Adam) of our in-the-wild dataset. All data and annotations from both datasets will be publicly released.

Implementation Details
We generate training data by creating sliding windows of size 64 with an overlap of 32 frames for each sequence of the in-the-wild training set. This yields 139K total sequences, for which we use a 70/30 training/validation split. For the encoder-decoder model, we use embedding dimensions P = D = Q = 256. We optimize using Adam with a batch size of 128 and a learning rate of 10 −4 . We train with an adversarial loss every third epoch, and without, for all other epochs. Training time is 2 hours on a single GeForce RTX 2080 GPU for 200 epochs. See Supp. for additional details.

3D Hand Synthesis
We first assess whether our approach generates realistic and perceptually convincing hand pose sequences for the speaker.
Perceptual evaluation Since it is difficult for people to objectively rate the quality of 3D hand synthesis outputs, we design an evaluation comparing our synthesis results (without using any hand observation), Ours w/ B, against MTC [42]. Comparing against state-of-the-art in 3D hand pose estimation allows us to qualitatively evaluate how close results from using the body prior only (ours) can get to results from using the true hand image.
For the evaluation, we collect a new corpus of 12 in-the-wild YouTube videos from 9 speakers not seen in the training set. We visualize the hands by reembodying the predicted hands on the corresponding MTC-extracted body pose. The reembodied hands are then overlayed on the corresponding video frame with a black box over the true hands of the speaker. To provide evaluators with additional perceptual context, we also include audio. See Fig. 3 and supp. video for example visualizations.
Evaluators watched a series of video pairs (shown synced side by side), in which the hands of one video are synthesized by our method, and the other, are estimated by MTC. Evaluators were then  asked to "vote" for which hand sequence looks more realistic on the given speaker. They were given unlimited time to answer and could replay the video. We randomly sampled 19 pairs of 12-second clips, each of resolution 1280 × 720, from the YouTube corpus. We performed on 45 evaluators.

Synthesis results
Our method, based purely on a body motion prior, is qualitatively competitive against current image-based state-of-the-art in 3D hand pose estimation. In total, 54.54% of votes indicated that the synthesized hands from Ours w/ B were more realistic than those extracted by MTC. This illustrates the surprising result that even without seeing the hands, our method can synthesize hands from body-input alone that people find more realistic than those synthesized by a method that uses the true hand image to estimate the actual shape of the hand.
In Figure 3, Ours w/ B outperforms MTC in predicting hands that are self-occluded or blurry, better capturing a clasped hand or a hand swinging up to point. The lowest votes our method receives on a video is 36%, and highest, 77%. We note that in videos where MTC outperforms, the speaker's hand pose is often semantically linked to auditory keywords (e.g. holding up the index finger and saying "one"), which may be missed by observing only the body pose of the speaker. See Supp. for details.
Our method successfully synthesizes perceptually convincing hand poses-without any pixel information-by observing the body pose of the speaker. This supports our key technical insight that hands are closely linked to body dynamics via a learnable relation. We can therefore explicitly leverage body pose as an effective prior to improve hand pose estimation.

Single-image 3D Hand Pose Estimation
We quantitatively assess the advantage of our body pose prior for 3D hand pose estimation. We use the Panoptic Studio capture with ground-truth data to compare against alternative approaches, including state-of-the art in monocular hand motion capture, MTC [42]. In this experiment, we use the same monocular video for all methods by selecting the camera view with the clearest, nonoccluded frontal view of the speaker for each game. Yet, the social game scenario (three individuals Figure 5: For each hand pose example query, we find 10 closest predicted hand poses from in-the-wild videos and visualize their corresponding body poses (darker means closer match). We reembody the query hands on its corresponding body shown in darkest shade. Body2Hands captures distinct correlations between the body and hands for common communicative gestures. -Monocular Total Capture (MTC) [42]: Current state-of-the-art full body + hand 3D pose estimation.
-Minimal Hand (Minimal) [48]: Current state-of-the-art monocular 3D hand pose estimation. For each video frame, we extract cropped bounding box images of both the left and right hands using OpenPose. We feed both images individually through the pretrained model provided by the authors to extract 3D keypoint estimations of each hand.
-Nearest Neighbors (NN): For a body pose sequence, we find its nearest neighbor from the training set and transfer its corresponding hand pose sequence.
-Always predict a median pose (Median): A simple yet strong baseline exploiting the prior that a speaker's hands are at rest most of the time [20]. We obtain median hand pose from the test set.
Evaluation metric We compute the average Euclidean distance between the predicted and ground truth 3D hand joint locations. Since the scale of 3D joints from each 3D parametric model differs (Adam [16] for ours and MTC and MANO [37] for Minimal Hand), we apply rigid alignment by Procrustes analysis [12] to normalize for scale and global orientation before computing errors (commonly done in the 3D pose estimation field [42,19]). We report average error over the entire sequence, and scale the error unit to millimeters based on a reference shoulder distance of 30 cm. Our body-only approach Ours w/ B makes significant gains over MTC and Minimal on frames with difficult views of the hands (≈ 60% of frames). While MTC expectedly outperforms on clean views of the hands, the margin separating it from both our body prior methods is much lower. This result supports our key technical novelty in using body as a strong prior for hand pose estimation. Despite 31 camera views in the Panoptic Dome, finding a view where hands are constantly and clearly in-view is difficult (shown in Figure 4). By observing the body dynamics, our method naturally overcomes occlusion and motion blur, outperforming image-based methods. Thus, information provided by the more visible body pose is essential for deriving accurate hand pose estimates. Further, the results demonstrate our model, trained on monologue videos, generalizes to broader conversational domains.
While Ours w/ B alone outperforms all other baselines, by adding a weak appearance-based cue, Ours w/ B + I improves performance, though the margin is smaller than those separating Ours w/ B from all other baselines. This demonstrates the effectiveness of using body priors to inform image-based methods towards more accurate hand pose estimates. Figure 5 shows examples of links our method discovers between the hand and body pose of various speakers. For each chosen hand pose, we find the 10 closest predicted hand poses from the test set and visualize the body pose at that instance. More distinctive hand patterns with semantic significance (e.g. pointing or pinching to show "a little bit") have the most consolidated body poses, demonstrating a stronger link between the hands as a function of the body. While non-semantic hand poses may be associated with a greater variety of body poses, the overall composition of the body poses is still noticeably distinct. For example, the bodies accompanying the straight hands are always upright, while those of the clasped hands have a bit of a lean.
In Figure 6 we show examples where Ours w/ B predicts incorrect, albeit reasonable, hand poses, while Ours w/ B + I uses the additional appearance cue to accurately snap to the correct hand pose. Table 2 quantifies the impact of the body prior in our network. As demonstrated in an ablation, using only a weak appearance cue Ours w/ I in our network is not sufficient to capture the speaker's hand pose, leading to poorer results. We also substitute in the 3D ground truth body pose of the speaker (i.e. the true pose given by the Panoptic Studio) for the extracted MTC body pose to test how the source of the body estimates affect our results in Ours w/ GT B + I and Ours w/ GT B. We see that more accurate body poses can further improve results, while improvement gains from the additional image feature is less impactful with more accurate body pose inputs. See Supp. for video, discussion on person-specific models, and additional architecture details.

Discussion
We present Body2Hands: a data-driven approach towards learning the interplay between a speaker's hand and their body dynamics. We leverage this link to exploit the more visible body pose as a strong prior for hand synthesis and estimation. Trained only on in-the-wild data, our method's results on two datasets across various individuals and conversational settings demonstrate the promise of our idea.
Future work includes integrating our prior with more sophisticated image-based approaches, incorporating additional observable priors such as audio, and extending beyond conversational gestures.