{'fraction_non_alphanumeric': 0.036583965081552955, 'fraction_numerical': 0.020359522168619342, 'mean_word_length': 4.651574164232392, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB

We propose a new single-shot method for multi-person 3D pose estimation in general scenes from a monocular RGB camera. Our approach uses novel occlusion-robust pose-maps (ORPM) which enable full body pose inference even under strong partial occlusions by other people and objects in the scene. ORPM outputs a fixed number of maps which encode the 3D joint locations of all people in the scene. Body part associations allow us to infer 3D pose for an arbitrary number of people without explicit bounding box prediction. To train our approach we introduce MuCo-3DHP, the first large scale training data set showing real images of sophisticated multi-person interactions and occlusions. We synthesize a large corpus of multi-person images by compositing images of individual people (with ground truth from mutli-view performance capture). We evaluate our method on our new challenging 3D annotated multi-person test set MuPoTs-3D where we achieve state-of-the-art performance. To further stimulate research in multi-person 3D pose estimation, we will make our new datasets, and associated code publicly available for research purposes.


Introduction
Single-person pose estimation, both 2D and 3D, from monocular RGB input is a challenging and widely studied problem in vision [4,3,33,34,7,11,28,37]. It has many applications, e.g., in activity recognition and content creation for graphics. While methods for 2D multi-person pose estimation exist [43,17,8,37], most 3D pose estimation methods are restricted to a single un-occluded subject. Natural human activities take place with multiple people in cluttered scenes hence exhibiting not only self-occlusions of the body, but also strong inter-person occlusions or occlusions by objects.
This work was funded by the ERC Starting Grant project CapReal (335545). We would also like to thank Hyeongwoo Kim and Eldar Insafutdinov for their assistance.
This makes the under-constrained problem of inferring 3D pose (of all subjects) from monocular RGB input even harder and leads to drastic failure of existing single person 3D pose estimation methods.
Recent work approaches this more general 3D multiperson pose estimation problem by decomposing it into multiple single-person instances [31,49], often with significant redundancy in the decomposition [49]. The single person predictions are post-processed to filter, refine and fuse the predictions into a coherent estimate. Bottom-up joint multi-person reasoning remains largely unsolved, and the multi-person 3D pose estimation lacks appropriate performance benchmarks.
We propose a new single shot CNN-based method to estimate multi-person 3D pose in general scenes from monocular input. We call our method single shot since it reasons about all people in a scene jointly in a single forward pass, and does not require explicit bounding box proposals by a separate algorithm as a preprocessing step [49,40]. The latter may fail under strong occlusions and may be expensive to compute in dense multi-person scenes. Our fully-convolutional method jointly infers 2D and 3D joint locations using our new occlusion-robust pose-map (ORPM) formulation. ORPM enables multi-person 3D pose estimates under strong (self-)occlusions by incorporating redundancy in the encoding, while using a fixed number of outputs regardless of the number of people in the scene. Our subsequent hierarchical read-out strategy starts with a base pose estimate, and is able to refine the estimate based on which joints of a person are visible, leading to robust 3D pose results.
To train our CNN we introduce a new multi-person 3D pose data set MuCo-3DHP. While there are several single-person datasets with 3D joint annotations, there are no annotated multi-person datasets containing large corpora of real video recordings of humanhuman interaction with large person and background diversity. Important advances in this direction have been made by Joo et al . [24] using a multi-camera studio setup but background diversity remains limited.  [3] dataset (blue), as well as our new MuPoTS-3D evaluation set (green). Our pose estimation approach works for general scenes, handling occlusions by objects or other people. Note that our 3D pose predictions are root-relative, and scaled and overlaid only for visualization.
Some prior work creates 3D annotated multi-person images by using 2D pose data augmented with 3D poses from motion capture datasets [49], or by finding 3D consistency in 2D part annotations from multi-view images recorded in a studio [53]. To create training data with much larger diversity in person appearance, camera view, occlusion and background, we transform the MPI-INF-3DHP single-person dataset [33] into the first multi-person set that shows images of real people in complex scenes. MuCo-3DHP is created by compositing multiple 2D person images with groundtruth 3D pose from multi-view marker-less motion capture. Background augmentation and shading-aware foreground augmentation of person appearance enable further data diversity. To validate the generalizability of our approach to real scenes, and since there are only very few annotated multi-person test sets [12] showing more than two people, we contribute a new multiperson 3D test set, MuPoTS-3D. It features indoor and outdoor scenes, challenging occlusions and interactions, varying backgrounds, more than two persons, and ground truth from commercial marker-less motion capture. All datasets will be made publicly available. In summary, we contribute: • A CNN-based single-shot multi-person pose estimation method based on a novel multi-person 3D pose-map formulation to jointly predict 2D and 3D joint locations of all persons in the scene. Our method is tailored for scenes with occlusion by objects or other people. • The first multi-person dataset of real person images with 3D ground truth that contains complex inter-person occlusions, motion, and background diversity. • A real in-the-wild test set for evaluating multiperson 3D pose estimation methods that contains diverse scenes, challenging multi-person interactions, occlusions, and motion.
Our method achieves state-of-the-art performance on challenging multi-person scenes where single-person methods completely fail. Although designed for the much harder multi-person task, it performs competitively on single-person test data.

Related Work
We focus on most directly related work estimating the pose of multiple people in 2D or a single person in 3D from monocular RGB input. [50] provide a more comprehensive review.
Multi-Person 2D Pose Estimation: A common approach for multi-person 2D pose estimation is to first detect single persons and then 2D pose [44,13,55,19,40]. Unfortunately, these methods fail when the detectors fail-a likely scenario with multiple persons and strong occlusions. Hence, a body of work first localizes the joints of each person with CNN-based detectors and then find the correct association between joints and subjects in a post-processing step [43,16,36,8].
Single-Person 3D Pose Estimation: Existing monocular single-person 3D pose methods show good performance on standard datasets [18,51,59,41,28,27]. However, since many methods train a discriminative predictor for 3D poses [5], they often do not generalize well to natural scenes with varied poses, appearances, backgrounds and occlusions. This is due to the fact that most 3D datasets are restricted to indoor setups with limited backgrounds and appearance. The advent of large real world image datasets with 2D annotations made 2D pose estimation in the wild remarkably accurate. However, annotating images with 3D pose is much harder with many recent work focusing on leveraging 2D image datasets for 3D human pose estimation or multi-view settings [24]. Additional annotations to these 2D image datasets allow some degree of 3D reasoning, either through body joint depth ordering constraints [42] or dense shape correspondences [14]. Some works split the problem in two: first estimate 2D joints and then lift them to 3D [58,54,9,66,35,69,1,52,20,32,6,26,38,57,2], e.g., by database matching, neural network regression, or fitting the SMPL body model [30]. Some works integrate SMPL within the CNN to exploit 3D and 2D annotations in an end-to-end fashion [39,25,42].
Other work leverages the features learned by a 2D pose estimation CNN for 3D pose estimation. For example, [60] learn to merge features from a 2D and 3D joint prediction network. Another approach is to train a network with separate 2D and 3D losses for the different data sources [46,68,56,65,31]. Some approaches jointly reason about 2D and 3D pose with multi-stage belief maps [62]. The advantage of such methods is that they can be trained end to end. A simpler yet very effective approach is to refine a network trained for 2D pose estimation for the task of 3D pose estimation [34,33]. A major limitation of methods that rely on 2D joint detections directly or on bounding boxes is that they easily fail under body occlusion or with incorrect 2D detections, both of which are common in multi-person scenes. In contrast, our approach is more robust to occlusions since a base 3D body pose estimate is available even under significant occlusion. [34] showed that 3D joint prediction works best when the receptive field is centered around the joint of interest. We build upon this insight to refine the base body pose where 2D joint detections are available.
Multi-Person 3D Pose Estimation: To our knowledge, only [49] tackle multi-person 3D pose estimation from single images. 1 They first identify bounding boxes likely to contain a person using [47]. Instead of a direct regression to pose, the bounding boxes are classified into a set of K-poses similar to [45]. These poses are scored by a classifier and refined using a regressor. The method implicitly reasons using bounding boxes and produces multiple proposals per subject that need to be accumulated and fused. However, performance of their method under large person-person occlusions is unclear. In contrast, our approach produces multi-person 2D joint locations and 3D pose maps in a single shot, from which the 3D pose can be inferred even under severe person-person occlusion.
3D Pose Datasets: Existing pose datasets are either for a single person in 3D [18,51,63,64,33] or multi-person with only 2D pose annotations [3,29]. Of 1 A second approach [67] was published in the review period. the two exceptions, the MARCOnI dataset [12] features 5 sequences but contains only 2 persons simultaneously, and there are no close interactions. The other is the Panoptic dataset [24] which has a limited capture volume, pose and background diversity. There is work on generating synthetic images [48,10] from mocap data, however the resulting images are not plausible. We choose to leverage the person segmentation masks available in MPI-INF-3DHP [33] to generate annotated multi-person 3D pose images of real people at scale through compositing using the available segmentation masks. Furthermore, we captured a 3D benchmark dataset featuring multiple closely interacting persons which was annotated by a video-based multi-camera motion capture system.

Multi-Person Dataset
Generating data by combining in-the-wild multiperson 2D pose data [3,29] and multi-person multiview motion capture for 3D annotation would be a straightforward extension of previous (single-person) approaches [33]. However, multi-person 3D motion capture under strong occlusions and interactions is challenging even for commercial systems, often requiring manual pose correction constraining 3D accuracy. Hence, we merely employ purely multi-view marker-less motion capture to create the 20 sequences of MuPoTs-3D, the first expressive in-the-wild multi-person 3D pose benchmark. For the much larger training set MuCo-3DHP, we resort to a new compositing and augmentation scheme that leverages the single-person image data of real people in MPI-INF-3DHP [33] to composite an arbitrary number of multi-person interaction images under user control, with 3D pose annotations.

MuCo-3DHP: Compositing-Based Training Set
The MPI-INF-3DHP [33] single-person 3D pose dataset provides marker-less motion capture based annotations for real images of 8 subjects, each captured with 2 clothing sets, using 14 cameras at different elevations. We build upon these person segmentation masks to create per-camera composites with 1 to 4 subjects, with frames randomly selected from the 8 × 2 sequences available per camera. Since we have groundtruth 3D skeleton pose for each video subject in the same space, we can composite in a 3D-aware manner resulting in correct depth ordering and overlap of subjects. We refer to this composited training set as the Multiperson Composited 3D Human Pose dataset (see Fig. 2 for examples). The compositing process results in plausible images covering a range of simulated inter-person overlap and activity scenarios. Furthermore, user-control over the desired pose and occlu- sion distribution, and foreground/background augmentation using the masks provided with MPI-INF-3DHP is possible (see supplementary document for more details). Even though the synthesized composites may not simulate all the nuances of human-human interaction fully, we observe that our approach trained on this data generalizes well to real world scenes in the test set.

MuPoTS-3D: Diverse Multi-Person 3D Test Set
We also present a new filmed (not composited) multi-person test set comprising 20 general real world scenes with ground-truth 3D pose for up to three subjects obtained with a multi-view marker-less motion capture system [61]. Additionally, per joint occlusion annotations are available. The set covers 5 indoor and 15 outdoor settings, with trees, office buildings, road, people, vehicles, and other stationary and moving distractors in the background. Some of the outdoor footage also has challenging elements like drastic illumination changes, and lens flare. The indoor sequences use 2048 × 2048px footage at 30fps, and outdoor sequences use 1920 × 1080px GoPro footage at 60fps. The test set consists of >8000 frames, split among the 20 sequences, with 8 subjects, in a variety of clothing styles, poses, interactions, and activities. Notably, the test sequences do not resemble the training data, and include real interaction scenarios. We call our new test set Multiperson Pose Test Set in 3D (MuPoTS-3D).
Evaluation Metric: We use the robust 3DPCK evaluation metric proposed in [33]. It treats a joint's prediction as correct if it lies within a 15cm ball centered at the ground-truth joint location, and is evaluated for the common minimum set of 14 joints marked in green in Fig. 5. We report the 3DPCK numbers per sequence, averaged over the subjects for which GT reference is available, and additionally report the performance breakdown for occluded and un-occluded joints.
The relative robustness of 3DPCK over MPJPE [18] is also useful to offset the effect of jitter that arises in all non-synthetic annotations, including ours. For completeness, we also report the MPJPE error for predictions matched to an annotated subject.

Method
At the core of our approach is a novel formulation which allows us to estimate the pose of multiple people in a scene even under strong occlusions with a single forward pass of a fully convolutional network. Our method builds upon the location-maps formulation [34] that links 3D pose inference more strongly to image evidence by inferring 3D joint positions at the respective 2D joint pixel locations.We first recap the location-map formulation before describing our approach.
Location-Maps [34]: A location-map is a joint specific feature channel storing the 3D coordinate x, y, or z at the joint 2D pixel location. For every joint, three location-maps, as well as a 2D pixel location heatmap are estimated. The latter encodes the 2D pixel location of the joint as a confidence map in the image plane. The 3D position of a joint can be read out from its location-map at the 2D pixel location of the joint, as shown in Fig. 4. For an image of size W × H, 3n location-maps of size W/k × H/k are used to store the 3D location of all n joints, where k is a down-sampling factor. During training, the L 2 loss between the ground truth and the estimated location-map is minimized in the area around the joint's 2D pixel location. Although this simple location-map formulation enables full 3D pose inference, it has several shortcomings. First, it assumes that all joints of a person are fully visible, and breaks down under partial occlusion, which is common in general scenes. Second, efficient extension to multiple people is not straightforward. Introducing separate location-maps per person requires dynamically changing the number of outputs.
Occlusion-Robust Pose-Maps (ORPMs): We propose a novel occlusion-robust formulation that has a fixed number of outputs regardless of the number of people in the scene, while enabling pose read-outs for strongly occluded people. Our key insight is the incorporation of redundancy into the location-maps. We represent the body by decomposing it into torso, four limbs, and head (see Fig. 5). Our occlusion-robust pose-maps (ORPMs) support multiple levels of redundancy: (1) they allow the read-out of the complete base pose P ∈ R 3×n at one of the torso joint locations (neck or pelvis), (2) the base pose (which may not capture the full extent of articulation) can be further refined by reading out the head and individual limb poses where 2D detections are available, and (3) the complete limb pose can be read out at any 2D joint location of that limb. Together, these ensure that a complete and as articulate as possible pose estimate is available even in the presence of heavy occlusions of the body (see In addition, the redundancy in ORPMs allows to encode the pose of multiple partially overlapping persons without loss of information, thus removing the need for a variable number of output channels. See Fig. 4. Naïve Redundancy: The naïve approach to introduce redundancy by allowing full pose read-out at all body joint locations breaks down for interacting and overlapping people, leading to supervision and read-out conflicts in all location-map channels. Our selective introduction of redundancy restricts these conflicts to pose-map channels of similar limbs, i.e., wrist of one person in the proximity of a knee of another person cannot cause read-out conflicts because their pose is encoded in their respective pose-maps. If the complete pose was encoded at each joint location, there would be conflicts for each pair of proximate joints across people. We now formally define ORPMs (Sec. 4.1) and explain the inference process (Sec. 4.2).

Formulation
Given a monocular RGB image I of size W × H, we seek to estimate the 3D pose P = {P i } m i=1 for each of the m persons in the image. Here, P i ∈ R 3×n describes the 3D locations of the n = 17 body joints of person i. The body joint locations are expressed relative to the parent joints as indicated in Fig. 5 and converted to pelvis-relative locations for evaluation. We first decompose the body into pelvis, neck, head, and a set of limbs: L = {{shoulder s , elbow s , wrist s }, {hip s , knee s , ankle s } | s ∈ {right, left}} . The 3D locations of the joints are then encoded in the occlusion-robust pose-maps denoted by In contrast to simple location-maps, the ORPM M j stores the 3D location of joint j not only at this joint's 2D pixel Note that-since joint j of all persons i is encoded in M j -it can happen that read-out locations coincide for different people, i.e., ρ i1 (j) ∩ ρ i2 (j) = ∅. In this case, M j contains information about the person closer to the camera at the overlapping locations. However, due to our built-in redundancy in the ORPMs, a pose estimate for the partially occluded person can still be obtained at other available read-out locations. To estimate where the pose-maps can be read out, we make use of 2D joint heatmaps H = {H j ∈ R W ×H } n j=1 predicted by our network. Additionally, we estimate part affinity fields A = {A j ∈ R W ×H×2 } n j=1 which represent a 2D vector field pointing from a joint of type j to its parent [8]. This facilitates association of 2D detections in the heatmaps (and hence read-out locations for the ORPMs) to person identities and enables per-person read-outs when multiple people are present. Note that we predict a fixed number of maps (n heatmaps, 3n pose-maps, and 2n part affinity fields) in a single forward pass irrespective of the number of persons in the scene, jointly encoding the 2D and 3D pose for all subjects, i.e., our network is single-shot.

Pose Inference
Read-out of 3D pose of multiple people from ORPMs starts with inference of 2D joint locations for each person i in the image. Explicit 2D joint-to-person association is done with the predicted heatmaps H and part affinity fields A using the approach of Cao et al . [8]. Next, we use the 2D joint locations P 2D and the joint detection confidences C 2D in conjunction with ORPMs M to infer the 3D pose of all persons in the scene.
Read-Out Process: By virtue of the ORPMs we can read out 3D joint locations at select multiple pixel locations as described above. We define extremity joints: the wrists, the ankles, and the head. The neck and pelvis 2D detections are usually reliable, these joints are most often not occluded and lie in the middle of the body. Therefore, we start reading the full base pose at the neck location. If the neck is invalid (as defined below) then the full pose is read at the pelvis instead. If both of these joints are invalid, we consider this person as not visible in the scene and we do not predict the person's pose. While robust, full poses read at the pelvis and neck tend to be closer to the average pose in the training data. Hence, for each limb, we continue by reading out the limb pose at the extremity joint. Note again that the complete limb pose can be accessed at any of that limb's 2D joint locations. If the extremity joint is valid, the limb pose replaces the corresponding elements of the base pose. If the extremity joint is invalid however, we walk up the kinematic chain and check the other joints of this limb for validity. If all joints of the limb are invalid, the base pose cannot be further refined. This read-out procedure is illustrated in Fig. 5 and algorithmically described in the supplementary document.
2D Joint Validation: We declare a 2D joint location P 2D j i = (u, v) i j of person i as valid read-out location iff (1) it is un-occluded, i.e., has confidence value higher than a threshold t C , and (2) it is sufficiently far (≥ t D ) away from all read-out locations of joint j of other individuals: Our ORPM formulation together with the occlusionaware inference strategy with limb refinement enables us to obtain accurate poses even for strongly occluded body parts while exploiting all available information if individual limbs are visible. We validate our performance on occluded joints and the importance of limb refinement on our new test set (see Sec. 5).

Network and Training Details
Our network has ResNet-50 [15] as the core, after which we split it into two-a 2DPose+Affinity stream and a 3DPose stream. The core network and the first branch are trained on MS-COCO [29] and the second branch is trained with MPI-INF-3DHP or MuCo-3DHP as per the scenario. Training and architectural specifics are in the supplementary document.
The 2DPose+Affinity stream predicts the 2D heatmaps H COCO for the MS-COCO body joint set, and part affinity fields A COCO . The 3DPose stream predicts 3D ORPMs M M P I as well as 2D heatmaps H M P I for the MPI-INF-3DHP [33] joint set, which has some overlap with the MS-COCO joint set. For pose read-out locations as described previously, we restrict ourselves to the common minimum joint set between the two, indicated by the circles in Fig. 5.
Loss: The 2D heatmaps H COCO and H M P I are trained with per-pixel L2 loss comparing the predictions to the reference which has unit peak Gaussians with a limited support at the ground truth 2D joint locations, as is common. The part affinity fields A COCO are similarly trained with a per-pixel L2 loss, using the framework made available by Cao et al . [8]. While training ORPMs with our MuCo-3DHP, per joint type j, for all subjects i in the scene, a per-pixel L2 loss is enforced in the neighborhood of all possible readout locations ρ i (j). The loss is weighted by a limited support Gaussian centered at the read-out location. Table 1. Sequence-wise evaluation of our method and LCR-net [49] on multi-person 3D pose test set MuPoTS-3D. We report both (a) the overall accuracy (3DPCK), and (b) accuracy only for person annotations matched to a prediction TS1 TS2 TS3 TS4 TS5 TS6 TS7 TS8 TS9 TS10 TS11 TS12 TS13 TS14 TS15 TS16 TS17 TS18 TS19

Results and Discussion
The main goal of our method is multi-person 3D pose estimation in general scenes, which exhibits specific and more difficult challenges than single-person pose estimation. However, we validate the usefulness of our ORPM formulation on the single-person pose estimation task as well. To validate our approach, we perform extensive experiments on our proposed multi-person test set MuPoTS-3D, as well as two publicly available single-person benchmarks, namely Human3.6m [18] and MPI-INF-3DHP [33]. Fig. 1 presents qualitative results of our method showcasing the ability to handle complex in-the-wild scenes with strong inter-person occlusion. An extensive collection of qualitative results is provided in the supplementary document and video.

Comparison with Prior Art
For sequences with strong occlusion, we obtain much better results than the state-of-the-art. For unoccluded sequences our results are comparable to methods designed for single person. We outperform the only other multi-person method (LCR-net [49]) quantitatively and qualitatively on both single-person and multi-person tasks. For fairness of comparison, in all evaluations, we re-target the predictions from LCRnet [49] to a skeleton with bone-lengths matching the ground truth.
Multi-Person Pose Performance: We use our proposed MuPoTS-3D (see Sec. 3.2) to evaluate multiperson 3D pose performance in general scenes for our approach and LCR-net [49]. In addition, we evaluate VNect [34] on images cropped with the ground truth bounding box around the subject. We evaluate for all subjects that have 3D pose annotations available. If an annotated subject is missed by our method, or by LCRnet, we consider all of its joints to be incorrect in the 3DPCK metric. Table 1(a) reports the 3DPCK metric for all 20 sequences when taking all available annotations into account. Our method performs significantly better than LCR-net for most sequences, while being comparable for a few, yielding an overall improved performance of 65.0 3DPCK vs 53.8 3DPCK for LCR-net. We provide a joint-wise breakdown of the overall accuracy in the supplementary document.
Overall, our approach detects 93% of the annotated subjects, whereas LCR-net was successful for 86%. This is an additional indicator of performance. Even ignoring the undetected annotated subjects, our approach outperforms LCR-net in terms of 3D pose error (69.8 vs 62.4 3DPCK, and 132.5 vs 146 mm MPJPE).
VNect is evaluated on ground truth crops of the subjects, and therefore it operates at a 100% detection rate. In contrast, we do not use ground truth crops, and missed detections by our method count as all joints wrong. Despite this our method achieves better accuracy (65.0 vs 61.1 3DPCK, 30.1 vs 27.6 AUC).
Single-Person Pose Performance: On the MPI-INF-3DHP dataset (see Table 2 For detailed comparisons on Human3.6m [18], refer to the supplementary document. Our approach at 69.6mm MPJPE performs ≈17mm better than LCR-net [49] (87.7mm), and outperforms the VNect location-map [34] (80.5mm) formulation by ≈10mm.  Note again that the single-person variant is trained on similar data as VNect and the occlusion robustness is inherent to the formulation. See the supplemental document for a more detailed breakdown by test sequence.
We use the per-joint occlusion annotations from MuPoTS-3D to further assess LCR-net and our approach under occlusion. Considering both self-and inter-personal occlusions, ≈23.7 % of the joints of all subjects are occluded. Our method is more robust than LCR-net on both occluded (48.7 vs 42 3DPCK) and un-occluded (70.0 vs 57.5 3DPCK) joints.

Ablative Analysis
We validate the improvement provided by our limb refinement strategy on MPI-INF-3DHP test set. We empirically found that the base pose read out at one of the torso joints tends towards the mean pose of the training data. Hence, for poses with significant articulation of the limbs, the base pose does not provide an accurate pose estimate, especially for the end effectors. On the other side, the limb poses read out further down in the kinematic chain, i.e., closer to the extremities, include more detailed articulation information for that limb. Our proposed read-out process exploits this fact, significantly improving overall pose quality by limb refinement when limbs are available. Table 2 shows that the benefit of the full read-out is consistent over all metrics and valid for our method independent of whether it is trained on single-person (MPI-INF-3DHP) or multi-person data (MuCu-3DHP), with a ≈10 3DPCK advantage over torso read-out. See Fig. 2,3 in the supplementary document.

Limitations and Future Work
As discussed in Section 4, we handle overlapping joints of the same type by only supervising the one closest to the camera. However, when joints of the same type are in close proximity (but not overlapping) the ground-truth ORPM for those may transition sharply from one person to the other, which are hard to regress and may lead to inaccurate predictions. One possible way to alleviate the issue is to increase the resolution of the output maps. Another source of failures is when 2D joints are mis-predicted or mis-associated. Furthermore, we have shown accurate root-relative 3D pose estimation, but estimating the relative sizes of people is challenging and remains an open problem for future work. While the compositing based MuCo-3DHP covers many plausible scenarios, further investigation into capturing/generating true person-person interactions at scale would be an important next step.

Conclusion
Multi-person 3D pose estimation from monocular RGB is a challenging problem which has not been fully addressed by previous work. Experiments on singleperson and multi-person benchmarks show that our method, relying on a novel occlusion robust pose formulation (ORPM), works well to estimate the 3D pose even under strong inter-person occlusions and humanhuman interactions better than previous approaches. The method has been trained on our new multi-person dataset (MuCo-3DHP ) synthesized at scale from existing single-person images with 3D pose annotations. Our method trained on this dataset generalizes well to real world scenes shown in our MuPOTS-3D evaluation set. We hope further investigation into monocular multi-person pose estimation would be spurred by the proposed training and evaluation data sets. for all limbs l ∈ {arm l , arm r , leg l , leg r , head} do 9: for j = getExtremity(l); j / ∈ {pelvis, neck}; j = parent(j) do 10: if isValidReadoutLoc(i, j) then return isol Figure 1. The network architecture with 2DPose+Affinity branch predicting the 2D heatmaps HCOCO and part affinity maps ACOCO with a spatial resolution of (W/8, H/8), and 3DPose branch predicting 2D heatmaps HMP I and ORPMs MMP I with a spatial resolution of (W/4, H/4), for an input image with resolution (W, H).

Data
We use 12 out of the 14 available camera viewpoints (using only 1 of the 3 available top down views) in MPI-INF-3DHP [33] training set, and create 400k composite frames of MuCo-3DHP, of which half are without appearance augmentation. For training, we crop around the subject closest to the camera, and apply rotation, scale, and bounding-box jitter augmentation. Since the data was originally captured in a relatively restricted space, the likelihood of there being multiple people visible in the crop around the main person is high. The combination of scale augmentation, bounding-box jitter, and cropping around the subject closest to the camera results in many examples with truncation from the frame boundary, in addition to the inter-person occlusions occurring naturally due to the compositing.

Training
We train our network using the Caffe [21] framework. The core network's weights were initialized with those trained for 2D body pose estimation on MPI [3] and LSP [22,23] datasets as done in [33]. The core network and the 2DPose + Affinity branch are trained for multi-person 2D pose estimation using the framework provided by Cao et al. [8]. We use the AdaDelta solver, with a momentum of 0.9 and weight decay multiplier of 0.005, and a batch size of 8. We train for 640k iterations with a cyclical learning rate ranging from 0.1 to 0.000005. The 3DPose branch is trained with the core network and 2DPose + Affinity branch weights frozen. We use a batch size of 6 and train for 360k iterations with a cyclical learning rate ranging from 0.1 to 0.000001. We empirically found that training the part affinity fields and occlusion-robust pose-maps at lower resolution (see Fig. 1) leads to better results. Figure 2 shows joint-wise accuracy comparison of our approach with LCR-net [49] on the single person MPI-INF-3DHP test set. For limb joints (elbow, wrist, knee, ankle) LCR-net performs comparably or better than our torso-only readout, but our full readout performs significantly better. See Figure 3. Figure 5 shows joint-wise accuracy comparison of our approach with LCR-net on our proposed multiperson 3D pose test set. We see that our approach obtains a better accuracy for all joint types for most sequences, only performing worse than LCR-net for a select few joint types on certain sequences (Test-Seq18,19,20).

Evaluation on Single-person Test Sets
Here we provide a detailed comparison against other methods for single-person 3D pose estimation. Evaluation on Human3.6m is in Table 1, and on MPI-INF-3DHP test set in Table 2. We additionally provide comparisons with the VNect location-maps trained on our training setup, which includes the 2D pretraining, and the 3D pose samples. Table 3 provides a sequencewise breakdown for the synthetic occlusion experiment on MPI-INF-3DHP test set wherein through randomly placed occlusions ≈14% of the joints are occluded. This doesn't account for self-occlusions.