{'fraction_non_alphanumeric': 0.03227276871182423, 'fraction_numerical': 0.018628255386391953, 'mean_word_length': 4.968081247733044, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Heatmap-Style Jigsaw Puzzles Provides Good Pretraining for 2D Human Pose Estimation

The target of 2D human pose estimation is to locate the keypoints of body parts from input 2D images. State-of-the-art methods for pose estimation usually construct pixel-wise heatmaps from keypoints as labels for learning convolution neural networks, which are usually initialized randomly or using classification models on ImageNet as their backbones. We note that 2D pose estimation task is highly dependent on the contextual relationship between image patches, thus we introduce a self-supervised method for pretraining 2D pose estimation networks. Specifically, we propose Heatmap-Style Jigsaw Puzzles (HSJP) problem as our pretext-task, whose target is to learn the location of each patch from an image composed of shuffled patches. During our pretraining process, we only use images of person instances in MS-COCO, rather than introducing extra and much larger ImageNet dataset. A heatmap-style label for patch location is designed and our learning process is in a non-contrastive way. The weights learned by HSJP pretext task are utilised as backbones of 2D human pose estimator, which are then finetuned on MS-COCO human keypoints dataset. With two popular and strong 2D human pose estimators, HRNet and SimpleBaseline, we evaluate mAP score on both MS-COCO validation and test-dev datasets. Our experiments show that downstream pose estimators with our self-supervised pretraining obtain much better performance than those trained from scratch, and are comparable to those using ImageNet classification models as their initial backbones.


Introduction
2D human pose estimation refers to the task of locating important human body parts (e.g. elbow, head, ankle, wrist, shoulder) from input images, which serves as a basic task for varieties of computer vision applications, such as person re-id [40], video analysis [53], action recognition [50], and human-computer interaction [20].
In recent years, the progress of 2D pose estimation has been greatly benefited by CNNs [18,45,51]. There are three mainstreams for this task: (1) top-down methods [8,42] firstly separate all person instances with person detector, and then localise joints for each person instance with singleperson pose estimation model; (2) bottom-up approaches [4,25] firstly utilise joint estimator to localise all person joints, and then assign them to different person instances; (3) single-stage methods [31,43] predict person instances and human keypoints simultaneously. These three types of methods are suitable for different circumstances, but stateof-the-art performances usually come from heatmap-based top-down approaches [44], such as HRNet-based Networks [19,42,54], Pyramid-based Networks [3,8,27,41], and Hourglass Networks [21,30]. These methods utilise CNNs to predict pixel-wise 2D heatmap for accurate human keypoint localisation. Our paper is focused on the pretraining of single-person pose estimator in the top-down framework.
While self-supervised learning has recently achieved significant progress in varieties of computer vision tasks [1,6,17,29,48], supervised methods are still dominant for the pretraining of 2D pose estimation. Instead of finetuning from ImageNet classification weights [10,42,49], we design Heatmap-Style Jigsaw Puzzles (HSJP) problem as an efficient self-supervised learning pretext task for 2D human pose estimation. The description of our HSJP task is demonstrated in Figure 1. During the preprocessing of our HSJP task, we split the image of each person instance into N × N patches and shuffle them, which are illustrated in Figure 1(a) and 1(b). As we have mentioned that the target of 2D pose estimation is to localise positions of human keypoints from input images, and in order to make our pretext task as similar to its downstream task as possible, the goal of HSJP task is to estimate the shuffled place of each original image patch with a heatmap-based approach. As Figure 1(c) shows, the ground truth of our heatmap for the channel corresponding to each original image patch is a 2D Gaussain distribution located in the shuffled centre. For example, the top-left patch in Figure 1(a) is located in the bottom-right of Figure 1(b); therefore, the peak of Gaussian Distribution in the first channel is located in its bottom-right. In this way, the target of our HSJP pretext task is quite similar to those of many state-of-the-art 2D pose estimation methods: all of them predict target locations by heatmap-based means. During the pretraining process of our HSJP task, the network learns to predict Gaussian peaks as centres of those shuffled patches for correspondences to solve the jigsaw puzzles problem: the image patch on the Gaussian peak of the ith channel in Figure 1(c) should be put back to the ith patch in Figure 1(a), i ∈ [0, N 2 ). These correspondences learnt in HSJP task are also helpful to the downstream task of 2D human pose estimation, for the case that knowing relative positions of image patches is also beneficial to the localisation of human keypoints. For instance, if we know whether each shuffled patch should be the top-left or topright corner in Figure 1, it's also helpful to locate shoulders of given person instance, as person shoulders are frequently located in neighbourhoods of top-left or top-right corners in original images before data augmentation.
Instead of supervised pretraining on labelled ImageNet dataset, our network is only pretrained with self-supervised HSJP pretext task on all person instances of MS-COCO [28] dataset, and finetuned on their human keypoint labels. Although we make no use of extra labels for pretraining, our keypoint performance is comparable to that of ImageNet pretraining on MS-COCO validation and test-dev datasets.

Heatmap-based human pose estimation
Since [44] firstly proposed to utilise a heatmap-based method for single-person pose estimation, recent state-ofthe-art performances have mostly been achieved by relative means. For example, Hourglass-based Networks [21,30] apply symmetric network structures with skip connections to generate pixel-wise feature maps for accurate keypoint prediction; Pyramid-based Networks [8,41] consolidate feature maps across ResNet stages, and some improvements were achieved by repeating pyramid architecture [3,27]; HRNet [42] keeps high-resolution representations and exchanges information across multi-resolution subnetworks, and some recent works [19,54] focus on improving the preprocessing and postprocessing details of their codebase.

Self-supervised pretraining for computer vision
Self-supervised pretraining methods aim at designing pretext tasks without using handcrafted annotations so as to benefit their downstream targets. The main idea of selfsupervised representation learning is to utilise annotations that are freely available from original datasets. It also saves labour costs of labelling datasets for machine learning. The past few years have witnessed the springing up of selfsupervised means for computer vision field. A variety of pretext tasks have been proposed, such as spatial position prediction [5,11,32], transformation prediction [12,48], image inpainting [33], image colourisation [26,46,55], and learning latent embeddings [6,16,17,47,52]. Recently, some papers also discussed the robustness and usefulness of self-supervised pretraining approaches [7,29].
Applications on various computer vision tasks have also been discovered by self-supervised learning researchers in recent years. For examples, [1] predicts boxes constructed from dataset to benefit cardiac MR image analysis, and [48] utilises self-supervised attention for weakly supervised semantic segmentation. However, supervised pretraining methods on large datasets such as ImageNet [10,42,49] are still dominant for 2D human pose estimation. Starting from the status quo, we design to solve Heatmap-Style Jigsaw Puzzles (HSJP) as pretext task for pretraining heatmapbased 2D pose estimation networks, and we have achieved competitive performance on MS-COCO benchmark.

Methods for solving jigsaw puzzles
The target of jigsaw puzzles problem is to put shuffled image patches back to their original places, which serves as an effective pretext for learning visuospatial dependencies in real world [36]. Some previous works [5,32] also made use of jigsaw puzzles to learn vector-based representations for computer vision tasks such as image classification or object detection, but never had we encountered any paper that solve jigsaw puzzles as the pretext task for 2D human pose estimation methods: they are usually non-pretrained [3,27,30], or pretrained on ImageNet dataset [10,42,49].
A large number of papers focus on providing accurate methods for solving jigsaw puzzles. Early works [14,24] consider using information from shape or appearance to solve this problem. Recently, there are two mainstreams for reassembling image patches: greedy methods [15,34,38] and global methods [2,9,39]. We don't concentrate on proposing any accurate method for solving jigsaw puzzles in this paper, but aim at learning suitable representations for pretraining heatmap-based 2D pose estimation networks.

Heatmap-Style Jigsaw Puzzles
As is mentioned in the introduction, the target of 2D pose estimation is to localise positions of human keypoints from 2D input images. In order to make our self-supervised pretraining target as relative to downstream task as possible, we propose Heatmap-Style Jigsaw Puzzles (HSJP) as pretext task for self-supervised pretraining, whose goal is to estimate the shuffled locations of each original image patch. The pretext and downstream tasks both predict certain part of a person instance (patch or keypoint) with a heatmapbased approach. In this section, we will elaborate details of how to pretrain a 2D human pose estimator with HSJP task.

Target description for HSJP pretext task
The target of our proposed HSJP pretext is to estimate the shuffled place of each original image patch with a heatmap-based approach, whose inputs include all person instances cropped from COCO dataset. To make our selfsupervised pretraining method comparable to ImageNet [10,42,49] pretraining, the resolution of input is 224×224, which is the same as most popular classification methods on ImageNet dataset. We separate input image I into N × N patches of 224 N × 224 N during data preprocessing, as is illustrated in Figure 1(a). We denote the index of each split image patch as i ∈ [0, N 2 ), which is also corresponding to the channel of output heatmap. The size of each image patch is rounded up for the case where N cannot divide 224.
We then apply Knuth-Durstenfeld algorithm [13,23] to randomly shuffle the split image patches, as is illustrated in Figure 1(b). The centre of each shuffled image patch is denoted as p i = (x i , y i ). In order to make the target G of pretext task as similar to 2D human pose estimation as possible, the ground truth heatmap for each output channel i is a 2-dimensional Gaussian Distribution located on each patch centre p i . For each output channel i, the heatmap response at location p = (x , y ) is computed as: where σ represents the standard deviation of 2D Gaussian Distribution, and σ should be less than 224 6N according to 3σ rule to avoid mislocation.
The network learns to predict 2D Gaussian heatmaps to obtain positions of shuffled image patches during the pretraining. We regress target heatmap by the same network as downstream task, except that the number of output channel is equal to N × N : the number of image patches rather than the number of human keypoints. Our best N for downstream performance is 6, which will be tested by ablation studies in Section 5.2.

Learning rich representations
There is obviously a challenging problem for our method proposed in Section 3.1: the centres of target 2D Gaussian Distributions are always located on centres of image grids, such as so the network is easy to fix their responses on centres of grids (as is shown in the first heatmap of Figure 2) rather than having responses anywhere. In this case, it's hard for the network to learn rich representations for downstream tasks, because the learnt responses are highly related to output positions (whether they're grid centres), instead of the contents of image patches.
To tackle with this challenge, we introduce scaling ). In the following three heatmaps, after rotating, rescaling, and translating augmentations in the training stage, the learnt responses are able to appear almost everywhere.
dimensional space of 224 × 224 (except that N is a small number like 1, 2), rather than obtaining responses only on centres of N × N grids like the first column in Figure 2. We don't care whether these spatial data augmentations are able to improve the performance of solving jigsaw puzzles, but they're beneficial to learn more spatial-comprehensive features for downstream tasks.
Apart from spatial augmentations, we follow common colour augmentations in recent papers of self-supervised learning [6,16,17] to improve the diversity of training dataset. Our colour augmentations for training include colour adjustments (hue, contrast, brightness, saturation), colour distortion, optional grayscale conversion, gaussian blur, and solarisation. By the means of colour and spatial augmentations, our HSJP task is able to provide the network with diverse semantics and comprehensive representations.

Loss function for HSJP pretext task
As the form of our pretext task (HSJP) resemble its downstream task of 2D pose estimation -both of them predict some part of a person instance in a heatmap-based way, our loss function for HSJP pretext is similar to recent 2D pose estimation methods [42,49,54]. The output heatmaps from pose estimation networks [42,49,54] are usually 1/4 the size of input images (224×224), and we follow the same settings: the output heatmaps for HSJP pretext task are also downsampled to 56 × 56.
We utilise the same MSE loss function as [42,49] to compare the predicted heatmap H with the groundtruth heatmap G during the training process. If the centre of ith patch is outside the image after spatial augmentation, the channel corresponding to ith patch is not considered into loss function during the training process. The MSE loss is calculated as follows: where G i xy is the groundtruth heatmap response at location (x, y) of channel i, and H i xy is the predicted response. M is a one-hot mask representing whether the centre of ith patch is inside the input image.

Optimiser for HSJP pretraining
Similar to common 2D pose estimation methods [3,19,42,49,54], we employ the same Adam optimiser [22] to train the pretext task. Our models for HSJP are trained for 240 epochs on all person instances of COCO train2017 dataset with 4 Nvidia Quadro RTX 6000 GPUs, and our batch size is 256. The base learning rate is 1e-3, and it is dropped to 1e-4 and 1e-5 at the 190th and 220th epoch.

Performance of solving jigsaw puzzles
To estimate the representation power of CNN after pretraining on HSJP pretext, we evaluate the average precision of jigsaw puzzles on all person instances in COCO val2017 dataset. We split and shuffle image patches as Section 3.1 does, but we don't use any data augmentation when testing the performance of HSJP, because the randomness of testtime augmentation makes the performance unstable. For the ith patch, if the peak of predicted Gaussian Distributionp i lies in the neighbourhood of its corresponding ground-truth patch centre p i , i.e., we consider that the ith image patch is an accurately located one for solving jigsaw puzzles (|| · || 2 represents the L2 norm). If all of the N × N patches in a shuffled image are correctly located, i.e., the jigsaw puzzles problem for that image is considered as successfully solved. We report the average precision of HSJP for all person instances in COCO val2017 dataset, i.e.
where T represent the number of correctly solved jigsaw puzzles across all person instances in COCO val2017, and F is the number of unsuccessfully solved ones. During our pretraining, the indicator P is used to select a powerful network for solving HSJP problem, whose weights are then utilised for initialising downstream 2D pose estimators.  Figure 3: Detailed performance analysis of downstream task, based on HRNet W32 pretrained with different approaches. Both ImageNet and HSJP pretraining perform much better than training from scratch in their downstream task, for all OKS thresholds. HSJP share similar performance with ImageNet pretraining at low OKS thresholds (ImageNet slightly better), while HSJP pretrained HRNet performs much better for 2D human pose estimation at high thresholds (OKS ≥ 0.85).

Experiments
This section is mainly composed of ablation studies and the downstream performances of 2D human pose estimation on MS-COCO [28] val2017 and test-dev2017 datasets.

Dataset and metric for downstream task
The MS-COCO human keypoint dataset includes more than 200k images and 250k person instances, and 150k of them are publicly available for training. We utilise OKSbased Mean Average Precision [28] (mAP score) under 10 different thresholds as evaluation metric on MS-COCO dataset, where OKS calculates the similarities between predicted keypoints and ground truth positions.

Finetuning on 2D human keypoint labels
We pretrain networks with HSJP on all person instances in MS-COCO train2017 dataset. After pretraining, we finetune our learnt weights on human keypoint labels with two popular 2D pose estimators, HRNet [42] and SimpleBaseline [49]. For fair comparison, we follow their training settings. The data augmentations include random rotation ([−45 • , 45 • ]), rescaling ([−35, 35]), and flipping. We also maintain the same number of training epochs as [49,42] under the same Adam optimiser, and the input size of our downstream task is fixed on 256 × 192. The only difference in our experiment is whether we initialise the downstream network with weights pretrained on labelled ImageNet [10] dataset or on our self-supervised HSJP pretext task. For fair   Figure 5: Analysing the transferability of HRNet W32 pretrained with HSJP pretext task. We freeze different layers of HRNet W32 during finetuning and report their mAP for 2D pose, to explore which layers are general or task-specific.
comparison, we also follow the same two-stage paradigm in SimpleBaseline [49], and HRNet [42]: we firstly detect all person instances with Faster-RCNN [35] and then estimate keypoints of detected human bodies. The performances of different pretraining methods and networks are reported in Table 1, which shows that the downstream 2D human pose estimators with our HSJP pretraining achieve much better performance than training from scratch, and the results are comparable to ImageNet pretraining (even a little better than ImageNet for 0.x%).

Detailed analysis of downstream performance
We also analyse detailed downstream performances with different OKS thresholds (from 0.5 to 0.95) on HRNet W32, as Figure 3 shows. It's easy to conclude that pretrained HRNet W32 (with ImageNet or HSJP) achieve much better precision at all OKS thresholds than training from scratch. Comparing the performance of ImageNet with HSJP pretraining in Figure 3(b) and 3(c), we can see that ImageNet pretrained HRNet performs slightly better (like 0.1%) at low OKS thresholds (for OKS ≤ 0. 8 Figure 6: Semi-supervised training with 10% to 90% of MS-COCO human keypoint labels. thresholds (for OKS > 0.8). Although ImageNet and HSJP share similar mAP on 2D human pose estimation, ImageNet pretrained HRNet W32 achieve this mAP score by slightly accumulating improved low-threshold performance, while HSJP pretrained HRNet W32 predicts more accurately at high OKS thresholds for 2D human pose estimation. Also, we use the method in [37] to analyse the error of HRNet W32 under different pretraining methods, as Figure  4 demonstrates, and we can draw a similar conclusion. Both ImageNet and HSJP pretrained HRNet perform much better than training from scratch. HSJP pretrained HRNet W32 enjoys the highest ratio of good prediction, and the lowest ratio for inverting/swapping errors. ImageNet and our HSJP pretraining perform similarly for jitter and missing errors.
Taking the finetuning experiments into consideration, we conclude that HSJP pretrained 2D human pose estimators are able to achieve much better performance than those trained from scratch, and their mAP score on the downstream task are also comparable to ImageNet pretrained networks. However, we don't use any supervised data in our HSJP pretraining, which saves the labour costs of labelling extra datasets (such as ImageNet) for pretraining.

Transferability analysis of different layers
To explore which layers are general or task-specific, we perform similar experiment as [32]: we freeze some layers of HSJP pretrained HRNet W32 during finetuning, to test the transferability of learnt feature maps. Our analyses on 4 layers of HRNet W32 are reported in Figure 5 as follows.
(1) When freezing the top layers (0 to 2) of HSJP pretrained HRNet W32 during finetuning, the mAP for downstream 2D pose estimator does not decrease so much, so the top layers learnt by HSJP pretext are general for extracting image features. (2) When freezing layers 0 to 3 of HSJP pretrained HRNet in the downstream task, the mAP score for 2D pose estimation decrease a lot, but the network does not collapse. Therefore, some features in layer 3 are general for feature extraction, and some features need to be taskspecific for the downstream task. (3) When we freeze all weights in layers 0 to 4 of HRNet W32 learnt by HSJP pretext task, just leaving the last feature fusion layer for final heatmap regression, the downstream performance collapses to mAP of 0. Hence we know that the layers next to the output of network are specific for downstream task.

Semi-supervised training on keypoint labels
The HSJP pretext task is able to build a semi-supervised scheme for 2D human pose estimation: after pretraining backbone network with HSJP pretext task, we fine-tune the HSJP pre-trained network on a randomly selected subset (such as 10%, 20%) of the MS-COCO data set. We do not use any human keypoint labels during pretraining, and we only use partial of 2D human keypoint labels in MS-COCO dataset during finetuning. In this way, our method is semi-supervised. In our experiments for semi-supervised 2D pose estimation, we finetune the HSJP pretrained HR-Net W32 on several subsets (from 10% to 90%) of MS-COCO human keypoint dataset, and we finally demonstrate the performances (mAP for 2D human pose estimation) of our semi-supervised learning in Figure 6.

Ablation Studies
We finally discuss several hyperparameter settings in this section. As we follow the previous settings of 2D human pose estimation [49,42] in the downstream task, our main concern is about hyperparameters in HSJP pretext task. The ablation studies mainly include three parts: (1) learning rate and number of epochs; (2) how many patches (N × N ) should we divide the original input image into; (3) whether using initial (unshuffled) image in HSJP pretext task.

How many epochs under which learning rate?
Our ablation study in this subsection is to explore the best epoch number and learning rate for training HSJP pretext task. We try 3 different learning rates (3e-4, 1e-3, and 2e-3) as well as 3 different epoch numbers (120, 240, and 480) for our HSJP pretext task on HRNet W32, and their performances are reported in Table 2. When training HSJP pretext task for 120 epochs, the network cannot learn rich enough representations, and the downstream task cannot reach its best performance. When training 480 epochs, the precision on HSJP pretext is higher than that of 240 epochs, but it does not help to increase the mAP of downstream task: 2D human pose estimation. We finally select 240 as the number of epochs for training our HSJP pretext task, as it's not only of high performance but also time-saving (Training for 480 epochs is time-consuming, but useless for

How many pieces of image patches?
We need to decide how many patches (the number N ) should we divide the original image into during HSJP pretraining. If the number of image patches is too small (like N = 1, 2), the network will be easy to converge, so it will be hard for the network to learn feature maps with rich representations, as the jigsaw puzzling task is too simple. However, if the number of image patches is too large (such as N > 8), the jigsaw puzzling task is too complicated, and as our resolution of input image is limited to 224 × 224, the pretraining procedure may not be easy to converge.
For this concern, we test different settings of N for our HSJP pretext task with HRNet W32 in this subsection, and finetune networks pretrained with different N on the downstream 2D pose estimator. The HSJP and downstream mAP performances on MS-COCO val2017 dataset are reported with the metrics in Section 3.5 and 4.2. As Table 3(a) shows, with the increment of N , the mAP score for downstream 2D human pose estimator firstly increases, and then decreases. However, the HSJP precision decreases with the increment of N , because the complexity of jigsaw puzzling problem also increases. When N = 6, the convergence and the representation capability of pretext task are able to reach a balance: the pretraining process on HSJP pretext task does not collapse, and the network also learns high-quality representations for downstream 2D pose estimator (although it's of low HSJP precision, our purpose is not to solve jigsaw puzzles more accurately, but to learn representations for downstream task). Therefore, we split our original input image of 224 × 224 into 6 × 6 image patches.

Whether using unshuffled image in pretext?
Similar to [5], we consider concatenating original unshuffled person image to the input of HSJP pretext, as is shown in Figure 7, for two reasons: (1) Taking unshuffled image as input makes the pretraining process similar to the downstream task of 2D human pose estimation, which also takes original person instance as input; (2) Concatenating original image helps to improve the precision of solving HSJP, because it provides a comparison for localising the initial positions of shuffled image patches.
As Figure 3 demonstrates, we compare our performance of pretext (HSJP) and downstream (2D pose estimation) tasks under different settings: whether adding unshuffled image into input. Our experiments are also performed on HRNet W32. Unfortunately, although concatenating unshuffled image benefits the HSJP precision for pretext, it does not improve the performance of pose estimation downstream task: the mAP score is even worse than almost all  weights pretrained without taking unshuffled images as inputs, and the performance is not better than training directly from scratch for 450 epochs. (In our scheme, the pretext HSJP task is trained for 240 epoch, the downstream task is trained for 210 epochs, so we try to train for 240+210=450 epochs from scratch for fair comparison.) Therefore, we do not concatenate the original image into input during HSJP pretraining, even if it seems to be reasonable.

Conclusion
In this paper, we propose self-supervised Heatmap-Style Jigsaw Puzzles (HSJP) problem as pretext task, for the pretraining of 2D human pose estimators. The target of HSJP is to learn the location of shuffled patches with a heatmapbased approach. We only use images of person instances in MS-COCO dataset during our HSJP pretraining, rather than introducing extra and much larger dataset, such as ImageNet. The weights learned by HSJP pretext task are utilised as backbones of 2D human pose estimator, which are then finetuned on MS-COCO human keypoints labels. Our downstream mAP scores are evaluated on two popular and strong 2D human pose estimators, HRNet and Simple-Baseline, and experimental results show that downstream pose estimators with our self-supervised pretraining obtain much better performance than those trained from scratch, and are comparable to those using ImageNet classification models as their initial backbones. However, we don't use any supervised data during our pretraining with HSJP pretext task, which saves the labour costs of labelling extra datasets (such as ImageNet) for pretraining.
For future works, our pretraining scheme with HSJP pretext task has potential to be used on other heatmap-based or pixel-labelling tasks, such as face keypoint detection, crowd counting, and semantic segmentation.

Supplement Material for Paper Learning Heatmap-Style Jigsaw Puzzles
Provides Good Pretraining for 2D Human Pose Estimation

Human Keypoint Priors in Jigsaw Puzzles
We have mentioned in the article that there are positional priors for relative locations of human keypoints in person instances, which is an important reason why we consider solving self-supervised jigsaw puzzles as pretext task can provide efficient pretraining for 2D human pose estimation. In this section, we will elaborate on priors of human keypoints in jigsaw puzzles by statistical means.

Method for Statistical Analysis
In order to illustrate that the distributions of human keypoints have positional priors (i.e., some human keypoints are mainly distributed in certain image patches), we count the percentage of different image patches for each human keypoint in MSCOCO dataset [1]. To make our statistical results simple and easy for understanding, we take N = 3 as example for counting experiment in this section. Firstly, we divide each person instance in MSCOCO into three parts along both vertical and horizontal directions (i.e., 3 × 3 = 9 image patches in total), as Figure 1(a) in the main text demonstrates. Then, we calculate the probability that 17 different human keypoints falling within these 3×3 patches.

Statistical Results
MSCOCO keypoint annotations are with 17 human keypoints, 5 of which are annotated for the head (eyes, ears, nose), 6 are for upper limbs (shoulders, elbows, wrists), and 6 are for lower limbs (hips, knees, ankles). In order to make our statistical report clear, we note the 9 patches with their relative locations in person instances: top-left, top, top-right, left, centre, right, bottom-left, bottom, and bottom-right. According to statistical analyses, different human keypoints usually have their own priors of relative positions in person instances, and details for human keypoint priors are in the following part of this subsection..  Figure 1. Most of the head keypoints are located in the top parts of person instances (top takes the largest percentage, while topleft/top-right also count for some ratios). Based on this statistical result, we believe the following statement: if the network is able to distinguish whether each patch is in the top of the person instance in pretext task, the learnt features are also beneficial for predicting keypoints on the head.
(2) Keypoint Priors for Lower Limbs. The percentages of keypoints on lower limbs are shown in Figure 2. Similar to the distribution of head keypoints, the keypoints of lower limbs are also mainly distributed in several image patches, and the other patches count for only a small percentage in total. The hips are mainly near the centre of person instances, while knees and ankles are often near the bottom (i.e., bottom, bottomleft, or bottom-right) of person instances. Therefore, we consider that learning to classify centre patches in pretext task helps to locate hips in downstream human keypoint estimator, and learning the relative positions of bottom patches helps to locate knees and ankles.
(3) Keypoint Priors for Upper Limbs. Statistical priors for human keypoints on upper limbs are illustrated in Figure 3. Left/right shoulders are frequently located in the neighbourhoods of corresponding top-left/topright patches. Apart from that, the left/right patches account for the largest percentages in the distributions of elbows/wrists, and their percentages are more than 10% higher than the second place. If the pretrained networks know how to distinguish the corresponding image patches in pretext task, it's helpful for localising keypoints on upper limbs in the downstream task.

Discussion for More Patches
We have expounded that different human keypoints have their own positional priors, and learning these priors by solving jigsaw puzzles problem is beneficial to the downstream pose estimator. We use N = 3 in this supplement material just to facilitate visualization and understanding. In practical experiments, N can be any positive integer, and the position of each key point may be related to the priors of several image patches when N is a larger number. The downstream 2D pose estimator performs best when N = 6.