{'fraction_non_alphanumeric': 0.03006976184748617, 'fraction_numerical': 0.014722155400529228, 'mean_word_length': 4.972988505747127, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Lightweight Human Pose Estimation Using Heatmap-Weighting Loss

Recent research on human pose estimation exploits complex structures to improve performance on benchmark datasets, ignoring the resource overhead and inference speed when the model is actually deployed. In this paper, we lighten the computation cost and parameters of the deconvolution head network in SimpleBaseline and introduce an attention mechanism that utilizes original, inter-level, and intra-level information to intensify the accuracy. Additionally, we propose a novel loss function called heatmap weighting loss, which generates weights for each pixel on the heatmap that makes the model more focused on keypoints. Experiments demonstrate our method achieves a balance between performance, resource volume, and inference speed. Specifically, our method can achieve 65.3 AP score on COCO test-dev, while the inference speed is 55 FPS and 18 FPS on the mobile GPU and CPU, respectively.


I. INTRODUCTION
Human pose estimation aims to locate the body joints from image or video data. It has received increasing attention from the computer vision community in the past couple of years and has been utilized in various applications such as animation [1], sports [2], [3], and healthcare [4]. Due to the development of deep convolutional neural networks, recent works achieve significant progress on this topic [5]- [8].
In order to get higher accuracy, many existing methods usually comprise wide and deep network architectures because they could bring powerful representation capacity. This leads to tremendous parameters and a huge number of floating-point operations (FLOPs). However, the most common operating environments for pose estimation applications are resourceconstrained edge devices such as smartphones or robots. The incremental computing cost and parameter memory are crucial barriers to the deployment of pose estimation.
There is also some work focus on the lightweight human pose estimation method. Lite-HRNet [9] follows High-Resolution Net (HRNet) [8] and reduces its parameters and computation cost. Lite-HRNet achieves extremely low parameters and FLOPs, however, the theoretical lightweight does not always mean fast in practice. The parallelism of the multi-branch structure does not reach the theoretical efficiency and the cross-resolution weight computation also slows down the inference of the model. It is even slower than the HR-Net during actual running. Benefit from network architecture search (NAS) technology, ViPNAS [10] reaches a notable balance between accuracy and speed, but its performance while handling picture tasks on the CPU is not satisfactory.
Other methods are short in terms of parameter quantity [11]. Research in the lightweight human pose estimation area is still inadequate.
The SimpleBaseline [6] is an elegant and effective method for human pose estimation, they provide the capacity of deconvolution layers in this problem. Based on their successful work, we focus on the design of lightweight and simple network architecture for human pose estimation. First, we use a lightweight network MobileNetV3 [12] as the backbone in our model, rather than the huge ResNet [13]. As the latest model in MobileNet series [12], [14], [15] MobileNetV3 has shown its performance in many computer vision tasks. Then we analyse the parameters and computation cost in the whole model and notice the deconvolution layers contain the dominant resources. Inspired by the design of the MobileNet series, we change the normal deconvolution layers to a depthwise deconvolution layer following a pointwise convolution layer. To compensate for the performance degradation caused by lightweighting, we introduce an attention mechanism. The attention block utilizes channel, spatial and original information of the input feature map to enhance the performance. Besides, we observe that Mean Square Error (MSE) loss is widely used by existing methods during the training stage. MSE loss just calculates the average error for different pixels on the heatmaps, without reflecting the inhomogenous from different locations. We propose a novel loss function called heatmap weighting loss to generate weights for every pixel on heatmaps that is used to calculate the final error.
In short, our contribution can be summarized as followed.
• We simplify the deconvolution head network in Simple-Baseline and present an attention mechanism that exploits channel, spatial and global representations to benefit the performance of the lightweight architecture. • We propose a novel loss function -heatmap weighting loss that explores the information from ground truth heatmaps to improve keypoints localization.

A. Human Pose Estimation
Human posture estimation has been an active research problem in the last decades. Before the rise of deep learning, human pose estimation was mainly based on graph structure models, using picture global features or handcraft filters to detect and localize the keypoints of human body [16]. Since Toshev et al. [17] propose DeepPose, which uses the deep neural network to regress the keypoints locations, the neural network based approach becomes mainstream. Newell et al. [5] propose Hourglass, which consists of multi-stage stacked hourglass networks, it is a milestone on the MPII [18] benchmark dataset. Chen et al. [7] propose Cascaded Pyramid Network (CPN), which is the champion of COCO 2017 keypoint challenge. CPN uses a GlobalNet [7] to estimate simple keypoints and handle complex keypoints with a RefineNet [7]. Xiao et al. [6] propose SimpleBaseline, which is a simple but effective method composed of a backbone and several deconvolution layers. Sun et al. [8] present a novel High-Resolution Net to maintain high-resolution representation by connecting multiresolution convolutions in parallel and conducting repeated multi-scale feature map fusion. Cai et al. [19] introduce Residual Steps Network (RSN) to learn more delicate local feature by intra-level information fusion, and propose Pose Refine Machine (PRM) to refine feature map by attention mechanism. This method is the winner of COCO 2019 keypoint challenge.

B. Lightweight Network Design
Design deep neural network architecture for achieving a trade-off between efficiency and accuracy has been an active research topic, especially in industrial research. In recent years, a large number of compact network architectures have been proposed. Xception [20] uses depthwise separable convolution operation to reduce parameters and computation. MobileNets series [12], [14], [15] are also based on depthwise convolution and pointwise convolution. MobileNetV2 [15] present inverted residual block and MobileNetV3 [12] exploit NAS method searching a optimal structure that has better performance and fewer FLOPs. ShuffleNet [21] proposes a channel shuffle operation for information interaction between feature map groups. ShuffleNetV2 [22] studies the lightweight network design principles and modifies the ShuffleNet according to the proposed guidelines. GhostNet [23] investigates the correlation and redundancy between feature maps and presents a cheap operation to reduce the computation of pointwise convolution. The development of lightweight networks has great significance for our work, on the one hand, we can directly use a more powerful lightweight network as a backbone in our model, on the other hand, we can draw on the idea of compact network design in our downstream tasks.

C. Attention Mechanism
Attention mechanism has shown powerful performance in many computer vision tasks such as image classification, scene segmentation, objection detection, and optical character recognition (OCR). Wang et al. [24] propose Non-Local Networks which introduce self-attention to capture long-range dependencies. Hu et al. [25] present Squeeze-and-Excitation Networks (SENet), using weight learned from global information to excite or suppress different channels, which is the champion of ILSVRC 2017. Li et al. [26] expand SENet and propose Selective Kernel Networks (SKNet), using feature maps generated by multiple kernel sizes to gather attention value. Woo et al. [27] propose Convolutional Block Attention Module (CBAM) to exploit both channel and spatial information to refine the feature map. Most attention modules can be easily embedded in our pose estimation network without introducing too much extra computation while improving the performance of the network.

III. METHOD A. Lightweight Deconvolution Head Network
Deconvolution head network is a different method to rehabilitate high-resolution feature map which is introduced in SimpleBaseline. Before that, upsampling is the dominant technology to recover high-resolution feature map [5], [7], and some convolution layers added to adjust the generated feature map. In the deconvolution head network, a deconvolution layer that could learn the upsampling parameters from input data during training combines upsampling and convolution parameters in a single operation.
The standard deconvolution layer is parameterized by the convolution kernel K of size D K × D K × C in × C out , where D K is the spatial dimension of the kernel assumed to be square and C in is the number of input channels, C out is the number of output channels, and l is the number of deconvolution layers. Despite the first layer for channel compression and the last layer for generating output feature map, the total number of parameters in a deconvolution head network is: The computational cost of a deconvolution layer multiplicatively depends on the size of kernel D K × D K , the number of input channels C in , the number of output channels C out , and the size of feature maps W × H. Therefore, the total computation in a l layer deconvolution head network is: Based on the design in MobileNets, we replace the standard deconvolution layer with a depthwise deconvolution layer following a pointwise convolution layer. Fig. 1 illustrates the structure of our lightweight deconvolution head network. Depthwise deconvolution applies a single filter on each channel for upsampling and producing the new representations in high-resolution feature map, while pointwise convolution provides a bridge for information fusion across channels. The influence of the output channels on the parameters and computation is eliminated in depthwise deconvolution, and in pointwise convolution, the kernel size will no longer be a factor. The computation cost of our new deconvolution head network can be written as: The number of parameters is proportional to the computation cost, just remove the size of feature map. Following the setting in SimpleBaseline, we set the layers to 3, kernel size to 4, and the number of channels to 256 in all layers. Thus we can get a reduction in computation cost of:

B. Attention Mechanism
To improve the capacity of our lightweight deconvolution head network, we introduce an attention mechanism. Motivated by the work of SENet, we use an adaptive average pooling to aggregate spatial information in each channel. After generating channel representations, two pointwise convolution layers are used to calculate the correlation between channels. To reduce the parameters, the input feature dimension C is squeezed to C/r after the first convolution, where r is the squeeze ratio. Before finally broadcasting along spatial dimension and multiplying with the original feature map, a hard sigmoid function is utilized to normalize the weight. Define the feature map as f , a convolution layer with p output channels and kernel size of m×n as F m×n p , the hard sigmoid as σ(·), the global average pooling as GAP(·), and the elementwise multiplication as . The channel attention H channel (·) is computed as: Since human pose estimation is a space-sensitive task, a spatial attention mechanism can help to better mine the information in spatial dimension. To converge information in different channels, we apply a pointwise convolution to get a linearly combined representation. After that, a standard convolution is used to produce the spatial attention map. A hard sigmoid layer is also used to rescale the activation values. Following the definition of the channel portion, the spatial attention H spatial (·) is written as: For the arrangement of attention modules, we use a parallel structure, as shown in Fig. 2. Channel attention and spatial attention are applied to the feature map individually and an identity connection is also employed. The identity mapping protects local features from being corrupted by dimensional compression in the attention block. Meanwhile, channel attention exploits inter-level representation to benefit keypoints identification and spatial attention improves keypoints by optimizing spatial information. Finally, the feature maps in three branches are added to get the output feature. Our attention mechanism can be formulated as:

C. Heatmap Weighting Loss
In the training phase, MSE is commonly used as the loss function [5]- [8]. Beyond the previous work, we argue that the weight of the pixels at different locations on the heatmaps should be various. The loss function should focus more on the points with larger values on the heatmaps, which means devoting more attention to those pixels that are closer to the keypoints. Based on this thinking, we propose heatmap weighting loss which generates weight for each pixel by adopting the transformation function on the ground truth heatmap, as shown in Fig. 3. Our loss function presents as: where J is the number of keypoints, P and P GT are predicted heatmap and ground truth heatmap respectively, F(·) is weight generation function, and denotes element-wise multiplication.
For the selection of weight generation function, in consideration of the target heatmaps are usually sampled from a two-dimensional Gaussian distribution [28]. We select several convex functions on [0, 1] interval such as linear function, power function, and exponential function to generate the weights and evaluate their performance. See Section IV-C for details.

IV. EXPERIMENTS
We evaluate our method on COCO [29] and MPII [18] datasets. Following the two-stage top-down paradigm [6], [8], our approach estimates keypoints confidence maps from the detected person boundary box. We report the comparisons with other methods and also conduct inference speed test and ablation studies on these datasets. The project page 1 has been set up with programs available.
A. Experimental Setup 1) Dataset and metric: COCO dataset contains over 200K images and 250K person instances with 17 keypoints. Our model is trained on the train2017 set (includes 57K images and 150K person instances) and validated on the val2017 set (includes 5K images). MPII includes around 25K images containing over 40K people with annotated body joints. We adopt object keypoint similarity (OKS) based mean average precision (AP) to evaluate the performance on COCO, for MPII, the standard PCKh (head-normalized percentage of correct keypoints) is used.
2) Image cropping: The human box is expanded in height or width to reach a fixed aspect ratio, e.g., height:width=4:3. Then we crop the box area from the image and finally resize it to a certain resolution such as 256 × 192 for COCO and 256 × 256 for MPII.
3) Data augmentation: We apply a series of data augmentation operations to make the model learn scale invariance and rotation invariance. Augmentation strategy includes flip, random rotation ([−80 • , 80 • ]), random scale ([0.5, 1.5]), and half body augmentation which keeps only the upper or the lower body at random for COCO.

4) Training:
We use Adam optimizer mini-batch size 64 to train our network, the initial learning rate is 5e-4 and reduced by a factor of 10 at the 170th and 200th epoch. The training process is terminated within 210 epochs. The warm-up strategy with a liner warm ratio of 0.001 is used in the beginning 500 iterators [30]. Our experiments are conducted on a single NVIDIA 3090 GPU. 5) Testing: In order to minimize the variance of prediction, following the normal practice [5], we average the heatmaps predicted from the original and flipped images. A quarter of a pixel offset in the direction from the highest response to the second-highest response is applied before the predicted keypoint locations transform back to the original coordinate space of the input image. Table I reports the comparison results of our method and other state-of-the-art methods. Our method, trained from input resolution with 256 × 192, achieves a 65.8 AP score. Compared to ShuffleNetV2, our method improves AP by 5.9 points with 42% FLOPs. We use about 35% of the parameters and computational cost while obtaining a gain of 1.2 AP points compared to MobileNetV2. Our method matches the performance of SimpleBaseline using MobileNetV3 as the backbone, with only 35% FLOPs and 39% parameters. Compared to Small HRNet-W16 [9], we improve 10.6 AP score. Our method also outperforms the Lite-HRNet-18 with 1.0 AP points. There are little gaps (1.3 points and 2.0 points) between our approach and the state-of-the-art methods, Lite-HRNet-30 and ViPNAS, respectively, but our method achieves a much higher inference speed. See Section IV-D for details. It is worth noting that the goal of this work is to introduce a simple lightweight human pose estimation network with high inference speed rather than improving the score of existing SOTA approache on benchmark datasets. For the input size of 384 × 288, our method achieves 69.9 AP score. Some prediction results are visualized in Fig. 4.

1) COCO val:
2) COCO test-dev: Table II reports the comparison results of our models and other lightweight methods. Our networks achieve 65.3 and 69.2 with the input size of 256 × 192 and 384 × 288, respectively. Our approach outperforms other methods except for Lite-HRNet-30 in AP scores, and our method is closer to SOTA with larger input size.
3) MPII val: Comprehensive results on MPII are reported in Table 3. Our method achieves 85.9 PCKh with a standard 256 × 256 input size, outperforms the ShuffleNetV2, Mo-bileNetV2 by 3.1 and 0.5 points, respectively.

C. Weight Generation Function
We experimentally study the influence of heatmap weighting loss. We evaluate different weight generation functions and compare them with the traditional MSE loss. Table IV reports the results on COCO val with the input size of 256 × 192.
We can find that the AP score got the most significant gain of 0.27 points when using the most simple endomorphism as the weight generation function. When we increase the transformation ratio in linear mapping, the gain becomes negligible (only 0.03 AP score). We also attempt the square and exponential functions that obtain gains of 0.09 and 0.14 points, respectively. Although the gain from our heatmap weighting loss function is not significant now, it provides a new optimization idea for human pose estimation. The notion of our heatmap weighting loss can be considered as a kind of adjustment of the handcrafted two-dimensional Gaussian distribution heatmap to make it approximate the actual distribution of the human keypoint.

D. Inference Speed
Operations (OPs) denote the number of computations required for the forward inference of the model and reflect the demand for hardware computational units. While float is the most commonly used data type, FLOPs becomes the general reference for evaluating model size. The actual inference speed on hardware is a complex issue. It is not only affected by   Inference speed experiments demonstrate that our method is more friendly to the mobile device and better suited to realworld applications.

E. Ablation Study
In this part, we validate the improvement of our attention architecture by performing ablation experiments on COCO val set with 256 × 192 input size, as shown in Table 7. We  compare the prior attention mechanism SE-Block, CBAM, and our attention module. Our approach outperforms the other two modules in AP score, which implies trimming the common attention block designed for image classification before applying it to a certain task is necessary.

V. CONCLUSION
In this paper, we propose a lightweight human pose estimation network with high inference speed by simplifying the deconvolution head network in SimpleBaseline and applying an attention mechanism that exploits channel, spatial and global representations to improve the performance. We also present a novel loss function, heatmap weighting loss, which explores the information from ground truth heatmaps to benefit keypoints localization. Our method achieves a balance between performance, resource volume, and inference speed. We hope that this work gives a new idea for lightweight human pose estimation.