{'fraction_non_alphanumeric': 0.028253424657534245, 'fraction_numerical': 0.011643835616438357, 'mean_word_length': 4.771245059288537, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Video Motion Capture from the Part Confidence Maps of Multi-Camera Images by Spatiotemporal Filtering Using the Human Skeletal Model

This paper discusses video motion capture, namely, 3D reconstruction of human motion from multi-camera images. After the Part Confidence Maps are computed from each camera image, the proposed spatiotemporal filter is applied to deliver the human motion data with accuracy and smoothness for human motion analysis. The spatiotemporal filter uses the human skeleton and mixes temporal smoothing in two-time inverse kinematics computations. The experimental results show that the mean per joint position error was 26.1mm for regular motions and 38.8mm for inverted motions.


I. INTRODUCTION
Motion capture technology is widely used for many applications, in particular with human motion, such as animation, video game production, sports training, biomechanical analysis, rehabilitation, medical diagnosis, and research of human motion modeling and recognition [1][2] [3].
The technology has three typical methods for measurements, namely, passive optical, active optical, and gyroaccelerometer measurements. The spatiotemporal accuracy is approximately 1mm and 1kHz for the typical human scale environments in the case of the high-end passive optical system [4] [5], and getting even higher by the advancement of imaging device technology. The passive optical method imposes the least constraint to a human subject, but still needs approximately 40 spherical reflective markers attached on the body. Other methods also have similar or even stronger constraints.
To solve this problem, image-based motion capture which imposes no constraint to the human subjects is demanded. The studies on RGB-D vision show promising results for human motion recognition using Kinect [6] [7] and the other systems with laser projection or scanning depth sensors [8] [9]. Recent studies on single RGB vision [10] [11][12] [13] show remarkable results where, taking an advantage of the large RGB image data set and the power of deep neural networks, the learning based methods for human motion recognition have been developed. Although image-based human motion recognition is considered a 2D problem and rather tolerant in spatial accuracy and temporal discontinuity, motion capture is a 3D reconstruction problem and requires spatiotemporal accuracy and smoothness for velocity and acceleration analysis.
In this paper, we report about our study on video motion capture, which means image-based 3D human motion reconstruction with spatiotemporal accuracy and smoothness closer to passive optical motion capture. We use synchronized multiple cameras to record video images of a human subject from different directions. The likelihood of joint positions are computed as the Part Confidence Maps (PCM) [14] [15] in each video image and provided by the colors of heatmap with the labels of individual joints by OpenPose [16]. Our initial study showed that the simple least square solution of the 3D position of each joint from the multi-camera images neither resulted in smooth 3D joint movements nor maintained link length between adjacent joints.
We propose to use the human skeletal model for reconstructing 3D human motion by spatiotemporal filtering of joint movements. We also discuss to enhance function of the PCM for rather difficult human postures. The proposed video motion capture was implemented and tested through various experiments. The results of evaluation are shown based on the reconstructions made by two methods, namely, the video motion capture and the passive optical motion capture.

II. VIDEO MOTION CAPTURE FROM MULTI-CAMERA IMAGES A. Flow of the Video Motion Capture
Proposed video motion capture can be described as the iterative update of human skeletal model's joint angles based on multi-camera images which surround a human subject like Fig.1. Note that in this paper the skeletal model means a virtual open tree-structure kinematic chain with 40 degrees of freedom (DOF) as shown in Fig.2 [a]. To fulfil this update, the weighted markers which are virtually generated in the 3D space are computed as shown in Fig.2 [b] based on the subject's probable joint positions. By minimizing the errors of virtual markers' positions and skeletal model's corresponding joint positions, the skeletal model's joint positions at the time are obtained. To solve this minimisation problem, inverse kinematics (IK) computation is adapted, and by repeating above process, the video motion capture is realized.
The flow chart of the method is shown in Fig.3. Note that the camera calibration must be done before the computation.
The camera parameters such as the focal length, the center of image, and the lens distortion enable restoration of distorted images and provide the transformation matrices P i that maps an arbitrary 3D point to the corresponding pixel location on the ith camera image plane [17] [18]. Other main blocks in the chart are explained in the following subsections.

B. Computation of the Part Confidence Maps
The OpenPose [16] provides the Part Confidence Maps (PCM) [14] and the Part Affinity Fields (PAF) [15] of multi-person's keypoints on a 2D image in real time using the trained Convolutional Neural Networks (CNN). The keypoints are 18 consisting of joint positions like shoulders, elbows, wrists, hips, knees, ankles and a neck, and the other feature points like eyes, ears, and a nose. Though the recent version of OpenPose includes hands, feet, and face, we used only the above 18 keypoints.
The PCM shows by its temperature the distribution of likelihood of the keypoint, namely, in each pixel location, the probability of the keypoint is expressed as the continuous   In order to compute IK, the skeletal model's adjacent joints must be connected by a constant-length link and the link length must be calculated according to the human subject. Furthermore, since proposed motion capture is based on iterative computation, it is reasonable to calculate the skeletal model's initial joint positions before the computation.
The link lengths of the skeletal model are identified for the subject by using the centroids of PCM on the images from multiple cameras as schematically shown in Fig.5. Initialization is confirmed when the centroids of every keypoint from the cameras show a 3D agreement. And simultaneously, skeletal model's initial joint positions are calculated.

Camera i
Camera j

D. Searching Probable Joint Positions
Let t J n and t Q n represent the 3D joint position and the joint angle of nth joint at t time-frame. We discretize the 3D space with a unit distance s, and consider the lattice of (2k + 1) × (2k + 1) × (2k + 1) with t J n at the center. We then search for t+1 J n on the lattice. The search space t J n is represented by A point on the lattice in t J n is described as t J n a,b,c . It is considered that the point with the highest score from summation of multi-camera PCM values is the most probable point of the joint existence. Let n c be the number of cameras and the function t+1 S n i be the PCM value that of a corresponding pixel positionof nth joint on ith camera image at t + 1. Then, the predicted joint position at t + 1 can be calculated as follows: At this predicted joint position, virtual marker is generated.

E. Computing Inverse Kinematics
Since the joint position t+1 J n is a function of joint angle t+1 Q expressed in forward kinematics (FK), t+1 Q can be obtained by solving following optimization problem [19]: Note that n j is the number of joints and the constant weight t+1 W n is obtained from the PCM score at the predicted joint position as follows.

F. Smoothing Joint Positions with Maintaining Link Length
Human movement is smooth and differentiable by time, but acquisition of PCM and IK computation does not use time-series data, so the output joint data is not to be smoothed. To solve this problem, joint position is smoothed by introducing IIR low-pass filter. However, the time-invariant parameters of skeletal model, link length, is changed every time-frame by the smoothing.
In proposed method, the position of the joint after the adaptation of the low-pass filter is set as the virtual marker of each joint, and the optimization by IK computation is performed again. This makes it possible to make use of the smoothness of the temporal change of the joint position under the skeletal condition that each link length is invariant.

A. Comparison with Optical Motion Capture
Human subject's motion was synchronously recorded by 4 RGB cameras (FLIR Corporation, Grasshopper3) at 60Hz with a 1024×768 pixel resolution and measured by the optical motion capture system with 17 infra-red cameras (Motion Analysis Corporation, Eagle and Raptor-4) at 200Hz with 41 markers. Fig.6 shows the positions of the attached markers. Ground truth of joint positions was calculated by these measured markers.
In this experiment, 9 motions were measured; Walk, Dance, Spin Jump, Football Juggling, Golf, Matrix, Handstand, Cartwheel, and Break Dance.   Note that PCM score is the value of t+1 W n defined in Eq.4. In every frame, proposed method achieves 25mm in MPJPE and PCM score keeps high. However, in inverted motions such as Handstand and Cartwheel, proposed method estimates the joint positions with larger errors, especially in Lowerbody. Fig.9 shows MPJPE & PCM score change in Handstand motion which is the largest in MPJPE. As the body tilts, MPJPE became larger and PCM score getting worse.  This problem is caused by the accuracy of PCM. Since OpenPose was created by supervised learning, PCM computation is greatly affected by the training data, and it can be considered that training data did not contain a sufficient amount of inverted posture. Fig.10 shows the PCM change by the input image rotation. By inputting rotated image according to the tilt of the human subject, better PCM can be computed.   From the skeletal model's neck and hip joint position at t frame, the tilt of trunk at t is calculated by each camera image. According to the tilt, input image at t + 1 is rotated and then LowerBody's PCM are computed. PCM of the others are computed from normal input image. Using these PCM, virtual markers in 3D space are generated and IK computation is performed illustrated as above.
The results of incorporating the input image rotation method into Handstand, Cartwheel, Break Dance are shown as follows. Fig.11 shows that joint position estimation became possible even in inverted motion such as Handstand & Cartwheel due to the input image rotation. However, in Break Dance, no significant change appeared. This can be attributed to the difference between whether the axis of rotation of each motion is perpendicular or parallel to the ground.  The conclusion of this paper is as follows: 1) An approach of video motion capture was proposed towards image-based 3D human motion reconstruction from multi-camera images using the Part Confidence Maps. 2) A method of spatiotemporal filtering was developed using the human skeletal model for the 3D reconstruction of human motion by mixing temporal smoothing in between two-time inverse kinematic resolutions.
3) The accuracy of the Part Confidence Map was improved using the rotated images for inverted motions such as a handstand and a cartwheel. 4) The experimental results of video motion capture showed that the mean per joint position error was 26.1mm for regular motions and 38.8mm for inverted motions. When the threshold was set at 100mm as the maximum error of all joint positions for the whole motion, the success rate was 99.8 % for regular motions and 96.4 % for inverted motions.