{'fraction_non_alphanumeric': 0.033476060724017126, 'fraction_numerical': 0.012110202845897668, 'mean_word_length': 4.68666994589277, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging

Hand modeling is critical for immersive VR/AR, action understanding, or human healthcare. Existing parametric models account only for hand shape, pose, or texture, without modeling the anatomical attributes like bone, which is essential for realistic hand biomechanics analysis. In this paper, we present PIANO, the first parametric bone model of human hands from MRI data. Our PIANO model is biologically correct, simple to animate, and differentiable, achieving more anatomically precise modeling of the inner hand kinematic structure in a data-driven manner than the traditional hand models based on the outer surface only. Furthermore, our PIANO model can be applied in neural network layers to enable training with a fine-grained semantic loss, which opens up the new task of data-driven fine-grained hand bone anatomic and semantic understanding from MRI or even RGB images. We make our model publicly available.


Introduction
Hands are essential for humans to interact with the world and hand modeling enables numerous applications for immersive VR/AR, action analysis and healthcare, etc. However, reconstructing realistic human hands in a biologically and anatomically plausible manner remains challenging and has received substantive attention in both the artificial intelligence and computer vision communities.
In recent years, there has been significant progress in hand pose, shape, geometry Smith et al., 2020], and even texture estimation  in a data-driven manner with the corresponding parametric models [Romero et al., 2017;Moon et al., 2020]. In contrast, there is little work that addresses the parametric modeling and data-driven estimation of human hand anatomy like bone and muscle. However, such fine-grained hand anatomical attribute modeling is important for understanding the complex hand biomechanics and producing realistic and anatomically precise virtual hands. On the other hand, modern 3D medical imaging techniques like Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) are widely used for treating hand injuries clinically [Kapandji and others, Shape Pose Figure 1: We present the first statistical internal hand bone model. Our model enables anatomically and physically precise modeling of the hand kinematic structure of different individuals, opening up the study of data-driven hand bone anatomic and semantic fine-grained understanding from MRI or even RGB images. 2007] or modeling the function of healthy hands [Miyata et al., 2005]. However, the above-mentioned modeling methods focus on the analysis of only a single individual. Moreover, those CT-based methods [Kurihara and Miyata, 2004;Yang et al., 2020] suffer from ethical issues due to the unacceptably high radiation doses during the scanning of various individuals under various hand poses for building a highquality parametric hand bone model. The recent MRI-based method [Wang et al., 2019] adopts a molding procedure to build an anatomy-correct hand bone rig of a target performer. However, it suffers from time-consuming and user-specified bone segmentation operations, which is impractical to apply to various individuals for building a parametric model.
To address the gaps above, we present the first statistical hand bone model from MRI data called PIANO for Para-metrIc hANd bOne model, which enables anatomically and physically precise modeling of the hand kinematic structure of different individuals (see Fig. 1). Our model is biologically correct, simple to animate, and differentiable, opening up the study of data-driven hand bone anatomic and semantic fine-grained understanding from MRI or even RGB images at a level of detail not previously possible.
To this end, we first collect a new large-scale dataset of high-quality MRI hand scans using a clinical 3T scanner and an efficient simplified molding procedure from [Wang et al., 2019], which consists of 200 scans of 35 subjects in up to 50 poses with rich and accurate 3D joint and bone segmentation annotations from experienced radiologists. Then, we use this data as well as an artist-rigged hand bone template mesh with 20 separated bones to train the statistical hand bone model PIANO similar to the outer human body surface model SMPL [Loper et al., 2015] and hand model MANO [Romero et al., 2017]. Due to the inherent per-bone rigidness in PI-ANO, we omit the pose blend shape and adopt a pre-defined blending weight. Thus, PIANO inherently factors the perbone anatomic geometric changes of each individual and has analogous components to those in SMPL and MANO: a hand bone template mesh with kinematic tree, shape blend shape, and a joint regressor. Especially, to build PIANO in an anatomically precise manner, we introduce a novel coarseto-fine multi-stage registration approach and a bone-aware model learning strategy, which jointly considers the accurate joint and bone segmentation annotations provided by radiologists. Our new parametric hand bone model allows obtaining more anatomically precise personalized rigging than the traditional hand models based on outer surface only. Furthermore, we introduce a novel neural network based on PIANO to produce fine-grained and biologically correct hand bone segmentation from an MRI scan or even a single RGB image input in an analysis-by-synthesis fashion. It enables a finegrained semantic loss and opens up a new task for data-driven internal hand bone modeling unseen before. To summarize, our main technical contributions include: • We present the first parametric hand bone model, PI-ANO, based on a new MRI hand dataset, which enables anatomically precise internal kinematic understanding of human hands in a data-driven manner. • We propose a novel multi-stage registration approach as well as a bone-aware model learning strategy, which utilizes the joint and bone segmentation annotations in MRI data provided by radiologists to build PIANO. • We present a proof-of-concept neural network design that enables data-driven fine-grained hand bone segmentation from an MRI scan or RGB image. • We make available our PIANO model and the new dataset of 200 MRI hand scans of 35 subjects with various poses and rich annotations 1 .  Miyata, 2004] and [Marai et al., 2003] have used CT scans to acquire hand bone skeleton and carpal movements. [Martín-Fernández et al., 2009] proposed automatic articulated registration of hands out of X-ray images. However, CT and X-ray emits ionizing radiation, while MRI is safer and provides more clear contrast between soft body tissue. [Rhee et al., 2007;Miyata et al., 2005] used MRI for understanding the hand anatomy and biomechanics on a small set of poses. Recently, [Wang et al., 2019] adopts a molding procedure for long time MRI scan, enabling anatomical hand bone analysis with complex poses. However, suffering from time-consuming and user-specified operations, these methods only focus on single individuals. Comparably, we employ a simplified molding procedure and a semi-automatic annotation method to build a large-scale dataset of high-quality MRI hand scans with precise annotation on bone and joints.

Parametric Hand Bone Model
In this work, our goal is to build a parametric internal bone model of human hands, which is anatomically correct, simple to animate and differentiable. First, we build a new MRI hand dataset via an efficient molding procedure, which captures precise anatomical structure of human hands with rich annotations provided by experienced radiologists (Sec. 3.1). Then, similar to outer surface modeling, we train our PIANO model to obtain the hand bone template mesh with kinematic tree, shape blend shape and joint regressor by iterating between model registration and model parameter learning in a flip-flop manner. Our detailed model formulation is provided in Sec. 3.2, followed by the illustration of our novel multistage model registration (Sec. 3.3) and our bone-aware parameter learning scheme (Sec. 3.4), so as to utilize the rich annotations in MRI data provided by radiologists.

Hand MRI Data Collection
Magnetic Resonance Imaging (MRI) scanning is performed to capture the anatomical structure under each hand pose. In all, a total of 200 hand volumetric images are captured with 50 different hand posture of 35 individual subjects. The scanning is performed with a 3.0 Tesla MR scanner of United Imaging (uMR780, Shanghai) with spatial resolution 0.4 × 0.4 × 1 mm 3 and T1-weighted sequence. Since MRI is a motion-sensitive imaging technique, during scanning, we adopt a simplified molding procedure to stabilize hand poses similar as that in [Wang et al., 2019]. Briefly, we put hands in a liquid solution which would solidify and encase the hand firmly in less than 3 minutes. We then scan the hand with the mold on for 10 minutes. The MRI volume is a regular volumetric grid of scalar values representing tissue mass density. To analyze anatomical structure, bone and joint location must be annotated. How- ever, this task is highly cumbersome and requires strong domain knowledge. [Wang et al., 2019] proposed an interactive method that takes 3 hours to segment one volume. However, to label all 200 scans in our dataset would take up to 600 hours. To this end, we adopt a semi-automated human-inthe-loop procedure to label all volumes with a deep neural network. By verifying network predictions and annotating miss labeled samples in an iterative procedure, we only acquire one-third of volume annotations with a moderate manual effort by experienced radiologists. Dataset samples are shown in Fig.2.

Model Formulation
To learn an anatomically correct and simple-to-animate statistical model from our MRI dataset, we follow the outer surface model SMPL [Loper et al., 2015] and MANO [Romero et al., 2017] to formulate our PIANO model into analogous components, including a hand bone template mesh with kinematic tree, shape blend shape and a joint regressor.
As illustrated in Fig. 3, we utilize an artist-rigged hand bone template mesh consisting of 20 separated semantic bone meshes with 3345 vertices, 6610 faces and 19 joints plus the global orientation. Then, our PIANO model defines a function M of shape β and pose θ: Here, LBS(·) denotes the Linear Blend Skinning (LBS) function applied to the warped bone mesh model T = T + B S (β), skinning weight W and joint rotation θ. To model inherent per-bone rigidness, we adopt a fixed blending weight pre-defined by bone label intuitively and omit the pose blend shape. Thus, we utilize the canonical template meshT ∈ R 3N of N = 3345 vertices under zero pose and shape, as well as the shape blend shape B S (β) to factor the per-bone anatomic geometric changes of each individual. In PIANO, we formulate the linear shape blend shape function as B S (β; S) = Sβ to enable the base shape to vary with identity, where β is the shape parameters and matrix S ∈ R 3N ×|β| represents orthonormal principal components of shape displacements, which is learned from registered training meshes. Besides, in PIANO we formulate the joint regressor J(β) = J (T + B S (β)) to map the various shapes to the corresponding anatomically precise joint positions of human hands, where J ∈ R 3K×|β| is the learned regression matrix that transforms rest pose vertices into rotation joints; K = 25 contains 19 rotation joints, 5 finger tips and 1 wrist keypoint, matching our dataset annotation. Our final PIANO model is M(β, θ; Φ) : R |β|×|θ| → R 3N which maps the shape and pose parameters to bone vertices of various individuals (|β| = 10 and |θ| = 57 in our setting). The model parameters Φ = {T , S, J } includes the canonical template, shape blend shape and joint regressor.

Model Registration
Here, we propose a multi-stage registration scheme to align the bone template to all the MRI scans, bringing them into correspondence. Our scheme encodes the internal structure information in a coarse-to-fine manner to fully utilize the anatomically precise joint and bone segmentation annotations in our MRI dataset. With the joint annotations and bone segmentation, we utilize the radial basis functions (RBF) interpolation [Rhee et al., 2007] to assign per-bone label of each joint and generate fine-grained segmentation of the bone volume. Then, we extract both the iso-surface of internal bones b S and the per-bone 3D geometry b S l of the l-th joint/bone by applying the Marching Cubes [Lorensen and Cline, 1987]. Coarse Registration. We first align the warped joints of the hand bone template mesh to the joint annotations by minimizing the per-joint L2 error and solving the inverse kinematics to obtain the initial pose and shape parameters. Since the model parameters Φ are unknown during the first iteration of model registration and parameter learning, we downgrade our model from Eq. 1 into M (θ) = LBS(T ,J, θ, W), where we discard shape blend shape function and replace the subject specific parameters {T, J} with a general template {T ,J}. Fine Registration. We further utilize the internal local and global structural information to perform per-bone non-rigid alignment at a finer scale. To this end, for the l-th bone, we adopt the embedded deformation [Xu et al., 2019b] to register from the initial geometry b R l after coarse registration to the annotated geometry b S l and b S from segmentation label. Let G denote the non-rigid motion field and ED(·; G) denote the embedded deformation function. Then, our per-bone alignment is formulated as: Here, the data term enforces alignment of the warped bone geometry to both current target bone segment b S l and the global bone structure b S to enable robust alignment: (3) where C and P denote the the set of correspondences from b R l to b S l and b S , respectively, in an Iterative Closest Point (ICP) manner. To prevent over-fitting to the noise of annotation, we adopt the same locally as-rigid-as-possible motion regularization λ reg and E reg from [Xu et al., 2019b]. Fig. 4 shows the intermediate alignment results and demonstrates the effectiveness of our registration scheme. After registration, we obtain per-bone aligned meshes with global consistent topology for the following model parameter learning process.

Parameter Training
To train our PIANO model parameters Φ = {T , S, J }, similar to previous work [Loper et al., 2015;Romero et al., 2017], we adopt a bone-aware training strategy to model pose and shape separately. Let b i,j,l denote the registered l-th bone geometry of the i-th subject in the j-th hand pose and J * i,j denote the corresponding annotated 3D joint. Then, we optimize the pose parameters θ i,j that transforms a general tem-plateT to a specific bone model b i,j with the following energy function: Here, similar to SMPL, the edge term measures the edge length difference between two models: where E(·) represents the geometry edges. Such edge difference enables good estimation of pose without knowing the subject specific shape. Then, we add a joint term based on the anatomically precise joint annotation in our dataset: whereJ i,j is the joint position after LBS with the pose θ i,j . Next, we fix the above pose parameters and optimize the subject templateT i and jointĴ i iteratively with the following vertex difference defined on semantic bone segments: which enforces the posed subject template to match with the aligned bone meshes. We then run principal component analysis (PCA) onT i to obtain shape space parameters {T , S}, whereT is the mean shape ofT i and S is a matrix composed of principal component vectors. Also, given subject templatê J i andT i in rest pose, we further compute the joint regression matrix J by minimizing ||JT i −Ĵ i || 2 2 . To build PIANO, we set λ reg , λ e and λ j to 5.0, 1.0 and 100.0 respectively and iterate through the whole process of registration and parameter training with Limited-memory BFGS optimizer and PyTorch auto differentiation.

Fine-grained Semantic Bone Analysis
We now demonstrate the use cases of our statistical model in hand bone anatomic and semantic understanding. We consider the usage of PIANO as a differentiable layer enabling end-to-end MRI fine-grained bone segmentation and joint estimation. Furthermore, we show hand anatomic understanding from a single monocular RGB image. PIANO Layer. We integrate PIANO as a differentiable layer that takes θ and β as input and outputs a triangulated semantic bone mesh and hand joints. When training with supervision on PIANO parameters, we define the parameter loss L pm on predicted shape and pose [β,θ] as: where [β * , θ * ] denotes ground truth. Alternatively, for training with sparse joint annotations, we define a joint loss as: Here,Ĵ denotes the estimated joints and J * is the 3D joint annotation, The resulting PIANO loss L P is thus defined as L P = L pm + L J . Anatomic inference from MRI. As illustrated in Fig.5, we propose a novel network structure for fine-grained MRI hand bone segmentation with PIANO layer. Our encoder consists of ResNet3D and three regression branches. Each has two fully connected layers with hidden size 512 and ReLU as activation function after the first layer. The encoder extracts a latent feature from an MRI volume. Then, the extracted feature is passed to regression branches to infer pose θ and shape β of PIANO as well as global rotation and translation G. PI-ANO layer then outputs a fine-graind semantic bone mesh. The bone mesh is then voxelized to a regular volumetric grid with scalar values representing semantic bone label. The Seg-Net takes the concatenation of the voxelized mesh and original volume as input and outputs label probabilities. We train this network on our training set, which contains annotation of 3D joints and fine-grained bone mask. Thus the loss term can be defined as the summation of fine-grained semantic segmentation loss and PIANO loss, where the segmentation loss is the linear combination of multi-class binary cross entropy loss (BCE) and dice loss (DC): (y l logŷ l + (1 − y l ) log(1 −ŷ l )); (10) whereŷ is the sigmoid output of the network, y is the one hot encoding of the ground truth segmentation. K = 20 is number of fine-grained classes. Anatomic inference from RGB. To extend the application to RGB images, similar to the MRI-based inference, we build a network with an image encoder and two regression branches that infer pose and shape on the FreiHand [Zimmermann et al., 2019] with 3D joint supervision. To account for the mismatch of joint positions, we add an additional linear layer to map from our joint to the dataset joint annotation. Since we only have sparse joint supervision for this task, following [Hasson et al., 2019], we use a regularizer on the hand shape to prevent extreme mesh deformations and formulate it as L β = ||β|| 2 , which constraints the hand shape to be close to the average shape in our MRI dataset, thus the loss is defined as L J + L β .

Experiment
In the following section, we present quantitative and qualitative evaluation of PIANO and the corresponding applications.

PIANO Evaluation
We first evaluate the shape space of PIANO qualitatively on the left of Fig.6, which provides a visualization of our shape  space projected to the first two principal components (PCs). The effect of each PC is illustrated by adding ±2 standard deviations(std) to the mean shape in the center, which depicts that the first component restricts the hand size, while the second component controls the bone thickness. We further evaluate the compactness and generalization of our model quantitatively on the right of Fig.6.
Compactness. The red curve in Fig. 6 illustrates the compactness of our model shape space. It shows the shape variance of our dataset recovered by a varying number of components. Note that the first principal component covers over 50% of the shape space. Meanwhile, 10 and 20 components manage to cover 90% and 95% of the complete shape space.
Generalization. The blue curve in Fig. 6 illustrates the generalization of our model. We use the leave-one-out evaluation of the shapes in our dataset and report the mean squared vertex error. Such quantitative evaluation depicts the generalization ability of our model to unseen shapes. As the number of principal components increases, the mean error decreases to a minimum error of 2.5 mm achieved by the full space.

Comparison
Here we compare our model on various internal hand bone inference tasks.    due to our anatomically precise modeling ability. Fig. 7 further reveals that MANO suffers from impractical bone deformations since it is based on outer surface only. In contrast, our PIANO model achieves more physically plausible deformation with internal modeling. For a more thorough comparison, we also compare against the non-model based method FetalPose [Xu et al., 2019a] on MRI joint estimation and report the results in Tab. 1. It can be seen that model based methods out performs FetalPose on both metrics. Fine-grained MRI Bone Segmentation. We adopt PIANO and the network in Fig. 5 for MRI hand bone segmentation and Fig. 8 shows our results on various challenging hand poses. We compare with fully supervised medical segmentation baselines VNet [Milletari et al., 2016], 3D-Unet [Ç içek et al., 2016] and KiU-Net [Valanarasu et al., 2020]. For quantitative analysis, the dice score (DICE) and the symmetric hausdorff distance (Hausdorff) are adopted as metrics. As shown in Fig. 9 and Tab. 2, the baselines suffer from severe local inference noise, while the approach using our PIANO model provides more complete results, significantly improving the performance of fine-grained bone segmentation. Anatomic Inference from RGB. In Fig. 10  qualitative results for obtaining PIANO model from a single RGB image, which is taken from the evaluation set of the FreiHand dataset. Note that our PIANO results are overlayed with the MANO surface in Fig. 10. While MANO captures the outer hand surface, our model is able to estimate inner bone structure which is anatomically precise.

Conclusion
We have presented PIANO, the first parametric hand bone model which enables anatomically precise modeling of the internal kinematic structure of human hands in a data-driven manner. Our model is biologically correct, simple to animate and differentiable, based on a new dataset of high-quality MRI hand scans of various subjects and poses. For model creation, our multi-stage registration approach and the boneaware model learning strategy enable us to utilize the joint and bone segmentation annotations in MRI data provided by radiologists. Furthermore, we demonstrated that our model enables more anatomically precise personalized rigging and fine-grained hand bone segmentation from an MRI scan or even a single RGB image in a data-driven fashion. We make our model publicly available to stimulate further work and potentially benefit virtual hands in games/film, robotics, human healthcare and medical education. In future, we plan to extend our parametric model to muscles, tendons and fats using our MRI dataset for more ingenious hand modeling.