{'fraction_non_alphanumeric': 0.032662091723853454, 'fraction_numerical': 0.00911238609517381, 'mean_word_length': 4.83800350262697, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semantic keypoint extraction for scanned animals using multi-depth-camera systems

Keypoint annotation in point clouds is an important task for 3D reconstruction, object tracking and alignment, in particular in deformable or moving scenes. In the context of agriculture robotics, it is a critical task for livestock automation to work toward condition assessment or behaviour recognition. In this work, we propose a novel approach for semantic keypoint annotation in point clouds, by reformulating the keypoint extraction as a regression problem of the distance between the keypoints and the rest of the point cloud. We use the distance on the point cloud manifold mapped into a radial basis function (RBF), which is then learned using an encoder-decoder architecture. Special consideration is given to the data augmentation specific to multi-depth-camera systems by considering noise over the extrinsic calibration and camera frame dropout. Additionally, we investigate computationally efficient non-rigid deformation methods that can be applied to animal point clouds. Our method is tested on data collected in the field, on moving beef cattle, with a calibrated system of multiple hardware-synchronised RGB-D cameras.


I. INTRODUCTION
Point clouds are a common data representation generated by robotics perception systems equipped with depth sensors, such as stereo/depth cameras and LiDARs. Complex perception systems integrate several depth sensors allowing them to maximise the coverage of their surroundings [1], [2]. For frameworks using multiple depth sensors, it is common to align and merge the point clouds using extrinsic calibration within the front-end data processing. The analysis of the merged point cloud (e.g., semantic segmentation [3], or scene classification [4]) is then performed as part of what we call back-end process, e.g., through deep learning models for real-time processing. Most deep learning models for point cloud analysis developed within the computer vision and computer graphics communities often overlook the noise generated during the data collection. However, this topic is at the core of robotics applications where both front-end and back-end processes have an impact on the model predictions accuracy and need to be considered. This paper studies how robustness to real-world data collection, such as calibration noise and camera frame dropping, can be built within the training of deep learning models for keypoint extraction in point clouds.
Keypoints extraction and matching is an important task of 3D scene understanding for robotics applications such as tracking, shape retrieval, and registration. For instance, a direct application can be found for non-rigid deformation 1 The authors are with the Robotics Institute, University of Technology Sidney, Australia raphael.guenot-falque@uts.edu.au  [16] or KPConv [17] to extract the keypoints (in blue). During training, a dataset is first manually annotated and the distance on the manifold is pre-computed (in orange). The weights of the encoder-decoder are then trained by using augmentation on the calibration, non-rigid deformation, and camera frame dropout (in green). In our experiments, the encoder-decoder inputs are n × 3 points and the outputs are the n × 6 distances to the 6 keypoints.
algorithms which require a good prior in order to solve the alignment of a template and a target as a gradient descent optimization. This surprisingly specific problem formulation finds applications in medical robotics for organs alignment [5], clothing manipulation [6], humans [7] and animals [8] body shape reconstruction. Furthermore, given a set of keypoints matching pairs, the rigid alignment of two shapes can be solved in a closed-form with singular value decomposition (SVD) [9] as an alternative to iterative closest point (ICP) methods [10].
In the livestock industry, the automation of animal health assessment and monitoring is critical to ensure animal wellbeing and improved farming productivity. Working toward this direction, automatic pipelines are required to capture and identify key areas in animal bodies [11], [12], [13], [14], [15]. Real-time 3D shape and model reconstruction have been identified as crucial steps towards explicit segmentation of animal body regions, which are meaningful for cattle posture or behaviour recognition [13]. We consider the challenge of extracting keypoints of cattle bodies using a perception system containing synchronised RGB-D cameras capturing multiple point clouds for each animal, shown in Figure 4.
We formulate the semantic keypoints extraction from point clouds as a regression problem with a segmentation encoderdecoder architecture used as the main building block [16], [17]. In contrast with [18], where an encoder-decoder is stacked into an MLP to solve the regression problem of keypoint extraction, we do not modify the encoder-decoder architecture and formulate the regression problem by redefining how the mean squared error (MSE) loss function is computed. A bloc diagram of the method is shown in Figure 1. As an additional focus of this paper, we investigate how data augmentation can be adapted to our problem by considering the integration of augmentation specific to multidepth-camera data and non-rigid deformation formulated in the form of shear transformation matrices.
The main contributions are: (1) we formulate the keypoint annotation problem of point clouds as a regression of the distance on the manifold mapped into an RBF function. The regression of the distance on the manifold is learned with en encoder-decoder architecture which essentially solves the problem of keypoint matching as a semantic problem (similarly to segmentation problems). We argue that any encoderdecoder model can be used as long as the architecture is able to learn on point clouds data. (2) We consider a complex data augmentation process that includes noise inherent to multidepth-camera systems, through the efficient use of non-rigid deformation methods for point clouds, and a computationally efficient approximation of the distance on the manifold.

II. RELATED WORK
Pose and body shape estimation from 2D images is a wellstudied topic thanks to the large datasets available on the internet (such as COCO [19]). For humans, the pose can be estimated with a deep neural network and the shape can then be optimised over the skeleton pose [7], [20]. Similar methods have been applied to various animals [21] and refined for specific breeds such as lions [22] and zebras [23]. For 2D images, large datasets with ground truth poses are available, making the use of deep learning applications possible [24]. In cases where the ground trough is not available, 2D images and annotations can still be generated from synthetics animated animals [25], however, these datasets have a lack of variety with respect to the 3D models of the animals.
In the field of computer graphics, the problem of shape correspondences [26] studies the point-wise association between different shapes (generally defined as triangular meshes) and therefore is a generalisation of the keypoints annotation problems. Amongst the most significant work, the study of the shape spectral decomposition and the spectral correspondence has been formulated in a framework called functional maps [27]. This work has later been extended for partial shapes [28], and deep learning approaches have been developed to leverage the spectral domain in a learning context [29], [30].
However, for robotics applications, where LIDAR and RGB-D cameras are used for scanning the environment, working with meshes is often a cumbersome task. Standard mesh reconstruction methods [31], [32] perform poorly on irregular, noisy, and non-closed point clouds. As a result, working directly on point clouds and bypassing the mesh reconstruction is often preferable. While functional maps could be adapted to work directly on point clouds, thanks to robust Laplacian operator [33], they lack robustness (e.g., functional maps can have catastrophic failures) and are overengineered for the problem of keypoints correspondences (i.e., extracting few keypoints is a significantly simpler task than solving the full shape correspondence).
In the case of keypoint annotation of 3D point clouds, research is sparser due to the complexity of gathering and annotating data. Traditional methods usually consider the keypoints extraction from point cloud data by extracting a 3D skeleton [34], [35], [36], [37], from which voting systems can establish correspondences [38]. Deep learning methods have recently been developed to predict the 3D skeleton from point clouds [37].
Most similar to our method is the recent work on unsupervised keypoint annotations from point clouds using deep learning models. Architectures for learning on point clouds are constantly evolving; the most widespread architectures developed over the last years are Pointnet [39], Pointnet++ [16], and Kp-CONV [17]. [18] used Pointnet stacked with additional MLP used to transform the Pointnet output into a regression problem. [40] focus on the loss functions capable of handling multiple human annotations and consider the keypoint extraction as a classification problem. [41] propose a different architecture that builds upon Pointnet and solves the problem of keypoint annotation as an unsupervised problem using symmetric linear basis shapes. Similarly, [42] considers the unsupervised problem. They use a Pointnet++ encoder and a custom-made decoder that generates the skeleton from the keypoints, which are then aligned between instances.

A. Problem Definition
Given the partial 3D shape of an animal represented as a set of point clouds captured by C depth-cameras {P 1 , . . . , P C } concatenated into a single point cloud such as .., p n } and p i ∈ R 3 , we aim at estimating the m ordered keypoints N = {n 1 , ..., n m }, where n j ∈ P . The keypoints are ordered in the sense that they are systematically annotated in the same order, such as {right rear leg, right front leg, hip, ...}. Therefore, the keypoints have a semantic meaning.
The proposed methodology is summarized in Figure 1, and each step is formally defined in this section.

B. Architecture and loss function
We propose to solve the keypoint extraction as a regression problem, where we learn the distance between each point from the point cloud to each annotated point, resulting in m distance vectors of size n. Given this distance, the keypoints could then be obtained by taking the argument of the distance vectors minimum.
From a machine learning perspective, we formulate our problem as a mapping between the n × 3 input matrix P input and the n × m output matrixD. This naturally fits an encoder-decoder architecture, where we aim to predict features for each point from the input. The point cloud encoder-decoder architectures are usually designed to solve semantic segmentation problems, where the neural network predicts the probability of each point belonging to a specific class. The formulation of an encoder-decoder into a segmentation problem is only enforced by squashing the network prediction into a probability, e.g., usually using a sigmoid function, and by backpropagating the expected class probability through the loss function.
To change the classical segmentation problem into a regression problem, we start by computing the distance on the point cloud manifold 1 , D g i,j ∈ R n×m , between the i th point and the j th keypoint using the heat kernel method [43]. The heat kernel method can be implemented on different data structures (e.g., mesh, tetrahedral, point clouds) as long as a Laplacian operator can be computed on this data structure. In our case, we use the tufted Laplacian [33], which is robust to noisy data and particularly adapted for point clouds generated by depth cameras.
The computation of the distance on the manifold is computationally expensive and is, therefore, performed as a pre-processing step (i.e., prior to training the network to avoid this computation for all epochs). The geodesic distance is then mapped into a radial basis function (RBF) with a Gaussian kernel to have a localised heat map as follows: where controls the sharpness of the annotation and should be chosen with respect to the scale of the point cloud (in our case = 10). Mapping the geodesic distance into an RBF serves multiple purposes. Firstly, we believe that it simplifies the learning process by reducing the areas where the distance has to be learned, i.e., only the distance in a local area needs to be learned instead of the full shape. Secondly, it squashes the learned features between 0 and 1, allowing us to use a standard mean square error (MSE) loss function as, without having to modify a network designed for semantic segmentation (i.e., an architecture with an encoder-decoder). Finally, it also allows us to make some specific assumptions in the data augmentation, as discussed in Section III-D. Given Equation (1), the position of the keypoints is now obtained by finding the argument that maximises the learned distances (i.e., argmax of prediction).
In this work, we argue that an encoder-decoder architecture can be used interchangeably as long as it offers the capability to learn on point clouds. Therefore, to demonstrate the flexibility of our approach, in the experiments, we test our method on both Pointnet++ [16] and Kp-CONV [17] which are commonly regarded as pillars for learning on point clouds.

C. Network's inputs data augmentation
The proposed data augmentation considers two main approaches: first, we analyse the data augmentation specific to multi-depth-camera systems, secondly we consider the data augmentation allowing us to augment the training set.
1) Multi-depth-camera system augmentation: In the literature, many of the public keypoint datasets on point clouds are built by sampling points from meshes. For applications on point clouds obtained from multiple depth cameras, several real-world challenges have to be considered. The camera rig is portable and needs to be mounted, calibrated, and unmounted for each data collection. Therefore, the deep learning model needs to be reliable to a slight change of calibration. This robustness is also important in the case of slight camera motion arising from the vibration of the camera system (in our case, this scenario is particularly relevant due to seldom, though very forceful, kicks from animals onto the structure used for data collection). Data augmentation is formally defined as with R a rotation matrix in SO (3) and t a translation vector that add calibration noise. Additionally to enforce robustness to calibration errors, the camera frames dropout has to be considered. This can occur due to network traffic between computers in the system or latency in data acquisition due to the depth camera's autoexposure. This problem has been studied in the context of multi-view CNN by embedding a dropview layer within the architecture at training time [44]. This can be adapted for encoder-decoder by dropping some of the point clouds from P such as : where x is the probability of dropping the camera point cloud. Keeping the point cloud dropout within the data loader as part of the data augmentation allows our method to stay agnostic to the model architecture.
2) Geometric transformation: Similarly to [16], [45], we propose to use changes of scale which can be formulated as a geometric transformation. We also consider random flipping similarly to [46], [45]. Following [47,Sec. 4.7.6], the scale and random flipping transformation matrices are defined as: where s x , s y , and s z are drawn from a normal distribution N (1, 0.1). f corresponds to a random reflection of the shape with respect to the YZ symmetry plane, such as: Additional augmentation methods can be obtained using geometric transformations such as translation and rotations. The networks for point clouds usually learn on structures defined from local neighbourhoods and are therefore invariant to global translation. Rotation transformations have been used in the literature [39], [16], [45]. However, in our case the dedicated perception system scans animals in a similar orientation as shown in Figure 4, therefore random rotations are out of the scope of our work.
3) Non-rigid deformation: Having control of the animal stance, i.e., how they stand during the data collection, is extremely challenging. As an alternative, one would rather learn to perform the keypoints annotation for all possible animal poses. This would ideally be obtained through the application of non-rigid deformation of the animal shape by simulating the motion of the animal.
In the field of non-rigid deformation, this is typically performed by rigging the shape of the animal (building a virtual skeleton) and animating its motion [48], [49]. In our case, such deformation is unobtainable as the virtual skeleton is unavailable. Further, this is to some extent what this paper is aiming to solve (with the difference that we are aiming at finding points on the point cloud rather than nodes from the skeleton). Alternatively, 3D shapes can be deformed without a virtual skeleton as a prior by maximising the local rigidity of the shape while trying to minimise the error on "handles" in motion (handles are points selected from the shape). While near real-time implementations of such methods exist for meshes [50], [51], the equivalent for point clouds are significantly slower [52]. Regardless of the computation time, the deformation requires the non-rigid motion to be parameterised, which is also challenging.
As an alternative to these traditional methods, we propose to use shear matrices [47,Sec. 4.7.6] to simulate the motion of the animals stepping sideways and leaning forward / backward. These shear matrices are defined as: where h s = tan(θ s ) and θ s ∼ N (0, π 20 ) the shearing angle with respect to the x axis. Similarly, h f = tan(θ f ) and θ f ∼ N (0, π 20 ) the shearing angle with respect to the z axis. Formulating the motion as additional transformation matrices allows us to obtain a non-rigid deformation matrix that can be easily parameterised and applied simultaneously as to the other data-augmentation discussed in Section III-C. tated keypoints as follow: A visualisation of the cloud data augmentation is proposed in Figure 2.

D. Network's outputs data augmentation
In the case of semantic segmentation, the data augmentation rarely impacts the networks outputs and is often overlooked in the literature (e.g., for semantic segmentation, points labels are constant). However, in the case of the regression of the distance on the manifold, the input nonrigid deformation (i.e., T scale , T forward , and T sideway ) should change the predicted distances and the data augmentation has to be considered during the loss function computation. Ideally, the distance on the manifold with respect to the annotated point would be recomputed. However, given the computational complexity of calculating the distance on the manifold, we propose to update the distance on the manifold with respect to the difference in the Euclidean distance between P and P * from Equation (8). More formally, the distance on the manifold is first approximated as and the RBF distances are updated as follow, where D e = d(P , N ) and D e * = d(P * , N * ) are the Euclidean distances before and after equation (8). D * is the updated distance on the manifold mapped into an RBF. The case of the flip transform, T flip , has to be handled differently. The transformation applies a symmetry on the YZ plane to the shape and therefore, it changes the semantic Fig. 3: Mean error and standard deviation for D g and D g * versus the recomputed geodesic distance following (8). As shown in the plot, while the approximation considerably reduce the error, our approximation is only valid in the vicinity of the keypoints. Therefore, the use of the RBF mapping in (11) makes this approximation acceptable.
meaning of the annotation (e.g., the left legs become the right legs within the transformation). To update the outputs, a simple reshuffling of the distance vector to the annotation is necessary. Using pseudo-code with 0 indexing, the update of the outputs is reshuffled as Finally, following the camera frame dropout implemented in Equation (4), D * is updated to drop the element with indexes corresponding to the dropped cameras.

IV. EXPERIMENTAL RESULTS
To demonstrate the validity of the approach, we use real data taken in a cattle research facility with an in-house built system. The data collection has been performed at the Tullimba feedlot research facility, where a race (i.e., corridor cattle walk through) has been modified to integrate a robotics system capable of scanning cattle, as shown in Figure 4. The robotics perception system is designed with 17 RGB-D Realsense D435 cameras extrinsically calibrated with a checkerboard prior to the data collection. Cameras are synchronised using a hardware trigger allowing to capture an animal's 3D shape in a single snapshot. The data collection was undertaken on two different occasions, with 274 animals scanned on the first date and a different cohort of 136 animals scanned on the second date. On each occasion, the system is assembled and calibrated. Therefore, we consider the data collections as two independent dataset with the larger one used as training set and the smaller one used as testing set. This allows us to test if the model overfits a specific dataset or if they can be used across different datasets regardless of the calibration.
For each animals, the point clouds are stitched together using the extrinsic calibration. The background is removed to isolate the animal, and outliers are filtered using radius outlier removal (ROR) and statistical outlier removal (SOR) filters [53] resulting in P . All the point clouds from both datasets have then been annotated manually with 6 keypoints (the top of each legs, the hip, and the neck), and the geodesic Fig. 4: Data collection system used to scan cattle: the perception system is made of multiple RGB-D realsense cameras that are positioned to allow capturing the full shape of the animals in a single snapshot (each side of the camera has two rows of four cameras and one camera is located above). Overlap between the cameras allows for extrinsic calibration and ensure that the data collection is more reliable. distance [43] between the annotated keypoints and the rest of the point cloud has been pre-computed. As a result, every instance of our dataset can been considered as a tuple of three matrices {P , N , D g }.
For the training, we investigated both the Pointnet++ and Kp-CONV architectures while using standard tools and default hyper-parameters during the training stage. While this is not the focus of the paper, we provide the hyper-parameters for the sake of algorithmic reproducibility. The optimisation of the network parameters has been done using stochastic gradient descent (SGD) with a learning rate of 0.01 using a momentum of 0.9 2 .
The error on the Euclidean distance between the annotated points and the points predicted by both networks is reported in Table I and a study on the sampling size of the training set is given in Table II. The data augmentation allows to maintain good performances while being able to learn on a relatively small dataset. It should be noted that the remaining error does not account for the error of manual annotations.
In Figure 5, we perform an ablation study of the data augmentation and show that every non-rigid deformation data augmentation reduce the gap between the training and testing set. When we combine all the augmentation methods from Section III-C.2, we can see that both networks do not overfit the training set in the first 50 epochs (which they would most likely do after a certain quantity of epochs). This is a significant improvement when compared to the case without data augmentation (in black), particularly for Pointnet++, which starts overfitting the training set after the first epoch.  [16] (left) and Kp-CONV [17] (right) networks. The top graphs are an ablation study for each data augmentation, showing they reduce the gap between training and testing loss functions while improving the loss for the testing set when compared to the benchmark with no data augmentation (in black). Once combined, the data augmentation avoids overfitting the training set which can be seen as the loss function on the testing set is bellow the training set. This is particularly relevant for the Pointnet++ architecture, where the networks stops learning after 20 epochs.  Fig. 6: Sample of point cloud with dropped frame. The camera frame dropout in the data augmentation allows avoiding being overconfident in the prediction of the green keypoints. The maximum value for the predicted distance on the manifold drops from 0.79 to 0.45. As a result, we can then use the predicted distance on the manifold as a measure of the keypoints uncertainty. A study case of the camera dropout is shown in Figure 6. The distance prediction from Pointnet++ trained without camera frame dropout (shown in the left) is overconfident in the distance measure. After including the camera frame dropout (shown on the right), the maximum distance measure drops from 0.79 to 0.45, showing that the keypoint is far from the argmax of the prediction. Therefore, the predicted distance of the manifold can be used as a quantification of the keypoints uncertainty.

V. CONCLUSION
In this paper, we propose a simple yet effective method for annotating keypoints using a supervised deep learning approach. We reformulate the problem of keypoints extraction into a regression problem by inferring the distance on the manifold to keypoints. We study the relevant data augmentation methods for multi-depth-camera systems and efficient non-rigid deformation methods. Our method shows relatively good results when compared to the size of an animal's complete scan while being trained on a relatively small dataset. The experiments show that the model can be trained on a relatively small dataset while being robust to camera drops and calibration noise.
Our approach uses a single set of manual annotations for each point clouds, the error on human annotation could be mitigated by using a consensus of expert annotations similarly to [40].
ACKNOWLEDGMENT This paper is supported by funding from the Australian Government Department of Agriculture and Water Resources as part of its Rural R&D for Profit program, MLA grant number V.RDP.2005.