{'fraction_non_alphanumeric': 0.028970091352771867, 'fraction_numerical': 0.009010136403453885, 'mean_word_length': 4.969187675070028, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-supervised Secondary Landmark Detection via 3D Representation Learning

Recent technological developments have spurred great advances in the computerized tracking of joints and other landmarks in moving animals, including humans. Such tracking promises important advances in biology and biomedicine. Modern tracking models depend critically on labor-intensive annotated datasets of primary landmarks by non-expert humans. However, such annotation approaches can be costly and impractical for secondary landmarks, that is, ones that reflect fine-grained geometry of animals, and that are often specific to customized behavioral tasks. Due to visual and geometric ambiguity, nonexperts are often not qualified for secondary landmark annotation, which can require anatomical and zoological knowledge. These barriers significantly impede downstream behavioral studies because the learned tracking models exhibit limited generalizability. We hypothesize that there exists a shared representation between the primary and secondary landmarks because the range of motion of the secondary landmarks can be approximately spanned by that of the primary landmarks. We present a method to learn this spatial relationship of the primary and secondary landmarks in three dimensional space, which can, in turn, self-supervise the secondary landmark detector. This 3D representation learning is generic, and can therefore be applied to various multiview settings across diverse organisms, including macaques, flies, and humans.


Introduction
Automated identification and tracking of important joints or other body landmarks has become an important method in biology and biomedicine. These tracking approaches have been leveraged by modern computer vision models that are designed to learn the complex visual and geometric relationships of landmarks from large annotated datasets (1)(2)(3)(4)(5)(6)(7)(8)(9). As a result, it is currently possible to computationally analyze the behaviors of many animals, including humans (1)(2)(3)(4)(5), mice (6), monkeys (7), and flies (8,9) without the use of specialized markers in a variety of contexts. These tracking algorithms are trained by use of datasets that are annotated manually by crowd-workers or non-experts who can specify the locations of primary landmarks, that is, ones that correspond to the visually distinctive features, e.g., major body extremities such as the wrist, foot, and nose. Secondary landmarks, on the other hand, typically characterize the fine-grained geometry of the subjects, e.g., an interphalangeal joint in a toe for arthritis assessment. These secondary landmarks are visually and geometrically ambiguous; consistently annotating them often requires expert knowledge in gross animal anatomy. Further, they are often task specific, and therefore, are not included in many existing landmark datasets. For example, the OpenMonkeyPose dataset (7) does not include the elbow, tail, and ear of macaques as shown in Figure 1, which can be critical for studying social interactions. These issues thus present a major impediment in obtaining a large annotated secondary landmark dataset comparable to that of the primary landmarks. We present a new method to annotate secondary landmarks in a self-supervised way by utilizing unlabeled multi-view images as shown in Figure 1. Our key insight is that there exists a strong spatial relationship between the primary and secondary landmarks, which implies that the primary landmarks (known) can be used to predict the secondary landmarks (unknown). This is possible because with a few exceptions, the primary landmarks on body extremities span a wide range of motion and deformation, and it is, therefore, likely to include the movement of the secondary landmarks. For instance, a secondary elbow landmark is close to the primary wrist and shoulder landmarks, which are strongly predictive of the elbow landmark. We formulate this secondary landmark prediction problem as learning a visual representation shared between the primary and secondary landmarks. Existing image based learning approaches (7,8) learn a visual representation in two dimensions (2D) without reasoning about underlying three dimensional (3D) geometry. 2D landmarks are a product of 3D landmarks and camera projection (3D to 2D), and therefore, learning a 2D representation alone implies learning an additional signal of camera projection. To learn the camera projection, larger annotated data seen from many viewpoints are needed. We argue that this limitation can be addressed by learning a 3D representation by factoring out the camera projection. Our 3D representation can therefore be compact, which can be learned from a substantially smaller number of annotated images. From our linear subspace analysis, we demonstrate the effectiveness of the 3D representation to express the joint subspace of the primary and secondary landmarks. Based on our hypothesis, we present a method to learn a coherent 3D spatial representation shared between the primary and secondary landmarks using multiview images, which allows us to self-supervise a 2D secondary landmark detector. We model this shared representation using a predictive pose model that predicts the 3D locations of the secondary landmarks given that of the primary landmarks. This 3D pose predictor is designed to be agnostic to viewpoints, which allows learning the compact representation with a small number of  1. This paper presents a novel semi-supervised learning approach to detect the secondary landmarks of different species, e.g., monkeys, humans and flies, by using unlabeled multiview images. (a) The primary landmarks (white circles) on body extremities characterize the overall pose where a number of labeled data are available. On the other hand, the secondary landmarks (green circle) specify the fine-grained geometry of dynamic organisms, which is specific to each behavioral task where attaining a large labeled dataset is challenging. (b) Given multiview cameras, we propose a self-supervised learning method to estimate the secondary landmarks such as elbow joint (green). We leverage a shared representation between primary and secondary landmarks to enforce geometric consistency. labeled data. The predicted 3D pose that includes the secondary landmarks is projected onto the image to supervise the secondary landmark detector (i.e., ensuring geometric consistency). Our approach differs from existing representation learning frameworks that learn the shared representation directly from 2D images (8,(10)(11)(12) because in those frameworks, the representation needs to take into account camera projection (viewpoint), which requires a good deal of labeled data. Our approach, then, is distinguished by its very low requirements for labeled data from secondary landmarks.
Another notable feature of our approach is that we employ multiview contrastive learning (13) to maximize discriminativity and/ uniqueness in the learned representation. That is, the landmarks with the same class are expected to be close in their feature space while the landmarks belonging to different classes are distant. For the secondary landmarks, we maximize the correlation between the visual features of the same landmarks while minimizing that of the different landmarks. This contrastive learning that is agnostic to the labels, in particular, plays a major role in representation learning when the number of labeled data is limited. Our approach is effective and generalizable. We show that the secondary landmarks can be reliably detected by annotating a fraction of data (using less than 10% of data). With the learned detector, we track the secondary landmarks of organisms with diverse kinematic topologies including humans, flies, and macaques.

Results
We evaluate our method on existing real-world datasets including OpenMonkeyPose (7), Human3.6M (14,15), and DeepFly (8). Figure 2 illustrates the qualitative results of our approach on the three datasets. For the Human3.6M and OpenMonkeyPose datasets, we observe that the elbow joints (secondary landmarks) are accurately estimated by using a strong spatial relationship between the primary and secondary landmarks. A similar observation can be made in case of the flies where the tibia-tarsus joints are accurately predicted.

Shared Representation Analysis.
Our method is built upon the main hypothesis that there exists a shared representation between the primary and secondary landmarks. This indicates that the motion of the secondary landmarks can be expressed by that of the primary landmarks. We validate this hypothesis using a linear subspace analysis. Consider a pose vector that is made of the primary and secondary landmarks V = Z T X T where Z = Z T 1 · · · Z T P T and X = X T 1 · · · X T S T are the set of the primary and secondary landmarks, respectively, i.e., Z, X ∈ R 3 are the 3D coordinates of the primary and secondary landmarks. P and S are the number of primary and secondary landmarks, respectively. We learn a set of linear bases that span the joint space of the primary and secondary Qualitative results of secondary landmark detection on OpenMonkeyPose, Human3.6M, and DeepFly3D datasets. White and green circles are the primary and secondary landmarks, respectively. For monkey dataset, we consider the elbows, ears, spine and mid-portion of tail as secondary landmarks for detection. For human dataset, we choose the elbows and spine as secondary landmarks. For flies, the three tibia-tarsus joints on the left-hand side limbs are used as secondary landmarks. landmarks using principal coordinate analysis (PCA): where B is the number of bases, b i is the i th linear basis, α i is its coefficient, and b is the mean pose. This joint space describes how the secondary landmark is related to the spatial configuration of the primary landmarks. Given the joint space, we measure the reconstruction error of the secondary landmarks from the primary landmarks: where E(X ) is the reconstruction error of the secondary  landmarks X . We decompose the basis into the primary landmark basis, b z i , and the secondary landmark basis, b The secondary landmarks are reconstructed by minimizing the reconstruction error of the primary landmarks. If the primary and secondary landmarks share a joint space, minimizing the reconstruction error of the primary landmarks must minimize that of the secondary landmarks. Figure 3 illustrates a comparison of reconstruction error measured by Equation (2) for 2D (light bar) and 3D (dark bar) shared representations on OpenMonkeyPose dataset. We consider seven primary landmark configurations to reconstruct the secondary landmarks as shown in the top row. For all configurations, the 3D representation that factors out camera projection shows 30.2-60.6% of error reduction compared to the 2D representation, i.e., the primary landmarks can predict the secondary landmarks more accurately through 3D representation. Among primary landmark configurations, the second configuration, i.e., the absence of wrist joints (primary) leads to an erroneous reconstruction of elbow joints (secondary). This aligns with an intuition that the motion of the secondary landmarks can be better predicted if they can be spanned by the range of motion of the primary landmarks, i.e., two joint extremities (e.g., shoulder and wrist) connecting to a secondary (e.g., elbow) are known. Similar observations can be made in the configurations #3 and #4, where an absence of the left wrist-shoulder and right wrist-shoulder, leads to inaccurate detection of left and right elbows respectively. Figure 4a shows the reconstruction error of the secondary landmarks with respect to the primary landmark configurations, where the size of the circles is proportional to the reconstruction error. For example, the secondary landmark reconstruction error for the left elbow and right elbow shown in the third and fourth columns are higher as compared to other secondary landmarks. Figure 4b, 4c and 4d show the detection rate of the secondary landmarks using probability of correct keypoint metric (16) with an error tolerance of the size of head (PCKh). For instance, in the configuration #2 and #4, the lack of the shoulder or wrist primary landmarks lead to inaccurate reconstruction of the right elbow secondary landmark ( Figure 4c). This indicates that spatial adjacency plays a pivotal role in determining the location of a given landmark, in this case the right elbow. Similar observations can be made in Figure 4d.
Impact of Contrastive Learning. In addition to joint representation learning, we leverage unsupervised contrastive feature learning to learn a discriminative representation. We maximize the feature correlation between the same landmarks while minimizing it between different landmarks. Figure 5a and 5b illustrate the visualization of UMAP of landmark feature space with and without contrastive learning. Unlike the feature distribution without contrastive learning, that with contrastive learning aligns the features of the same secondary landmarks, forming distinctive clusters with respect to the secondary landmarks regardless of views and poses. Figure 5c and 5d illustrates the feature correlation of the landmarks across views. The red points indicate the correlation among the same secondary landmarks from different views, denoted as self-correlation. The blue points indicate the correlation among different secondary landmarks, denoted here as cross landmark correlation. The one with contrastive learning produces the visual feature of a secondary landmark that is highly correlated with that of another secondary landmark (most correlation is higher than 0.7). In contrast, this correlation is minimized with contrastive learning, which makes the visual feature more discriminative.

Ablation Study
We conduct an ablation study to measure the impact of each component. (1) L L : a landmark detection model that is fully supervised by the labeled data. This model is equivalent to the supervised learning models.
(2) L L + L t U : a semi-supervised learning model that uses the reprojection of triangulated secondary landmarks to self-supervise the secondary landmark locations. This method is an application of Günel et al. (8) for the task of secondary landmark prediction. (3) L L + L g U : a semisupervised learning model that uses 3D shared representation to self-supervise the secondary landmark locations. 3D reconstructed secondary landmarks are projected onto multiview images to supervise the landmark detector (minimizing reprojection error). (4) L L + L g U + L c U : a semi-supervised learning model (our full model) that use both the shared representation and contrastive learning for self-supervision. It minimizes geometric error but also maximizes visual feature correlation, resulting in a view-invariant representation.   Table 1 summarizes the performance of each method measured at PCKh@t where t={0.25, 0.5, 0.75} on the secondary landmarks for three datasets. In general, semi-supervised learning L g U that uses multiview unlabeled data significantly outperforms the supervised learning method L L . Further, the model trained with contrastive learning, L L + L g U + L c U , improves the performance on generalization. In particular, the left-ear, right-ear, spine, and mid-tail joints in OpenMon-keyPose, right-elbow, left-elbow and spine joints in Humans 3.6M and J3, J8 and J13 joints in DeepFly3D show significant improvement. Figure 6 shows the PCKh performance of the baseline methods to detect secondary landmarks on human, monkey and fly subjects. Our approach significantly outperforms the listed baseline approaches.
Comparison with State-of-the-Art approaches. We compare the accuracy of the secondary landmark detection with 7 baseline algorithms. (1) Alternating least squares (17) (ALS): this is a matrix completion method that can predict the secondary landmarks by considering them as missing entries in a matrix and by applying rank minimization. We construct the matrix made of the coordinates of the primary and secondary landmarks. Given a set of primary landmarks, the algorithm finds a nearest neighbor from the labeled set, and completes the missing secondary landmarks by minimizing the rank of the matrix. (2) Biased alternating least squares (18) (BALS): this is a variant of alternating least squares with weighted-λ-regularization. (3) Variational autoencoder (19) (VAE): this learns a latent code that can express the data distribution in the presence of missing data. A VAE has been used in 2D human pose estimation with occlusion (19). We apply these three methods on the 2D and 3D secondary landmark prediction. (4) Supervised approach L L : we use the labeled secondary and primary landmarks to train the landmark detector in a fully supervised manner. We use a metric called PCKh (16) to measure accuracy of secondary landmark detection. A predicted landmark is con-OpenMonkeyPose (7) Human3.6M (14,15) DeepFly (    sidered as correct if it is within tL pixels from the ground truth landmark, where t is an error tolerance proportion given the reference length L: the length between the head and neck joints for the OpenMonkeyPose and Human3.6M datasets, and the length between joints J0 and J4 for L for the Deep-Fly3D dataset.
We evaluate the approaches based on the accuracy of the predictions and use PCKh@0.5 as the evaluation metric summarized in Table 2. It is worth noting that while a few approaches exhibit strength for particular landmarks, none of these baseline methods consistently dominate over all the datasets. As expected, due to the limited amount of data, the supervised method performs poorly. The quality of our secondary landmark detection strongly outperforms the baselines with a large margin. In Figure 7, we report the qualitative comparison that shows the performance of the baselines listed in Table 2, to detect the secondary landmarks.
In Table 3, we evaluate the performance of the secondary landmark detection on the OpenMonkeyPose dataset by vary-  Table 3. We study the performance of the secondary landmark detection as vary- ing the amount of labeled data |D X | while that of unlabeled multiview data |D U X | remains constant.
ing the amount of labeled data, keeping the unlabeled multiview data constant. We observe that the performance of our method increases as the amount of secondary landmark labels increases. At an observed ratio of 1:10, the proposed method outperforms the fully-supervised approach.

Discussion
We propose a new solution to a relatively under-studied problem in the field of automated behavioral tracking, that of secondary landmark detection. Unlike primary landmarks that describe generic and coarse body geometry, secondary landmarks are of particular interest because their spatial configuration specifies the fine-grained geometry of organisms. Indeed, they are particularly likely to be useful in customized behavioral tasks or to answer bespoke tracking questions. Our secondary landmark detector is learned from unlabeled multiview images in conjunction with a small set of annotated secondary landmarks. It leverages the key insight that there exists a strong spatial relationship between the primary and secondary landmarks, which allows us to learn their shared representation from unlabeled data. We develop a self-supervised predictive model that can estimate the secondary landmarks from the primary landmarks. The spatial relationship between the primary and secondary is more apparent in 3D, i.e., it is easier to predict 3D landmarks than 2D landmarks that are a function of camera projection. Therefore, the use of multiview images is a critical element of our method. By using multiview images, we can triangulate the primary landmarks that can in turn be used to predict 3D secondary landmarks. These reconstructed 3D secondary landmarks are, in turn, projected onto each view to supervise the 2D secondary landmark detector. This process is helped by a contrastive learning scheme that learns the distinctive and unique visual representation of the secondary landmarks. These later processes are completely label agnostic, i.e., the learning relies on self-supervision. Our method is generic, applicable to diverse species, camera poses, and primary/secondary landmark configurations. Indeed, we demonstrate that our method can reliably augment landmarks with a smaller number of secondary landmarks on humans, monkeys, and flies. Through a linear subspace analysis, we demonstrate that there exists a subspace shared between the primary and secondary landmarks, and that, this subspace can be used to predict the secondary landmarks given the primary landmarks.
Especially when constrained to a limited amount of data, the 3D shared representation is more effective and expressive than the 2D shared representation, which agrees with our central hypothesis. Our 3D shared representation learning differentiates it from existing approaches (8) that enforce learning a 2D spatial configuration without reasoning about 3D geometry. In contrast, we explicitly learn the 2D spatial configuration through the 3D shared representation that provides self-supervision to the 2D secondary landmark detector. Based on this linear analysis, we characterize the secondary landmark prediction as a function of the primary landmark configuration so that a secondary landmark can be accurately predicted. Our secondary landmark detection can be thought of as a new landmark annotation paradigm parallel to the existing transfer learning paradigm used in neuroscience and biology. For example, DeepLabCut (6, 20) uses a small number of annotations to learn a generalizable visual representation via transfer learning: transferring a generic image representation learned from a large image dataset such as ImageNet to the target animal images. On the other hand, our approach takes an incremental bootstrapping that can substantially reduce manual annotation efforts updating a visual representation by introducing a new set of landmarks at each time given the previously learned representation for the existing landmarks (e.g., primary landmarks). Similar to transfer learning, this bootstrapping leverages a strong prior of the visual representation, which allows building a generalizable augmented landmark detector. This, however, implies the limitation of our approach: it needs an initial good representation to start the bootstrapping process, which requires a sufficient amount of annotated data for the primary landmarks. We assume that such annotated data can be attainable from existing annotation tools such as DeepLabCut (6,20) or OpenMonkeyStudio (7).

ACKNOWLEDGEMENTS
This work was supported by NSF IIS 2024581 to HSP, JZ, and BYH NSF IIS 1846031 to HSP, PCB NIMH 125377 to BYH

Method
We present a new method to learn a secondary landmark detector given the primary landmarks and unlabeled multiview images. The primary landmarks refer to the base landmarks that characterize the overall pose such as the body extremities, and the secondary landmarks refer to the customized landmarks that describe their fine-grained geometry.
We denote the sets of the primary and secondary landmarks by Z = {z k } P k=1 and X = {x k } S k=1 where z, x ∈ R 2 are the 2D locations of the primary and secondary landmarks in an image, respectively, and P and S are the number of primary and secondary landmarks, respectively. D Z = {I i , Z i } i and D X = {I i , X i } i are the labeled primary and secondary landmark datasets where I i is the i th image. Since the secondary landmarks are difficult to annotate, the size of the secondary landmark dataset is substantially (at least an order of magnitude) smaller than that of the primary landmark dataset, i.e., |D Z | |D X |. We denote the total dataset by X is the dataset of the unlabeled multiview images of the secondary landmarks, and D L = D Z ∪ D X is the labeled dataset of the primary and secondary landmarks. We assume that the multiview cameras are static, and their intrinsic and extrinsic parameters are pre-calibrated.

3D Secondary Landmark Prediction.
We learn a visual representation by predicting the 3D locations of the secondary landmarks. Specifically, we learn a new function that encodes the spatial relationship between the primary and secondary landmarks, i.e., where Z 3D = {Z k } P k=1 and X 3D = {X k } S k=1 are the sets of the 3D primary and secondary landmarks, respectively, and Z, X ∈ R 3 are the 3D locations of the primary and secondary landmarks.
As an input to Equation (3), we reconstruct the 3D primary landmarks from two views: where z i k is the k th 2D primary landmark in the i th view image, and φ reconstructs the 3D primary landmarks from two views. Π i : R 3 → R 2 is the projection function onto the i th image that encodes its camera intrinsic and extrinsic parameters. In practice, we leverage a direct linear transform (21) to linearly triangulate the 3D landmarks, which is differentiable. Instead of using the triangulated coordinates directly, we use a coordinate normalization to learn a geometrically coherent representation for the secondary landmark predictor f . Given a triangulated primary landmark set Z 3D , we perform Procrustes analysis (22) to align the primary landmarks: where s ∈ R, R ∈ SO(3), and t ∈ R 3 are the scale, rotation, and translation, estimated by a Procrustes analysis. In practice, we use the spine and shoulder limbs to define the coordinate system of a pose, i.e., the spine limb as the x-axis, and the right shoulder limb to the y-axis where the coordinate is scaled such that the spline limb has a unit length.

Semi-supervised Multiview Loss.
We denote the landmark detector for the k th landmark by Ψ k : I → R 2 that takes an image I and outputs the 2D location of the landmark, i.e., x k = Ψ k (I). We model the landmark detector by decomposing it into the feature extractor that learns a common visual representation across landmarks, and the landmark localizer the finds the 2D location given the visual representation. The feature extractor Φ : R 2 × I → R n is a function that extracts a visual feature from an image where I = R 3×H×W is the image range (H and W are its height and width, respectively), and n is the dimension of the visual feature, i.e., Φ(x, I) is the visual feature (e.g., the penultimate layer of a 2D landmark detector) at x ∈ R 2 on the image I ∈ I. On the other hand, the landmark localizer ψ k : R n × · · · × R n W H → R 2 estimates the 2D location of the k th landmark from the visual representations, i.e., In practice, we design the feature extractor with deep convolutional layers to learn a high level and generic representation for all landmarks while the landmark localizer is modeled by shallow layers that are responsible for the landmark classification.
We minimize the following objective to jointly learn the landmark detector Ψ and predictor f using both labeled and unlabeled multiview images: where L U and L L are the losses for the unlabeled and labeled data, and λ L controls the balance between two losses. θ Φ , θ f , θ ψ are the weights parametrizing the functions Φ, f, ψ. X and Z are the ground truth primary and secondary landmarks, respectively. (I i , I j ) is a pair of synchronized multiview images. L U measures the geometric consistency of the secondary landmarks between a pair of views and uses contrastive learn- Overall architecture of the proposed framework. We design twin networks to predict the 2D primary and secondary landmarks from two views. The predicted primary landmarks are triangulated to form the 3D primary landmarks. With normalization through Procrustes analysis, the 3D secondary landmarks can be predicted by 3D landmark predictor. We use the predicted secondary landmarks to evaluate geometric error (reprojection error) and equivariance measures (feature correlation). We minimize the label and unlabel losses (L L and L U ). ing: where X k is predicted by f from the triangulation of the primary landmarks via Equation (3). ·, · measures the normalized cross-correlation between two vectors. Note that the loss is agnostic to the labels of the secondary landmarks where the total data D including the unlabeled multiview image pairs can be used to learn the detector and predictor jointly. The loss for the unlabeled data encodes the complementary relationship of geometry and visual semantics. The first two terms ensure the geometric consistency by minimizing the reprojection error, i.e., the detected secondary landmarks from Ψ(I) must align with the projection of the predicted 3D landmarks Π(X). The third term enforces contrastive learning across views, i.e., the visual representation of the corresponding landmarks must be view invariant. In addition, the last term enforces the uniqueness of the visual features across landmarks, e.g., the visual feature of elbow must be sufficiently different from that of wrist. These consistency and uniqueness measures facilitate self-supervised learning, i.e., the geometric consistency allows us to precisely localize the landmarks via a consensus of predictions, and the equivariance enforces view-invariance in learning the visual representation for the landmark detector. As a result, the feature correlation of the same landmark across views is high while that of different landmarks within the same view is low as shown in Figure 5c. Jointly learning visual representation and 3D landmark prediction allows us to apply multiview supervision where visual information from one view can be transferred to the other views through multiview geometry. This results in utilizing a large amount of unlabeled data of the secondary landmarks in conjunction with a small set of labeled data.
L L is the loss for the labeled data, which can be defined as: where z and x are the ground truth primary and secondary landmarks, respectively. The first S outputs from the landmark detector Ψ are the secondary landmarks and the next P outputs are the primary landmarks.
Network Design and Implementation. We design a neural network that facilitates jointly learning the landmark detection Ψ and 3D landmark prediction f , by leveraging multiview supervision as shown in Figure 8. Given a pair of images from different views at the same time instant, the twin landmark detectors that share the weights produce the predictions of the primary and secondary landmarks in the form of heatmaps (i.e., probability of the landmarks). We use the 6 stage convolutional pose machine (1) as a pose estimator that takes as input an image with the size of 368×368 and outputs a set of heatmaps with the size of 46×46×(P + S + 1) including the background. The soft-argmax on the heatmaps is used to estimate the coordinates of the landmarks. This landmark detector is complementary to other pose detectors (2)(3)(4), and our method is agnostic to the choice of networks. The primary landmarks from these two views are triangulated to form the 3D primary landmarks, and transformed to the normalized coordinate. The visual features are extracted from the penultimate layer of the pose estimator at the 2D location of the predicted primary landmarks. We implement the 3D landmark predictor f using a multi-layer perceptron with three hidden layers that predicts the 3D secondary landmarks. These predicted secondary landmarks are projected onto each view for geometric and semantic consistency (contrastive learning). In practice, we pretrain the landmark detector and 3D predictor using the labeled data D L , and then refine them by minimizing the overall loss L with the total data D that includes labeled and unlabeled multiview data in an end-to-end fashion. In training, we use batch size 10, learning rate 10 −4 , and learning decay rate 0.8 with 2000 steps. We use the ADAM optimizer (23) of TensorFlow with