{'fraction_non_alphanumeric': 0.03301141352063213, 'fraction_numerical': 0.013227977758267485, 'mean_word_length': 4.853374443302501, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Rethinking Causality-driven Robot Tool Segmentation with Temporal Constraints

Purpose: Vision-based robot tool segmentation plays a fundamental role in surgical robots and downstream tasks. CaRTS, based on a complementary causal model, has shown promising performance in unseen counterfactual surgical environments in the presence of smoke, blood, etc. However, CaRTS requires over 30 iterations of optimization to converge for a single image due to limited observability. Method: To address the above limitations, we take temporal relation into consideration and propose a temporal causal model for robot tool segmentation on video sequences. We design an architecture named Temporally Constrained CaRTS (TC-CaRTS). TC-CaRTS has three novel modules to complement CaRTS - temporal optimization pipeline, kinematics correction network, and spatial-temporal regularization. Results: Experiment results show that TC-CaRTS requires much fewer iterations to achieve the same or better performance as CaRTS. TC- CaRTS also has the same or better performance in different domains compared to CaRTS. All three modules are proven to be effective. Conclusion: We propose TC-CaRTS, which takes advantage of temporal constraints as additional observability. We show that TC-CaRTS outperforms prior work in the robot tool segmentation task with improved convergence speed on test datasets from different domains.


Introduction
With the widespread application of surgical robots and the growing demand for autonomous surgery, vision-based robot tool segmentation plays a fundamental role in robot perception [1][2][3][4][5][6][7][8][9][10]. In surgical scenes, performances and robustness of segmentation algorithms are both important aspects for the success and safety of downstream operations. Various feed-forward networks and machine-learning techniques designed for semantic/instance segmentation have achieved promising performance [11][12][13][14][15][16][17][18]. However, their performance does not generalize when tested on data from different domains [19]. To improve robustness, some effort has already been made [20][21][22][23][24]. CaRTS [24], designed from a complementary causal model, shows a promising and robust performance when tested on counterfactual surgical environments for robot tool segmentation. However, optimization from an image-wise perspective faces limited observability. This limitation makes CaRTS hard to optimize.
In order to alleviate this issue, we propose a temporal causal model which frames robot tool segmentation along a sequence. This temporal causal model is shown in Fig. 1. We use the same idea from CaRTS where images I t and segmentation S t at timestamp t are directly determined by all unobserved robot and camera parameters T t , and the environment E t at the same timestamp t. We assume occlusion has no effect on segmentation and there is no interaction between tools and environments. As the model describes, T t+1 at timestamp t + 1 are directly determined by T t , T t−1 , and all Ts in the past. Exploring this temporal causal effect might provide temporal constraints that are effective to deal with the issues mentioned above.
We explore the underlining temporal constraints in this temporal causal model. The first direction is to differentiate time-variant and time-invariant factors in T and optimize them differently according to their property. Kinematics, with notation K, is representative of time-variant factors. Modeling the causal effect of Ks by P (K t | K t−1 , K t−2 , . . . ) can be a promising direction since the model can learn motion property from the previous trajectory to provide constraints. Another direction for the time-variant factors is making the spatial-temporal smoothness assumption, which assumes the measurement error and the inter-frame motion is small. This assumption should be safe to make since the difference between two adjacent timestamps is small and the speed of the robot is limited. Inspired by these directions, we propose temporally constrained CaRTS (TC-CaRTS) architecture. TC-CaRTS has three novel modules upon CaRTS. The first is a temporal optimization pipeline that enables the optimization of time-invariant factors like base configuration. The second is a kinematics correction network (KCN) that models the temporal causal effect for kinematics. The third is a spatial-temporal regularization based on the spatial-temporal smoothness assumption.
Our experiments show that TC-CaRTS effectively reduced the required iterations to achieve the same or better segmentation performance compared to CaRTS. Ablation studies indicate this improvement comes from the temporal constraints utilized by all three proposed modules. The code will be released. Fig. 1: Illustration of the temporal causal model for robot tool segmentation task. Arrow lines mean direct causal effect. At timestamp t, We note I t for image, E t for environment, S t for segmentation, T t for true robot kinematics and camera poses, T t m for measured robot kinematics and camera poses In summary, the main contributions of this paper are as follows: (1) Modeling temporal causal relations and exploring potential direction temporal constraints in this model. (2) Proposing TC-CaRTS that utilize the temporal constraints in (1) that requires fewer iterations to achieve the same or better performance compared to CaRTS.

Causality in Computer Vision:
Causality has been receiving increasing attention in computer vision research, especially medical area. Some researchers use ideas from causal inference to design feature representation learning methods [20][21][22][23] for domain generalization. Some researchers use the concept of counterfactual for generative models [28,29]. Lenis et al. [30] use this concept for the interpretability of medical image classifiers. Some researchers focus on posing the underlying causal model of the vision task [24,31].

CaRTS:
Ding et al. [24] have proposed a novel causal model where the segmentation is directly determined by the robot kinematics, camera poses, and the environment instead of the observed image. Based on this causal model they design CaRTS architecture that iteratively optimizes feature similarity between rendered images and observed images w.r.t the measured kinematics to estimate true kinematics. The final segmentation is the rendered silhouette of the robot model given the estimated kinematics. CaRTS achieves outstanding robustness across testing domains compared to other feed-forward networks. However, limited observability in image-wise optimization makes CaRTS hard to achieve real-time inference. Our temporal causal model and TC-CaRTS architecture incorporate temporal constraints that are intuitively helpful for this limitation.

Method
We propose the TC-CaRTS architecture based on our temporal causal model. The basic modeling of TC-CaRTS is similar to CaRTS but has three novel modules. We introduce the temporal optimization pipeline first and then introduce KCN and spatial-temporal regularization.

Temporal Optimization Pipeline
The temporal optimization pipeline is illustrated in Fig.2. Notations are described in the caption of Fig.2. The optimization objective function in TC-CaRTS is the spatial-temporal regularized ACSLoss between deep feature maps extracted from the observed image I t and rendered imageÎ t . ACSLoss calculates an attentional cosine similarity between feature maps. CaRTS use pre-trained U-Net to extract feature maps from observed image and a hybrid image made up of rendered robot tool and the average background from the training dataset. Optimizing ACSLoss between these two feature maps aligns the calculated robot configuration to the observed robot. Since the robot tool is differentially rendered, the gradient can be backpropagated to the input kinematics. Thus, gradient descent can be performed to correct the measurement error. More details about the optimization pipeline can be found in [24] Different from the image-wise CaRTS pipeline, the temporal optimization pipeline differentiates the time-variant factor and time-invariant factor. In TC-CaRTS, time-variant means robot kinematics sequence K s m and time-invariant factor means robot base configuration B. Other factors C remain constant during optimization. The optimization objective function can be written as Eq. 1 where θ is KCN's weights.
arg min During optimization, we alternatively perform gradient descent for θ and B. At each timestamp t, we first calculate ACSLoss, backpropagate gradient, and perform gradient descent for θ in KCN for k iterations to learn the temporal relation of kinematics for KCN. Then, we freeze KCN and repeat the above optimization process for B for one iteration. The final corrected kinematicsK t , base configurationB, and C are used to render the predicted segmentationŜ t .

Kinematics Correction Network
The Kinematics Correction Network (KCN) is an MLP network F θ . The input kinematics K s m is a n × d matrix, d is the dimension of the kinematics, and n is the number of timestamps. P E(n) is the positional encoding for each timestamp. The output is the corrected kinematicsK t that estimates the true kinematics. KCN can be expressed as Eq. 2:

Spatial-temporal Regularization
The spatial-temporal regularization is based on the spatial-temporal smoothness assumptions that (a) the measurement error is small and (b) the inter-frame motion between consecutive timestamps is small. As we use joint angles as kinematics to optimize, we use the L2 norm to regularize them. The Regularization term can be written as: where i represents the dimension indices of the kinematics K t,i calculated via Eq. 2. R denotes the whole regularization term, λ 1 , λ 2 are hyperparameters for adjusting regularization strength of each term. ( K t,i − K t,i ) 2 in the first term regularizes according to assumption (a) and ( K t,i − K t−1,i ) 2 in the second term regularizes according to assumption (b).

Experiment
We perform the robot tool segmentation experiment on the dataset from CaRTS [24]. The dataset has nine videos (seven for training, one for validation, and one for testing). Each video contains 400 frames. Each frame has its corresponding kinematics for both patient-side manipulators (PSMs). All videos were recorded under the same camera setting and robot base configuration which are roughly measured before recording. Robot motion recorded in all videos is in free space and no occlusion exists. We train the U-Net [11] feature extractors and all baseline models on the training dataset which only contains videos recorded from one domain without any corruption. We call this domain the regular domain. The validation and test dataset contain counterfactual videos that are recorded on other domains, e.g. smoke, bleeding, etc. Examples of the counterfactual image from different domains are shown in Fig. 3.
In our experiment, we use Dice as the metric for measuring segmentation performance. We first explore the inference speed improvement of our TC-CaRTS architecture compared to CaRTS on the test dataset. We also compare the performance of TC-CaRTS to other deep learning algorithms on the test dataset. Then we present ablation studies on the validation dataset.

Implementation Details
In our experiment, we use the same implementation setting from CaRTS. For KCN, we choose input length n = 5 and optimize all six joint angles and one tool angle of the two PSMs which makes the dimension of the kinematics d = 14, KCN has five hidden layers with 32, 64, 128, 128, 64, 32 channels. For spatial-temporal regularization we set λ 1 = 10 and λ 2 = 1. We use Adam optimizer with a learning rate of 5 × 10 −5 / 3 × 10 −6 for θ / B. All baselines are trained for 50 epochs with smoke augmentation. All of the experiments run on a single NVIDIA GeForce RTX 3090 graphic card.

Speed Improvement
To measure speed improvement, we draw inference time vs performance plots for both CaRTS and TC-CaRTS when inferring with different iteration times k per frame. We choose k    converge at a dice score of 93.6 on the regular domain while TC-CaRTS requires much fewer iterations (5 vs 50) than CaRTS. On the bleeding domain, TC-CaRTS after a single iteration outperformed CaRTS optimized with 10 iterations. Although TC-CaRTS is not real-time yet, it decreases the number of iterations for the optimization to converge. This is a significant step towards real-time inference.

Performance on Robot Tool Segmentation
In this experiment, we compare the tool's Dice score of TC-CaRTS to CaRTS [24], image-based baselines including HRNet [17], Swin Transformer [16], and method by Colleoni et al [10] and a video-based baseline STCN [18]. All baselines are trained with simulated smoke augmentation. As Tab. 2 shows, all the feed-forward network-based methods perform well on the regular domain and they can achieve real-time inference. However, their performances deteriorate significantly in other domains that are unseen in the training dataset. TC-CaRTS retains comparable performance as CaRTS on all domains.

Ablation Study
We perform ablation studies on the regular domain of the validation dataset to explore the effectiveness of all modules and design choices.

Effectiveness of all modules:
We explore the effectiveness of the modules by adding them to the CaRTS architecture. From Tab. 3, we find that using the temporal optimization pipeline can improve performance. Adding KCN can also improve performance when k is small But when k becomes larger, it might overfit some frames and fail to generalize without spatial-temporal regularization which is shown as the result when k = 10. The spatial-temporal regularization not only improves the performance when k is small and accelerates the convergence but also stabilizes the optimization process of KCN for larger k.

Input Length:
We perform an ablation study to see the influence of the input length n for KCN. We test on n = 1, 3, 5, 10, 40. From Tab. 4 We find that if we only use the current frame, i.e. n = 1, there will be a limited improvement compared to CaRTS. Once previous frames are provided n ≥ 3, the improvement becomes obvious. This indicates that the improvement comes from temporal constraints. However, with a further increase in the input length, the performance does not increase. We suppose this is because increasing the input length will also increase the difficulty of the optimization. We perform an ablation study to see the influence of the regularization strength λ 1 , λ 2 in the spatial-temporal regularization. We separately test on λ 1 = 0, 1, 10, 100, 1000, 10000 when λ 2 = 1 and λ 2 = 0, 0.1, 1, 10, 100, 1000 when λ 1 = 10. From Tab. 5 we find that when we set either λ 1 or λ 2 to 0, there are performance drops (1.5 / 0.5 dice score) compared to the default setting. When regularization is too strong (λ 1 ≥ 1000, λ 2 ≥ 100), optimization might also become harder.

Effectiveness over time:
We perform an ablation study to show an insight into the TC-CaRTS's effectiveness over time. We calculate the average Dice difference of the first m frames for m = 1, 2, . . . , 400 between TC-CaRTS and CaRTS. The result plots are shown in Fig. 5. As the plots show, the Dice difference increases as m increases. This indicates that with more temporal information having been processed, TC-CaRTS's advantage with temporal constraints becomes more obvious. We also find that when m is small (≤ 50), TC-CaRTS performs worse than CaRTS when k = 5. This indicates that TC-CaRTS might overfit some early frames. However, when more m increases, TC-CaRTS start to outperform CaRTS.

Limitations
Although TC-CaRTS is proven to be effective. There are still limitations that are not fully resolved. Firstly, the challenge to achieve real-time remains. On the one hand, differentiable rendering and backpropagation require more computation than a single feed-forward network. On the other hand, redundant feature extraction operations on similar rendered images also restrict the inference speed. Secondly, the architecture works under the assumption that there is no occlusion and interaction. To deal with occlusion and interaction, information for the environmental factor E is necessary. The representation of E might be estimated through vision or other sensors. All of these limitations also imply essential directions for future work.

Conclusion
In summary, limited observability causes slow convergence for CaRTS. We propose a temporal causal model and explore underlying temporal constraints in this model. Inspired by the temporal causal model, we propose TC-CaRTS with three novel modules to complement CaRTS-temporal optimization pipeline, kinematics correction network, and spatial-temporal regularization. TC-CaRTS requires fewer iterations to achieve the same or better performance as CaRTS while achieving the same or better performance in different domains compared to CaRTS. Ablation studies indicate that all modules are effective and the effectiveness comes from temporal constraints.