{'fraction_non_alphanumeric': 0.024720563013779645, 'fraction_numerical': 0.013897924182388078, 'mean_word_length': 5.191871109483706, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Development and Validation of a Model for Laparoscopic Colorectal Surgical Instrument Recognition Using Convolutional Neural Network–Based Instance Segmentation and Videos of Laparoscopic Procedures

Key Points Question Is an automatic surgical instrument recognition model with wide applicability to multiple types of instruments and pixel-level high recognition accuracy feasible? Findings This quality improvement study for development and validation used a multi-institutional data set that consisted of 337 laparoscopic colorectal surgical videos. The mean average precisions of the instance segmentation of surgical instruments were 90.9% for 3 instruments, 90.3% for 4 instruments, 91.6% for 6 instruments, and 91.8% for 8 instruments. Meaning This study suggests that deep learning can be used to simultaneously recognize multiple types of instruments with high accuracy.


Introduction
Minimally invasive surgery (MIS), such as laparoscopic and robotic types, continues to play an important role in general surgery as an alternative to traditional open surgery. Since the 1980s, technological advancement and innovation have led to a rapid increase in the number of MIS surgical techniques, because they are viewed as more desirable. Minimally invasive surgery has better cosmetic results than traditional open surgery and has become increasingly common during the past years. 1 On the other hand, the reliance on a surgeon's sense of touch when performing surgical procedures has waned, and surgeons have become highly dependent on the visual information provided by laparoscopic monitors. To be more dependable, laparoscopic cameras must be able to rapidly capture and track the surgical instrument being used so that unsafe blind surgical procedures can be avoided. 2 Therefore, a prerequisite in MIS is the ability to accurately recognize the location of surgical instruments and track them; in other words, we can extract much of the information associated with the surgery from such kinematic data.
In recent years, computer-based automation, which self-captures and self-analyzes large amounts and a wide variety of data-exemplified by self-driving cars-has been advancing and rapidly pervading all aspects of daily life. Artificial intelligence, especially its subdomain of deep learning, is used to make big data interpretable and usable. 3 One area of artificial intelligence that has particularly bourgeoned during the last decade is computer vision (CV), which is an interdisciplinary scientific field that deals with how computers gain a high-level understanding of digital images or videos and use the knowledge to perform functions, such as object identification, tracking, and scene recognition. 4 Deep learning-based CV technology has been applied to various tasks in laparoscopic surgery, such as anatomical structures, surgical action, and step or phase recognition. [4][5][6] Applying this technology to surgical instrument recognition can enable the development of robotic camera holders with automated real-time instrument tracking during surgery, 7,8 and it can be used for surgical skill assessment, 9 augmented reality, 10 and depth enhancement 11 during MIS. Although the goal of computer-assisted surgery is full automation, it must be realized in stages (eg, starting with intestinal anastomosis). 12 In any stage, automatic surgical instrument recognition using deep learning-based CV is an indispensable fundamental technology. Therefore, an automatic surgical instrument recognition model with wide applicability to multiple types of instruments and pixel-level high recognition accuracy is required for research and development in the field of computerassisted surgery.
The objectives of this study were to develop a deep learning model that can simultaneously recognize 8 types of surgical instruments frequently used in laparoscopic colorectal operations and to evaluate its recognition performance. In addition, the applicability of the recognition results of the developed model to surgical progress monitoring was examined.

Study Design
This quality improvement study for development and validation was conducted at a single institution using a multi-institutional intraoperative video data set. This study followed the reporting guidelines of the Standards for Quality Improvement Reporting Excellence (SQUIRE) and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines. The study protocol was reviewed and approved by the Ethics Committee of the National Cancer Center Hospital East, Chiba, Japan. Informed consent was obtained from patients in the form of an opt-out on the study website, and data from those who rejected participation were excluded. The study conformed to the provisions of the Declaration of Helsinki 13 in 1964 (and revised in Brazil in 2013). and a suction/irrigation system (HiQ+ suction/irrigation system, Olympus Co Ltd) (SI-8). Reference images of these 8 types of surgical instruments are shown in Figure 1A.

CNN-Based Instance Segmentation
Convolutional neural network (CNN)-based instance segmentation was performed with Mask R-CNN 14 as the architecture network and ResNet-50 15 as the backbone network. Instance segmentation is an image recognition approach that recognizes each object individually and pixel by pixel, instead of enclosing all objects within a bounding box. The network weight was initialized to a pretrained one on the Microsoft Common Objects in Context data set, 16 which is a large-scale data set for object detection, segmentation, and captioning that contains 2 500 000 labeled instances in 328 000 images for 91 common object categories. Every annotated image in the training set was input into the deep learning model, and fine-tuning was then performed for the training set. The best epoch model, based on the model performance on the validation set, was selected. For data augmentation, the resize, randomhorizontalflip, randomverticalflip, and normalize functions were used.

Code and Computer Specifications
The code was written in Python, version 3.6 (Python Software Foundation) and is available via

Validation and Evaluation
Average precision (AP) was used as an evaluation metric for the CNN-based surgical instrument recognition task in this study. Average precision is calculated from the area under the precision-recall curve, which is described based on the number of true-positive (TP), false-positive (FP), and falsenegative (FN) results. Intersection over union (IoU) is a metric that signifies the degree of overlap between the area annotated as the ground truth (A) and the predicted area output by the CNN (B); an IoU of 1 signifies that there is perfect overlap for the corresponding pair, A and B, whereas an IoU of 0 signifies that there is no overlap. An IoU reference image is shown in Figure 2A.
In this study, the threshold value of IoU was set to 0.75; that is, when the IoU of the corresponding pair of A and B was more than 0.75, it was defined as TP; when the IoU was less than 0.75, it was defined as FN; and when there was no A corresponding to B, it was defined as FP. The formulas for precision, recall, and IoU are as follows: The mean AP (mAP) metric is widely used in object detection and instance segmentation tasks. [18][19][20] In this study, mAP denotes the mean AP value for 8 types of surgical instruments. Five-fold cross-validation and hold-out validation were used as the validation methods in this study as appropriate. In 5-fold cross-validation, the annotation data set was split into 5 segments, of which 4 were used as training sets and the remainder as the validation set. The data split was performed at the per-case level instead of the per-frame level; thus, the images extracted from an intraoperative video in the training set never appeared in the validation set. The validation was performed for all 5 validation sets, and the average mAP was calculated. A reference image of the 5-fold cross-validation is shown in Figure 2B. In addition, a misrecognition pattern analysis was performed.

Automatic Surgical Progress Monitoring
In colorectal surgery, transection of the inferior mesenteric artery (IMA) and rectum is an important landmark for monitoring surgical progress. In the standard IMA root transection procedure, the dorsal side of the IMA root is dissected with SI-4 and clipped with SI-5; SI-6 is only used for rectal transections.
To examine the applicability of the proposed model to an automatic surgical progress

Annotation Data Set
In total, 337 laparoscopic colorectal surgical videos were used in this study, and 81 760 labels were annotated for 38 628 static images extracted from the video data set. The distribution of the number of annotation labels for each surgical instrument is shown in Figure 3A: 5694 for SI-1, 4358 for SI-2, 41 778 for SI-3, 5248 for SI-4, 5254 for SI-5, 5810 for SI-6, 7022 for SI-7, and 6596 for SI-8. SI-3 had numerous annotation labels because up to 4 could appear in a single static image, and they frequently appeared with other surgical instruments.

Instance Segmentation Accuracy
The average (SD) mAP of instance segmentation for the 8 types of surgical instruments obtained from 5-fold cross-validation was 91.9% (0.34%). In this study, the threshold value of IoU was set to 0.75, but as supplemental data, the mAP values at multiple thresholds ranging from 0.5 to 0.95 are plotted and shown in the eFigure in the Supplement.  The change in the mAP when the number of types of surgical instruments to be recognized was increased is shown in Figure 3B. The mAPs of instance segmentation were 90.9% for 3 surgical instruments, 90.3% for 4 surgical instruments, 91.6% for 6 surgical instruments, and 91.8% for 8 surgical instruments.
The AP of instance segmentation for each surgical instrument is shown in Figure 3C:  instruments in a complete laparoscopic sigmoid colon resection procedure are shown in Figure 4 and the Video.

Misrecognition Pattern Analysis
The misrecognition pattern obtained when instance segmentation was performed on 1 validation set is shown in the confusion matrix of Figure 5A;

Automatic Surgical Progress Monitoring
As shown in Figure 4, the continuous recognition of SI-4 followed by the continuous recognition of SI-5 was included in the surgical step of the IMA transection. In addition, the timing of the continuous recognition of SI-6 was included in the surgical step of rectal transection.

Discussion
In this quality improvement study, we successfully developed a CNN-based instance segmentation model that can simultaneously recognize 8 types of surgical instruments with high recognition accuracy. Almost all surgical instruments used in laparoscopic colorectal operations are covered by these 8 types; therefore, the developed model can be applied to intraoperative navigation for a series of laparoscopic colorectal surgical procedures.
There are several approaches to CNN-based CV for image recognition tasks. The bounding box-based object detection approach has often been used 22,23 ; however, in future developments in the surgical research field, such as the automation of surgery, more detailed pixel-by-pixel recognition information will be essential, instead of bounding box-based object detection.
Semantic segmentation is a CNN-based CV approach that enables pixel-by-pixel image recognition. Recently, it has also been actively used in the research field of CV in surgery. 24,25 Semantic segmentation attempts to specifically understand the role of each pixel in an image through the process of dividing whole images into pixel groupings that can then be labeled and classified. The boundaries of each object can be delineated; therefore, dense pixel-based predictions can be achieved. However, semantic segmentation cannot distinguish them individually when objects of the same type overlap. Instance segmentation overcomes this limitation because it can distinguish overlapping objects, even when they are of the same type. It enables the visualization of complicated images with multiple overlapping objects and identifies their relationships. In this study, we selected instance segmentation as the image recognition approach because the crossing of multiple surgical instruments during surgery is a common occurrence.
In general, multiclass recognition is difficult-the more types of objects there are to be recognized, the more difficult the task becomes. However, the accuracy of the recognition task for the 3, 4, 6, and 8 types of surgical instruments was almost the same in this study ( Figure 3B). One possible reason for this is that we prepared a sufficient amount of annotated data. These data comprised more than 80 000 annotation labels for the training of the CNN-based instant segmentation model, enabling us to overcome the difficulties besetting the multiclass recognition task.
Although the recognition accuracy was quite high, several misrecognition patterns needed to be measured, which were mainly caused by intracorporeal artifacts other than surgical instruments in this study ( Figure 5). However, we believe that this problem can be solved in the future by intentionally adding annotation images that capture such artifacts and training the model such that they are part of the background. Otherwise, such artifacts can be added to the recognition target with annotations. In addition, although the situation that the surgical instrument tip was out of view was also expected to cause misrecognition, this pattern was minor in this study. This pattern was thought to be because such situations were rare during surgery to begin with, and much of the information on shafts and tips was used in the predictions. Differential identification among surgical instruments with the same shaft and different tip options is a challenge that will be tackled in future work.
The proposed instance segmentation model for surgical instrument recognition has various applications in surgical research and developing fields, such as automated robotic camera holders and automatic surgical skill assessments based on surgical instrument tracking. This study demonstrated the feasibility of applying the model to automatic surgical progress monitoring.
Accurately monitoring surgical progress in real time for optimal operating room (OR) logistics is necessary. 26 This monitoring enables OR resource optimization, and the spare time and human In this study, we focused on the appearance information of SI-4/5 and SI-6 in the surgical steps of IMA transection and rectal transection in laparoscopic colorectal surgery, respectively; however, other surgical instruments appear in specific surgical steps in every type of surgery. Therefore, the proposed approach for automatic surgical progress monitoring based on surgical instrument recognition is promising because of its high applicability. In the future, it may be possible to monitor not only the progress of the surgery but also the event of intraoperative bleeding based on the appearance information of surgical instruments used for hemostasis, such as SI-8.

Limitations
This study has several limitations. First, although the proposed instance segmentation model for surgical instrument recognition has various applications in surgical research and development, only a proof-of-concept evaluation was performed; the model was not applied to actual surgical practice.
Second, only validation was conducted; tests using data from an external cohort were not performed. However, the annotation data set was sufficiently large, and stable recognition accuracy was obtained from 5-fold cross-validation; therefore, the generalization performance was high.
Third, for automatic surgical progress monitoring, only preliminary results were obtained from a single intraoperative video. For future validation, we must determine the minimum frame rate for continuous recognition. Fourth, because SI-3 tended to appear with complex intersections with other surgical instruments, the corresponding mAP was lower than for the other instruments, despite having more annotation labels. Improving the recognition accuracy of SI-3 is also a future task.

Conclusions
We developed an instance segmentation model that can simultaneously recognize 8 types of surgical instruments that are frequently used in laparoscopic colorectal operations. In this quality improvement study, the recognition accuracy of the model was high and was maintained even when the number of types of surgical instruments to be recognized increased. Surgical instrument recognition is an important fundamental technology for various surgical research and development areas. The proposed model can be applied not only to the proposed automatic surgical progress monitoring in this study but also for future computer-assisted surgery realization or surgical automation.