{'fraction_non_alphanumeric': 0.03805309734513274, 'fraction_numerical': 0.021277414390150056, 'mean_word_length': 4.596683893195521, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automatic Estimation of Ulcerative Colitis Severity from Endoscopy Videos using Ordinal Multi-Instance Learning

Ulcerative colitis (UC) is a chronic inflammatory bowel disease characterized by relapsing inflammation of the large intestine. The severity of UC is often represented by the Mayo Endoscopic Subscore (MES) which quantifies mucosal disease activity from endoscopy videos. In clinical trials, an endoscopy video is assigned an MES based upon the most severe disease activity observed in the video. For this reason, severe inflammation spread throughout the colon will receive the same MES as an otherwise healthy colon with severe inflammation restricted to a small, localized segment. Therefore, the extent of disease activity throughout the large intestine, and overall response to treatment, may not be completely captured by the MES. In this work, we aim to automatically estimate UC severity for each frame in an endoscopy video to provide a higher resolution assessment of disease activity throughout the colon. Because annotating severity at the frame-level is expensive, labor-intensive, and highly subjective, we propose a novel weakly supervised, ordinal classification method to estimate frame severity from video MES labels alone. Using clinical trial data, we first achieved 0.92 and 0.90 AUC for predicting mucosal healing and remission of UC, respectively. Then, for severity estimation, we demonstrate that our models achieve substantial Cohen's Kappa agreement with ground truth MES labels, comparable to the inter-rater agreement of expert clinicians. These findings indicate that our framework could serve as a foundation for novel clinical endpoints, based on a more localized scoring system, to better evaluate UC drug efficacy in clinical trials.


Introduction
Ulcerative colitis (UC) is a disabling and chronic inflammatory bowel disease (IBD) characterized by relapsing inflammation and ulceration of the large intestinal mucosa. Clinical trials in IBD use standardized scoring systems to assess both clinical outcomes and changes in disease activity. One commonly used disease severity score for UC is the total Mayo Score (Schroeder et al. 1987), which combines clinical disease features, physician global assessment, and mucosal disease burden as determined by video endoscopy. An important component of the total Mayo Score is the Mayo Endoscopic Subscore (MES) which is used to define patient-level UC severity from endoscopic videos and is graded on a discrete 0-3 scale: Normal (MES 0), Mild (MES 1), Moderate (MES 2), Severe (MES 3). See Fig. 1 for representative video frames. The MES scoring system, as deployed in clinical trials and used in clinical practice, has two notable limitations. First, gastroenterologists (GIs), often serving as central readers in a clinical trial, are required to assign a single video-level MES based upon the most severe disease activity observed in the video. Therefore, as illustrated in Fig. 2, if a patient with severe and widespread UC (MES 3) has marked improvement after treatment, except for a highly localized severe disease activity, that patient will still receive an MES 3. A consequence of this ambiguity is the relative insensitivity of the MES to quantify meaningful, local therapeutic effects of drug treatments which could negatively impact drug development and medical decision making in the clinic.
These two limitations, a lack of granularity and moderate inter-rater agreement, motivate the development of an automatic disease activity scoring system for UC which uses frame-level severity scores to provide a more precise measurement of disease distribution and severity throughout the bowel. Such an approach would allow for finer assessments of meaningful therapeutic effects in IBD clinical trials.
Prior work have examined the automatic estimation of frame-level IBD severity scores from endoscopy videos. In particular, Stidham et al. (2019); Ozawa et al. (2019); Yao et al. (2020) solve supervised learning problems that require frame-level labels from trained experts. Employing a set of trained experts to perform frame-level severity labeling, however, is a very time-consuming and expensive task, requiring large numbers of labeled frames suitable for training deep neural networks. In Sec. 4, we report that four highly experienced GI clinicians required 3-4 hours per person to label Figure 2. Human central readers in UC clinical trials assign a video-level MES based on the most severe inflammation observed in an entire endoscopy video. Therefore, as shown in this illustration, a patient will be scored an MES 3 before and after treatment, without sensitivity to wide-spread improvements (i.e., red (MES 3) reduces to green (MES 0), yellow (MES 1) or orange (MES 2)). This example demonstrates how the current clinical scoring system fails to capture important treatment effects from endoscopy videos and the need to measure severity at a higher resolution.
only 200 frames of our dataset.
Therefore, to overcome the difficulties of frame-level labeling, we utilize weakly supervised deep learning to learn frame-level estimations trained only with videolevel labels. In particular, we combine the weakly supervised algorithm multi-instance learning (MIL), with the task of ordinal classification to arrive at a novel ordinal MIL framework for frame-level severity estimation. To the best of our knowledge, this approach has not been previously developed. Most closely related to our work is Gutierrez Becker et al. (2021), which utilizes a similar form of weak supervision but does not attempt ordinal classification. Additionally, Mohammed et al. (2020) solves an alternative multi-class disease detection problem on annotated video segments. This paper is organized as follows: In Sec. 2.1 we introduce the standard MIL framework and build upon this to derive our novel ordinal MIL in Sec 2.2. Then, in Sec. 3 we describe our dataset, pre-processing, training, and experimental setup. We present our results on clinical trial data in Sec. 4.

Methods
In this section, we combine MIL with ordinal classification to estimate ordinal categories of UC severity at the frame-level using only video-level labels. In Sec. 2.1 we introduce the standard MIL algorithm, traditionally developed for binary classification, which we apply to solve two clinically relevant binary classification tasks. We then extend MIL to the case of ordinal classification for UC severity estimation in Sec. 2.2 and compare two established ordinal classification frameworks: 1) an aggregation of an ensemble of ranked binary classifiers and 2) regression with discrete ordinal labels.

Multi-Instance Learning Applied to Endoscopy Videos
Multi-instance learning (MIL) is a framework for solving weakly labeled problems in which labeled data can be viewed as a collection of smaller unlabeled parts (instances) Figure 3. Binary MIL Framework Overview. Endoscopy videos are split into frames and fed into a CNN to generate frame-level probability scores. The frame with the maximum score is selected as the representative for the video and a binary threshold is used to produce a class for the video and each frame.
that we are interested in estimating. An example medical application is the localization of disease pathology in medical images. While ground truth disease labels (eg. disease vs. no disease) may be relatively easy to provide at the image-level, annotating the specific location of disease pathology in an image has proven to be a more time consuming and difficult undertaking. MIL has been applied by dividing a labeled image into smaller unlabeled image patches and estimating the presence or absence of disease in those patches using weak supervision.
MIL has been traditionally formulated for binary classification, exploiting the assumption that for images with positive disease labels, at least one image patch will be positive, which explains the positive label of the image. Conversely, all image patches from a negative image must be negative. This framework has shown effective performance across image modalities like digital histopathology (Campanella et al. 2019) and chest X-ray (Schwab et al. 2020). Multi-class formulations have also been explored (Remedios et al. 2020;Lu et al. 2021).
In our setting, we apply the same weakly supervised paradigm by substituting videos for images and video frames for image patches. Our first application can be posed as two clinically relevant binary classification tasks: 1) MES 0 vs. (1,2,3) to detect the presence of UC, and 2) MES (0,1) vs. (2,3) to detect mucosal healing which is associated with improved long-term patient outcomes (Turner et al. 2020).
Using a neural network, MIL first estimates a probabilistic score for each frame, and then compares an aggregate of those scores with the video label in the loss function. The most typical aggregation is the maximum operator. It effectively selects the frame with the maximum score as representative of the entire video. The max aggregation is well suited for our problem because it mimics the established clinical framework of reporting the maximum MES observed in the video. This framework also allows us to model frames as independent samples, utilizing a classical MIL setting without more advanced temporal modeling.
More specifically, let V be an endoscopic video with F image frames {f i } F i=1 and binary label y = {0, 1}. Because we have image data, we use a convolutional neural network (CNN), φ, to estimate continuous disease probability scores for each frame, Then, to obtain the video-level probability, p v , we select the frame with the maximum score as the representative for the video, i.e. f v with v = argmax i p i , and present this to the binary cross-entropy loss function during training. Applying a threshold t ∈ [0, 1] to p v produces the video class predictionŷ ∈ {0, 1} and at the same time, class predictionsŷ i ∈ {0, 1} for each frame in the video. Fig. 3 provides an overview of MIL for binary classification. In the next section we extend the standard binary MIL formulation to the case of ordinal classification.

Ordinal MIL for Frame-Level Severity Estimation
Ordinal classification is appropriate for ordered data labels without an apparent numerical distance between them (i.e., Normal < Mild < Moderate < Severe) 1 . Unlike multi-class problems, where misclassifications are weighted equally, misclassifying Severe as Normal should be penalized more than misclassifying Severe as Moderate. Next, we derive the novel combination of MIL with two established ordinal classification frameworks.

Ensemble of Ranked Binary Classifiers
One approach to ordinal classification is to restructure a problem with M ordinal labels as an ensemble of M −1 ranked binary classification tasks and aggregate each output to estimate ordinal classes. Within a traditional fully supervised framework, Frank and Hall (2001)  In our application, with four ordinal classes, we must train and aggregate three binary classifiers: φ >0 : 0 vs. (1, 2, 3), φ >1 : (0, 1) vs. (2, 3), φ >2 : (0, 1, 2) vs. 3, where φ >m is a model to decide whether the target class is greater than value m. This approach is appealing because MIL is already well-defined for binary classifications and φ >0 and φ >1 are already relevant tasks introduced in Sec. 2.1. Thus, it only remains to repeat the MIL method for φ >2 . In the MIL setting, however, the output of each binary classifier results from the frame with the maximum score. Since each classifier in the ensemble is applied to the data independently, the frame with the maximum score may be different for each classifier. We therefore, must first aggregate frame-level binary outputs to produce frame-level ordinal scores and then take the frame with the maximum ordinal score as the representative for the video. Let be the ranked binary probability scores in [0, 1] for each frame and letŷ i >0 ,ŷ i >1 , andŷ i >2 be binary class predictions {0, 1} for each frame. We experiment with three ways to aggregate these probabilities to estimate discrete ordinal classes,ẑ i ∈ {0, 1, 2, 3}, for each frame: (1) (Convert) As in Frank and Hall (2001), convert the 3 binary probabilities into ordinal class probabilities, , and take the max multi-class probability as the ordinal class per frame,ẑ i = max 0≤j≤3 p i j .
(3) (Sum) First sum the three binary probabilities together to produce a continuous severity score Then apply three ordinal thresholds 0 < t 0 < t 1 < t 2 < 3 to bin the continuous scores into ordinal classesẑ i per frame.
Then, to mimic the clinical scoring system, we take the max ordinal class over all frames as the video-level MES severity class estimation,ẑ = max 1≤i≤Nẑ i . While the first two methods (Convert and Threshold) produce discrete MES severity classes per frame, the third method (Sum) also produces a continuous severity score in [0,3] per  (1, 2, 3), φ >1 : (0, 1) vs (2, 3), φ >2 : (0, 1, 2) vs 3. The output of (A) is three probability scores per frame p >0 , p >1 , p >2 ∈ [0, 1]. We experiment with three different aggregation methods, (B), (C), (D), of the frame probability scores. In (B), p >0 , p >1 and p >2 are converted into multi-class probabilities p 0 , p 1 , p 2 , p 3 ∈ [0, 1], each corresponding to the respective MES class. The maximum probability is selected as the ordinal class for each frame. In (C), three distinct binary thresholds are applied to p >0 , p >1 , and p >3 resulting in three binary classes per frame. These are summed to generate ordinal classes per frame. In (D), p >0 , p >1 , and p >2 are first summed to create a continuous score in [0, 3] per frame. A set of ordinal thresholds are then applied to generate ordinal classes per frame. In each method, the frame with the maximum class is selected as the video class. frame which may be useful for generating a more precise severity scoring system. Fig. 4 illustrates the proposed ordinal MIL using an ensemble of ranked binary classifiers and the three different aggregation methods.

Regression with Discrete Labels
Another common technique for ordinal classification is to apply regression while maintaining discrete class labels. Here the only difference between classification and regression using CNNs is the final output layer. Instead of a sigmoid activation for binary classification, the final layer is linear with one neuron. Let φ r be this regression CNN. Therefore, instead of a binary probability score p i ∈ [0, 1] as above, this network produces a continuous severity score s i ∈ R per frame such that s i = φ r (f i ). A clipping function is used to restrict the range of values to [0,3].
To combine ordinal regression with MIL, we simply take the frame with the maximum severity score, f v , as the severity score as the representative for the video, i.e. s v = φ r (f v ) with v = argmax i s i . We then apply three ordinal thresholds 0 < t 0 < t 1 < t 2 < 3 to bin the continuous scores into ordinal classesẑ i ∈ {0, 1, 2, 3} per frame. Fig. 5 gives an overview of the ordinal MIL regression method. In the next section we compare each of the proposed ordinal MIL methods on clinical trial endoscopy data and validate them on a sample of labeled video frames.

Data and Training
The data we use in this work originates from the UNIFI clinical trial of Ustekinumab (STELARA[R]) in UC (Sands et al. 2019). Our dataset consists of 1881 endoscopic videos from 726 subjects with 1 to 4 videos per subject lasting anywhere from 35 seconds to 47 minutes and a mean duration of 10 minutes. We extracted frame images from our video files at a frame rate of 1 frame per second (as in Yao et al. (2020)), resulting in a total of 1,129,188 frames with an average of 600 frames per video.
To address the class imbalance that skews towards the positive class, we propose a method that exploits the MIL assumption that all frames in a class 0 video must be class 0. Then, instead of selecting the maximum frame that represents a class 0 video, we can select the top K frames (Campanella et al. 2019). This is akin to increasing the sample size K times for the 0 class. While it may be tempting to choose a very large K, or all frames from each class 0 video, this may lead to a reverse class imbalance. We experimented with K = 1, 5, 10, 20, 40, 100, and 200 for class 0. For class 1 videos, we leave K = 1 since, by construction of MIL, there is no guarantee that more than one frame will exhibit the positive class. (If a class 0 video has less than K frames, we select all the frames in that video.) Only the maximum frame is selected at test time.
After experimentation with various CNN architectures, ResNet34 was chosen as the best performing. The frame image dimensions were resized from 640×510×3 to 224×224×3 to fit the input to the CNN. After exhaustive parameter testing, we chose an Adam optimizer with learning rate=1e-5, weight decay=0.01, and binary cross entropy loss for 100 epochs for the binary classification tasks. For the ordinal MIL regression we experimented with multiple loss functions including Mean Square Error, Mean Absolute Error, Smooth L1, and Log Cosh and found that Mean Absolute Error provided the best results. We employed online image augmentation of horizontal and vertical flips, rotations, and hue changes. We ran 5-fold cross validation with splits grouped at the subject-level with proportional label distributions.

Quality Control
Due to the transparent nature of MIL, we ensured a level of quality control by visually inspecting the top scoring frames of true and false positive predictions for consistent image features that may have misled the algorithm. We found that frames that were entirely red ("red wall") ( Fig. 6a), where the camera is pressed against the colon, were consistently the highest scoring instead of severe UC. We thus sought to remove these misclassified frames from our dataset to improve performance.
Taking a supervised approach, we manually labeled a set of 4,200 frames from several videos as being a red wall or not. We used a ResNet pre-trained on ImageNet to extract a feature vector of length 512 from each frame and trained a support vector machine (SVM), achieving 90% accuracy on a validation set of frames from videos without overlapping of subjects. We applied the classifier to the entire dataset and removed 273,262 red wall frames. After successfully removing these red wall frames, we then discovered that images with bleeding (Fig. 6b) became the highest scoring frames for both true and false positives. We repeated this approach to remove images of bleeding by manually labeling 4,604 images and training a new SVM. With 92% accuracy on a validation set, we removed 222,906 more frames.
Following the removal of these two types of image artifacts, performance improved allowing severe images to score highly and drive the optimization. We found it was not necessary to remove frames with objects such as forceps 2 (Fig. 6c) or artifacts like rinse water (Fig. 6d) because these artifacts did not contribute to severe disease scores. The two rounds of quality control led to a total removal of 496,168 frames, a 32.6% average decrease in the number of frames per video.

Video-Level Analysis
Our first results are for the three ranked binary classification tasks. With K = {40, 200, 100} we achieved AUCs of 0.90 ± 0.03, 0.92 ± 0.02, and 0.88 ± 0.02 for φ >0 , φ >1 , and φ >2 , respectively, across 5-fold cross validation. While K parameters were tuned to achieve these AUCs for each classifier independently, we found that parameters may also be tuned with respect to their combination for ordinal classification. The combination of K = {40, 100, 40} gave us the best average ordinal classification performance over the 5-fold cross validation for each aggregation method in Sec. 2.2.1.
For the ordinal ensemble model 2 (Threshold), binary thresholds are needed for each of the three binary classifiers. Using a grid search over 5-fold cross validation, we selected the optimal combination of binary thresholds {t 1 , t 2 , t 3 } = {0.20, 0.19, 0.22}. For the ordinal ensemble model 3 (Sum) which produces continuous severity scores ranging from [0,3], ordinal thresholds are needed to bin these scores into discrete MES. Using a grid search over 5-fold cross validation, we selected ordinal thresholds {t 1 , t 2 , t 3 } = {0.1, 1.04, 2.02} to maximize performance. Fig. 7 illustrates our frame-level severity estimation for two example videos from a validation set using the ensemble summation method 3 to produce continuous severity scores. The frames with the maximum score correctly estimate the ground truth MES. For the ordinal MIL regression method, we found K = 10 produced the best performance. Over 5-fold cross validation we found optimal ordinal thresholds {t 1 , t 2 , t 3 } = {1.03, 1.97, 2.79}.
In evaluating the performances of our methods, we calculated a quadratic weighted Cohen's κ value between the output of our models and the ground truth labels. Because the ground truth have been shown to be subjective and variable as indicated by a moderate Fleiss 3 κ, measuring accuracy may be a misleading quantity. Instead, we aim to demonstrate that we achieve an acceptable level of agreement with the ground truth. Our three ordinal MIL ensemble methods achieved substantial agreement with κ = 0.64 (95% CI 0.56-0.71), κ = 0.67 (95% CI 0.66-0.69), and κ = 0.65 (95% CI 0.62-0.69), respectively. The ordinal MIL regression model also achieved a substantial agreement with κ = 0.68 (95% CI 0.64-0.72). Fig. 8 shows a plot of the κ comparisons and confusion matrices for the ordinal MIL regression model. These results show a very similar performance between all methods at the video-level. In the next section, we evaluate performance at the frame-level.  Figure 8. Comparison of inter-rater agreement measured by quadratic weighted Cohen's κ between the MES estimations of each of the proposed methods and MES labels. Red circles (with 95% confidence intervals (CI) over 5-fold cross validation) indicate the agreement between model estimations and video-level labels. Each method achieves substantial agreement with ground truth labels. Blue triangles indicate the agreement between model estimations and frame-level consensus labels for 200 frames within a validation split. Purple triangle is the average inter-rater agreement between each human rater and the frame-level consensus labels with 95% CI. The confusion matrices show MES estimations vs. ground truth for Ordinal MIL Regression for video and frame-level labels. The video-level confusion matrix includes estimations over all 5-folds.

Frame-Level Analysis
While the above analyses show substantial κ values at the video level, it is also important that we also validate our methods at the frame-level. To this end, we collected frame-level MES labels from 4 experienced, non-central reader GIs to compute interrater agreement at the frame-level. We randomly selected 50 frames from videos of each of the 4 MES classes within a single validation split of our dataset, resulting in 200 total frames. This task was time consuming, requiring 3-4 hours per physician to label all 200 frames. To compare the inter-rater agreement of the 4 raters, we calculated a Fleiss κ = 0.42, which is on the low end of moderate and significantly below the reported video-level Fleiss κ = 0.60 (95% CI 0.51-0.69) (Sands et al. 2019).
In addition, because we know the ground truth video labels from which each of the 200 frames were extracted, we analyzed how well the frame-level labels coincided with the video-level labels using the fact that the frame MES label should not exceed the video MES label. In fact, 27.5% of the annotated frame labels were greater than the video scores. We adjusted the frame labels to be consistent with the video labels by taking the minimum of the two and then took a majority vote consensus to produce our frame labels. In addition to a Fleiss κ, we calculated how well each rater agreed with the adjusted consensus frame labels by calculating the quadratic weighted Cohen's κ = 0.80, 0.78, 0.77, 0.62 with an average κ = 0.74 (CI 0.61-0.87).
In comparison, for each of our models we achieved quadratic weighted Cohen's κ = 0.44 (Convert) κ = 0.63 (Threshold), κ = 0.63 (Sum), and κ = 0.66 (Ordinal MIL Regression), respectively. We found that the Threshold, Sum and the ordinal MIL regression models achieve a substantial κ value that falls within the 95% CI of the human raters at the frame-level. See Fig. 8 for this κ comparison and accompanying confusion matrices for the ordinal MIL regression. These results indicate that the proposed ordinal MIL methods, can not only automate the established MES scoring system at the video level with substantial agreement to the ground truth, but can also provide frame-level MES estimations comparable to human raters.

Conclusion
In this work, we proposed a set of novel weakly supervised ordinal classification methods to automatically estimate the severity of UC from endoscopy videos. Our methods have the ability to estimate frame-level severity on a continuous scale by training only on discrete video-level severity labels. This is beneficial for expanding models quickly to other disease types and clinical indications without expensive, time-consuming and subjective labeling for each new target of interest. Our proposed ordinal MIL framework has the potential to effectively automate the established video-level MES scoring system with a substantial level of score agreement. Furthermore, we validate that the resulting frame-level severity estimations are comparable to that of human raters. With continuous severity scores at the frame level, our method will foster the development of more localized and granular endoscopic endpoints that may be more sensitive to meaningful therapeutic effects and better predict patient outcomes in IBD.