{'fraction_non_alphanumeric': 0.02015199385193408, 'fraction_numerical': 0.009990607121509692, 'mean_word_length': 5.3137466307277625, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Artificial intelligence versus expert: a comparison of rapid visual inferior vena cava collapsibility assessment between POCUS experts and a deep learning algorithm

Abstract Objectives We sought to create a deep learning algorithm to determine the degree of inferior vena cava (IVC) collapsibility in critically ill patients to enable novice point‐of‐care ultrasound (POCUS) providers. Methods We used publicly available long short term memory (LSTM) deep learning basic architecture that can track temporal changes and relationships in real‐time video, to create an algorithm for ultrasound video analysis. The algorithm was trained on public domain IVC ultrasound videos to improve its ability to recognize changes in varied ultrasound video. A total of 220 IVC videos were used, 10% of the data was randomly used for cross correlation during training. Data were augmented through video rotation and manipulation to multiply effective training data quantity. After training, the algorithm was tested on the 50 new IVC ultrasound video obtained from public domain sources and not part of the data set used in training or cross validation. Fleiss’ κ was calculated to compare level of agreement between the 3 POCUS experts and between deep learning algorithm and POCUS experts. Results There was very substantial agreement between the 3 POCUS experts with κ = 0.65 (95% CI = 0.49–0.81). Agreement between experts and algorithm was moderate with κ = 0.45 (95% CI = 0.33–0.56). Conclusions Our algorithm showed good agreement with POCUS experts in visually estimating degree of IVC collapsibility that has been shown in previously published studies to differentiate fluid responsive from fluid unresponsive septic shock patients. Such an algorithm could be adopted to run in real‐time on any ultrasound machine with a video output, easing the burden on novice POCUS users by limiting their task to obtaining and maintaining a sagittal proximal IVC view and allowing the artificial intelligence make real‐time determinations.


INTRODUCTION
Point-of-care ultrasound (POCUS) assessment of the inferior vena cava (IVC) has evolved over time in response to various studies either supporting or questioning its use as a non-invasive corollary for patient volume status, central venous pressure, and right atrial pressure. [1][2][3] Although initially heralded as an accurate assessment of patient volume status dating back to the 1990s, IVC ultrasound use has been appropriately challenged due to the variability of study results and inter-rater reliability challenges. 4 6,7 Two recent studies have indicated that a 25% collapsibility index may be adequately sensitive to differentiate between fluid responsive shock patients and fluid unresponsive ones, resulting in an ROC of 0.82 for predicting fluid responsiveness. 6,7 Some authors have suggested significant difference in area under the curve (AUC) results between novice sonologists performing manual measurements and calculating IVC collapsibility at the patient's bedside and the same measurements by POCUS experts made at a later time when reviewing DICOM video of these patients. 7 Similarly, other studies have noted significant inter-rater variability between novice and expert sonologist. 8,9 Although not surprising, this raises the question of reliability in clinical situations and also highlights the workload burden imposed on novices if they have to freeze images, move forward and backward 1 frame at a time through a cine loop to find maximal and minimal diameters of an IVC, and then carefully measure at the same location. Automation of the process holds the potential to improve inter-rater reliability and even automating the steps of performing and documenting the ultrasound examination in evaluating the IVC. 10 Artificial intelligence is rapidly infiltrating modern medicine. Deep learning, a branch of artificial intelligence, is currently the most promising application for medical image analysis and interpretation. Considerable work has been performed in consultative diagnostic imaging including automatic analysis of computed tomography (CT), chest xray (CXR), and magnetic resonance imaging (MRI). [11][12][13] Deep learning applications can also be found in ultrasound, but have been largely limited to costly imaging platforms. 14 16 We developed a deep learning algorithm with temporal tracking to assist in real-time video interpretation required for ultrasound assessment of IVC collapsibility and possible prediction of fluid responsiveness in critically ill patients or patient fluid status.

Study design
This was a study of deep learning algorithm development to automatically assess whether selected ultrasound videos showed IVC collapsibility to be ≥25%.

Data
Ultrasound video data of sagittal proximal IVC ultrasound examina-

Data manipulation and labeling
All videos, used for deep learning algorithm training, were kept in their original aspect ratio and size and sorted into 1 of 2 categories.
A POCUS expert with 28 years of research, education, and clinical use experience and >200 peer reviewed research manuscript publications, ranked all videos as either collapsing at least 25%, or collapsing <25%. The assessments were made visually using video editing software MicroDicom (Sofia, Bulgaria) that allowed frame-by-frame advance and reverse viewing ( Figure 1). Diameters were compared to the reference centimeter scales present on ultrasound videos for greater accuracy.
All training videos were then augmented using FFMPEG open access software. This is a common technique in deep learning when limited data are available. The total size of the training video dataset was

Algorithm design
We   Figure 2 summarizes the steps involved with algorithm construction, training, and testing.

Statistical analysis
Fleiss' κ was calculated to compare level of agreement between the 3 POCUS experts and deep learning algorithm and POCUS experts.
Fleiss' κ is a measure of inter-rater agreement used to determine the level of agreement between 2 or more raters. IVC collapsibility has been explored in several published research studies on automatic interpretation and analysis. The first commercially available automated IVC analysis application by a POCUS ultrasound vendor was introduced nearly 4 years ago, but was not originally designed using artificial intelligence. 18 The goal of any automation is to enable less experienced health care providers and those with less medical training to be able to assess volume status and potentially fluid responsiveness in sick patients who may be suffering from some type of shock. A group of researchers specifically focused on eliminating the time and labor intensive IVC measurement burden in clinical settings by creating and testing an automated process in 8 pigs. 19 A total data set of 48 IVC evaluations was generated from the animals, and researchers were able to create a method to automatically identify and measure the IVC, offline and not in real-time. Although this is an important laboratory step forward, the authors describe a lengthy stepwise process required to manipulate the ultrasound cine loops with various filters and image size adjustment, limiting any real-time clinical application potential. Despite the limited data, this automated method accurately identified 97.9% of pig IVCs. The majority of IVC measurements were within 15% of those made by 2 sonographers.
A final limitation was that the methodology required DICOM data and took place offline, unless imbedded on an ultrasound device-that was not tested. 19 An important collaboration between clinicians and engineering experts resulted in the creation of an automated technique for IVC diameter change assessment using a complex pyramidal structure that resulted in good agreement with manual measurements. The authors reported that when measurements between physicians and the algorithm differed, in 95% of these cases the difference was <10%. 20 The complex algorithm performed well when tested in 50 hemodialysis patients. However, it required multiple steps and significant engineering expertise to design and implement. Rather than undertake complex additional steps, which will inherently increase the difficulty of implementing such an algorithm on a wide range of POCUS machines, we sought to use the ability of deep learning algorithms to find novel associations to make predictions based on regression (referring to the type of deep learning methodology used as opposed to image segmentation and actual measurement). Therefore, no actual anatomic localization, border identification, or diameter measurements are required for prediction of outcome. Further, rather than requiring software to extract optimal individual images or breaking ultrasound cine loops into individual images to analyze offline, we explored a real-time application that could analyze a cine loop immediately on an ultrasound machine or even run in real-time while a novice sonologists was scanning the patient's IVC.  Our study had a number of limitations including a very small dataset.
Ideally, the dataset would have been in the thousands of videos and would likely have resulted in much better correlation. Additionally, if the videos were in DICOM format, and we were able to obtain accurate distance and diameter measurements for training, the precision of the algorithm in differentiating between 25% collapse or greater versus less would be expected to grow significantly. Thus, the criterion standard used of a single POCUS expert to review and assess degree of IVC collapse itself introduces some amount of error. All of the videos were obtained from public domain sources, and although from actual patients, they varied in quality as did the equipment used to obtain them. It was impossible to obtain accurate patient states for all videos, and it is possible the 220 training and 50 test videos represented an atypical set of images. Data augmentation significantly increased the effective dataset size. However, for optimal results, only 1 type of ultrasound machine would have provided a very large dataset used to train the deep learning algorithm. This is not a realistic approach clinically, because many hospitals and even departments rely on a variety of ultrasound machines, therefore necessitating a robust algorithm trained on different ultrasound devices. Although tested on a separate set of ultrasound videos not previously seen by the artificial intelligence algorithm, we did not have other data from different medical centers nor were we able to test it prospectively in different settings.
This would be an important future research topic, but this pilot study lays an important foundation for this approach.
In conclusion, a non-commercial LSTM deep learning algorithm in this study showed good agreement with 3 blinded POCUS expert reviewers in determining degree of IVC collapsibility in ultrasound videos of spontaneously breathing patients. This study indicates that an artificial intelligence algorithm has the potential to improve CIVC measurements.

AUTHOR CONTRIBUTIONS
MB and LNB conceived the study concept, gathered data, performed image classification, and wrote the article. MB and LNB created algorithmic approach and performed algorithm training and testing. SA, EAS, and YTL assisted in project creating, acted as blinded ultrasound reviewers, and helped write the Introduction and Discussion. MB takes the final responsibility of the article.

CONFLICTS OF INTEREST
MB consults with EchoNous Inc., Sonosim Inc. Ethos Medical and 410Medical. None of the companies had influence or contribution to this study and article or knowledge of its performance.