{'fraction_non_alphanumeric': 0.019514682987061792, 'fraction_numerical': 0.011273957158962795, 'mean_word_length': 5.303722504230119, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep learning to distinguish pancreatic cancer tissue from non-cancerous pancreatic tissue: a retrospective study with cross-racial external validation

Background The diagnostic performance of CT for pancreatic cancer is interpreter-dependent, and approximately 40% of tumours smaller than 2 cm evade detection. Convolutional neural networks (CNNs) have shown promise in image analysis, but the networks’ potential for pancreatic cancer detection and diagnosis is unclear. We aimed to investigate whether CNN could distinguish individuals with and without pancreatic cancer on CT, compared with radiologist interpretation.


Introduction
Pancreatic cancer is projected to become the second leading cause of cancer deaths in the USA by 2030, 1 with dismal survival once the tumour size exceeds 2 cm. 2,3 CT is the major imaging modality used for detection and assessment of pancreatic cancer, 4 but the method's diag nostic performance depends on radiologists' experience. Furthermore, approximately 40% of tumours that are smaller than 2 cm evade detection by CT, 5 underscoring an urgent need for novel methods to supplement radiologist interpretation in improving the sensitivity for the detection of pancreatic cancer.
Deep learning with convolutional neural networks (CNNs) has shown great promise in medical image analysis. 6 The construction of neural networks is based on a stack of neurons composed of activation functions and parameters to extract and integrate features from the images and establish a model that captures the complex relationship between images and diagnoses. CNN has been reported to achieve a high accuracy in the imaging diagnosis of various conditions including skin cancer, 7 diabetic retinopathy, 8 and liver masses. 9 However, the potential usefulness of CNN for the detection and diagnosis of pancreatic cancer has not been widely investigated. Most pancreatic cancers present with irregular contours and illdefined margins on CT and thus are often obscure at an early stage, posing substantial challenges even for the most experienced radiologists. 10,11 How well a CNN can capture the elusive CT features of pancreatic cancer and be used to detect or diagnose pancreatic cancer is unclear.
Ascertaining the potential usefulness of CNN in detecting and diagnosing pancreatic cancer has important implications. We hypothesised that with high quality training data and image preprocessing that is tailored to the characteristics of pancreatic cancer, CNN could effectively be used to distinguish pancreatic cancer from noncancerous pancreas. To provide a proof of concept, we aimed to train a CNNbased classification model for differentiating between pancreatic cancer and noncancerous pancreas on CT and assess the method's performance using local and external test datasets with comparison with radiologists' reports.

Study design
We did a retrospective, diagnostic study, using CT images from the imaging archive of National Taiwan University Hospital (NTUH), the Medical Segmentation Decathlon dataset, and The Cancer Imaging Archive (TCIA) dataset.
This study was approved by the Research Ethics Committee of NTUH, a tertiary referral centre with many patients with pancreatic cancer. Informed consent from individual patients was waived because of the retrospective design of the study. Patients or the public were not involved in the design, or conduct, or reporting, or dissemination of our research. Detailed methods of image processing and analyses are provided in the appendix (p 3).

CT image datasets and radiologist reports
Patients with histologically confirmed or cytologically confirmed pancreatic adenocarcinoma were identified from the NTUH Cancer Registry, and the CT images of those patients obtained before the date of pancreatic cancer diagnosis were extracted from the imaging archive of NTUH for review. Contrastenhanced portal venous CT images of 370 pancreatic cancer patients diagnosed between Jan 1, 2013, and Dec 31, 2018, and 320 patients with normal pancreas during the same period were randomly selected for training and testing of the deep learning model. For patients who had undergone more than one CT examination, only the examination that immediately preceded the date of diagnosis was selected. The control participants were selected on the basis of the statement of a negative or unremarkable pancreas in the original radiologist report and verified during manual labelling of the pancreas by one of the two radiologists participating in this study. The dataset was then randomly split into the training and validation set (220 patients with pancreatic cancer and 192 controls in the training set, and 75 patients with pancreatic cancer and 64 controls in the validation set) and a test set (75 patients with pancreatic cancer and 64 controls [local test set 1]). CT images of another 101 patients with pancreatic cancer and 88 controls were randomly selected from the NTUH imaging archive between Jan 1, 2006, and Dec 31, 2012, and used as the second test set (local test set 2; figure 1). Among the 471 pancreatic cancer patients of NTUH who were included in the study, pancreatic cancer was diagnosed by histology in 355 (75·4%) patients and by cytology in

Research in context
Evidence before this study We searched PubMed from inception to Jan 4, 2020, for original research articles, using the search terms "pancreatic cancer convolutional neural network" and "pancreatic cancer deep learning", without language restrictions. All the identified references were reviewed.
Convolutional neural network (CNN) has been reported for segmentation of pancreas, risk stratification of intraductal papillary mucinous neoplasms, prediction of grading of pancreatic neuroendocrine neoplasms, targeting pancreatic tumour for radiotherapy, and classification of pancreatic cysts. A case study reported the initial experience of using deep learning algorithms for the classification of patients with pancreatic cancer and healthy controls, without formal testing of the algorithm in independent datasets. The study analysed only a subset of available patients without describing the inclusion and exclusion criteria, and no information was provided regarding the reference standard used and the interval between the index test and reference standard. Therefore, that study is at risk of bias according to the Quality Assessment of Diagnostic Accuracy Studies-2 tool.

Added value of this study
We trained a CNN using contrast enhanced-CT images of Asian patients to distinguish pancreatic cancer from healthy pancreases. CNN achieved excellent accuracy and improved sensitivity compared with radiologist interpretation in independent Asian test sets, with acceptable performance in a North American test set obtained from patients of various races and ethnicities using diverse scanners and settings. These results provide the first solid proof of concept that CNN can capture the elusive CT features of pancreatic cancer to assist and supplement radiologists in the detection and diagnosis of pancreatic cancer.
Implications of all the available evidence CNN can accurately differentiate pancreatic cancer from non-cancerous pancreas, and with improvements might accommodate variations in patient race and ethnicity and imaging parameters that are inevitable in real-world clinical practice. CNN holds promise for developing computer-aided detection and diagnosis tools for pancreatic cancer to supplement radiologist interpretation. the others. The median interval between CT examination and histological or cytological diagnosis of pancreatic cancer was 16 days (IQR 8-30 days). The formal radiologist reports of the CT images of pancreatic cancer patients in the two local tests were retrieved from the NTUH electronic health record. After removal of the identity of the patient and interpreting radiologist, the radiologist reports were jointly reviewed by two radiologists (PTC and KLL) without reference with the CT images. A radiologist report was considered to have correctly classified the patient as having pancreatic cancer if a description of a definite or suspicious pancreatic tumour was found in the report; otherwise the cancer was considered to be missed by the interpreting radiologist.
CT examinations were done using six scanners (Brilliance iCT 256, Philips Healthcare [Best, Netherlands]; Sensation 64 and SOMATOM Definition AS+, Siemens Healthcare [Forchheim, Germany]; Aquilion one, Toshiba [Tochigi, Japan]; Revolution CT and LightSpeed VCT, GE Medical system, [Milwaukee, WI, USA) with 100, 120, 130 kV, or automatic mA control without extra noise reduction processes. The slice thickness was 0·7-1·5 mm, and image size was 512 × 512 pixels. The portal venous scan was obtained at 70-80 s after intravenous admini stration of contrast medium. The volume of the contrast medium (mL) was determined by multiplying the bodyweight (kg) by 1·5, with an upper limit of 150 mL. All images were reconstructed into 5 mm slices for subsequent interpretation and analysis.

Image preprocessing
All axial CT images that contained the pancreas or pancreatic cancer from an individual participant were . Because the pancreas bordered multiple organs and structures, and pancreatic cancers often had indistinct borders with the surrounding tissue, inter observer differences might exist regarding the exact extent of the pancreas and the tumour. Therefore, the labelled pancreas and tumour on the images were jointly reviewed for consensus by the radiologists before further processing and analysis. The window width was set as 250 Hounsfield units (HU) and window level as 75 HU. The images were normalised to [0, 1] by linear interpolation, and the portions that were neither pancreas nor tumour were excluded from further analysis. The images were then cropped into square subregions (ie, patches) using the moving window method on the axial (xy) plane, starting from the topleft corner. The window moved along the xaxis between the right and left boundaries, at which the window moved one step downward along the yaxis and then resumed moving along the xaxis toward the other border, until finally ending at the bottomright corner. To increase the variation and size of training data, the moving distance was set as half of the patch dimension to generate overlapping patches. The patches that contained pancreatic cancer were labelled as cancerous, whereas patches that contained only noncancerous pancreatic parenchyma were labelled as noncancerous.

Patch-based and patient-based analyses
Using all patches of the training and validation set, a deep CNN was trained to identify the probability that a patch harbours pancreatic cancer and to classify the patch as  cancerous or noncancerous on the basis of the cutoff that achieved the highest Youden index in the receiver operating characteristic (ROC) curve constructed using the training and validation set (patchbased analysis). The optimal patch size was found by comparing the performance of models trained with patches of different sizes with fivefold cross validation. For differentiation between patients with and without pancreatic cancer (patientbased analysis), the entire pancreas of controls and tumours in patients with pancreatic cancer were treated as regions of interest (ROIs), which were processed into patches and subject to analysis by using the trained CNN. Patients were classified as with or without pancreatic cancer on the basis of the proportion of patches generated from the patient's ROIs that were diagnosed as cancerous by the CNN model, using the cutoff that achieved the highest Youden index in the ROC curve that was constructed using the validation set.

Model architectures and training
The CNN model was modified from Visual Geometry Group (VGG) network 12 , a network widely used in image classification. First, the model contained three convo lutional blocks. Each block consisted of two convolution layers followed by the rectified linear unit as the activation function, and then connected with a maxpooling layer before the next block. Thereafter, a flatten node was attached in the last convolution block, and three fully connected (dense) layers were attached at the end of the CNN model. Finally, weighted binary crossentropy 13 was used as the loss function to account for imbalance between the number of cancerous and noncancerous patches. Details of the CNN, including the layer structures, kernel sizes, channels, and output sizes of the network are given in the appendix (p 5). The process of model training and testing is summarised in the appendix (p 3). To optimise model performance, two callbacks based on validation loss were used during the training process. First, learning rate would be reduced to 10% if loss did not decrease for ten iterations. Second, early stopping callback would stop the training after the validation loss remained stable after 40 iterations to avoid overfitting. All codes were written in Python (version 3.6.8) using Keras (version 2.2.4) 14 and Tensorflow (version 1.7.0) 15 libraries.

Model testing with local and external datasets
Model performance was assessed using local and external test sets to ascertain sensitivity, specificity, and accuracy with respect to patchbased and patientbased analyses. The external test set consisted of two opensource datasets from research institutions in the USA (figure 1). The Medical Segmentation Decathlon dataset 16 from Memorial Sloan Kettering Cancer Center, New York, NY, USA included CT images of 281 patients with pancreatic cancer, and TCIA dataset [17][18][19] included CT images of 82 individuals with normal pancreas from the US National Institutes of Health Clinical Center.

Statistical analysis
For comparison between groups, Fisher's exact test was used for comparing categorical variables and Mann Whitney U test for comparing continuous variables.

Role of the funding source
The funder of this study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. All authors had full access to all the data in the study, and WCL and WW had final responsibility for the decision to submit for publication.

Results
The clinical characteristics of the patients with pancreatic cancer in the local dataset are summarised in table 1. The dataset included 14 780 CT images (7223 pancreatic cancers and 7557 controls). The training and validation set and two local test sets were similar with respect to demographics and tumour stage. Tumour size was slightly smaller in local test set 2 compared with that in the training and validation set.
The performance of the CNNs that were trained with varying patch sizes are shown in Data are mean (SD), n (%), or median (IQR).
In local test set 1, the sensitivity for detecting pancreatic cancer for CNNbased analysis was similar to that of radiologists (table 3). In local test set 2, in which participants had smaller tumours than did those in test set 1, CNNbased analysis achieved significantly higher sensitivity than that of radiologists (table 3). For both local test sets combined, the sensitivity of CNN based analysis was also significantly higher than that of radiologists (0·983  were missed by CNNbased analysis, and 12 (7·1%, excluding eight patients without radiology reports) were missed by radiologists (appendix p 2). The three tumours missed by CNN were 1·1-1·2 cm in size ( figure 4A, B), of which two were correctly classified by the radiologists. The 12 tumours missed by radiologists were 1·0-3·3 cm in size, of which 11 (92%) were correctly classified by CNNbased analysis ( figure 4C, D). One tumour sized 1·2 cm was missed by both the CNN and the radiologist. The specificity of the radiologist reports was not assessed because the control participants were selected on the basis of the statement of a normal or unremarkable pancreas in the radiologist report.
The external test set consisted of 4898 cancerous and 1427 noncancerous CT images, yielding 144 249 cancerous patches and 807 911 noncancerous patches. The sensi tivity, specificity, accuracy, and balanced accuracy of the trained CNN in differentiating between cancerous and noncancerous patches and between participants with and without pancreatic cancer are given in figure 2G and H and table 3.
Because the major limitation of radiologist interpre tation is inadequate sensitivity for small pancreatic cancers, we further compared the sensitivity of CNN based analysis and radiologists in detecting pancreatic cancer stratified by cancer stages and categories of primary tumour size ( In the external test set, the sensitivity of CNN based analysis was 0·631 (0·502 to 0·747) for tumours smaller than 2 cm, 0·823 (0·760 to 0·875) for tumours 2-4 cm, and 0·933 (0·779 to 0·992) for tumours larger than 4 cm (p trend <0·001). Information on cancer stage was not available in the external test set.

Discussion
This study provided a proof of concept that CNNbased analysis could accurately distinguish patients with and without pancreatic cancer on portal venous CT. The CNN based analysis achieved an accuracy approaching 99% and missed fewer tumours compared with that of radiologists. Furthermore, the CNN that was trained using CT images of Asian patients with pancreatic cancer from a single institution achieved acceptable performance when tested by CT images obtained from patients of different races and ethnicities using diverse scanners and settings. These results support that CNN can robustly recognise the CT characteristics of pancreatic cancer and thus hold great promise for developing computeraided detection and diagnosis tools for pancreatic cancer.
Although deep learning has been studied for the diagnosis of pancreatic cystic neoplasms, 20,21 neuro endocrine tumours 22 and segmentation of the pancreas, 23-26 the technique's potential usefulness in the diagnosis of pancreatic cancer, the most common pancreatic malignancy, has not been widely investigated. This study focused on examining whether CNN can tell the subtle differences between pancreatic cancer and normal pancreas that often evade the naked eye, because these differences are a potential window of opportunity to fulfil the utmost unmet need of improving the sensitivity for detecting small pancreatic cancers. Major challenges exist in applying deep learning to the diagnosis of pancreatic cancer. First, 80% of pancreatic cancers have irregular contours and illdefined margins on CT, leading to inconspicuous borders with surrounding tissue. 10 Second, the fact that a tumour occupies only a portion of the pancreas, which is surrounded by multiple organs and structures, poses severe challenges in model fitting because the CNN might be distracted away from the tumour towards irrelevant regions, which comprise most of the image. Third, deep learning requires a large number of labelled images, but labelling of the pancreas and tumour is cumbersome and time consuming because of the organ's anatomical complexity. 27 To tackle these difficulties, we preprocessed CT images into patches and used tech niques including moving window and flipping as a means of data augmentation. Our patchbased analytical approach also had the advantage of allowing the CNN to interrogate the tumour multiple times, rather than once only if the pancreas and tumour was analysed as a whole. Because the diagnosis of pancreatic cancer versus non pancreatic cancer (ie, patientbased analysis) was based on analyses of multiple patches, the results might be less sensitive to variations in image quality that would affect the accuracy of patchbased analysis, and thus were more robust than results based on a single analysis of the pancreas and tumour as a whole. Therefore, the sensitivity and specificity of patientbased analyses could exceed those of patchbased analyses in all datasets in this study. We also excluded unrelated neighbouring organs and structures when training the CNN. This approach yielded significantly better performance compared with CNNs trained with neighbouring organs and structures included. The CNNbased analysis achieved almost 99% accuracy in multiple local test sets, with acceptable generalisability to the US test set, providing the first proof of concept that with proper image preprocessing, CNN can capture the complex imaging characteristics of pancreatic cancer. Notably, the CNNbased analysis was able to correctly identify the real site of the tumour, even in the presence of seemingly similar secondary findings such as a dilated pancreatic duct, which might seem similar to the tumour. The potential ability of CNNs in detecting and distinguishing various pancreatic diseases in addition to pancreatic cancer (eg, pancreatitis and other pancreatic tumours) should be explored in future research. The neural network architecture used in this study was modified from the VGG network, a classic CNN architecture, which has been widely used for image detection 28 and segmentation. 29 The original VGG net work was trained for object recognition and consisted of 13 convolution layers and three fully connected layers. 12 The VGG network has been shown to achieve good performance on the ImageNet dataset 12 and subsequently applied to medical image analysis in tasks such as classification of common diseases on chest xray, 30 identification of metastatic breast cancer on histological evaluation of lymph node biopsies, 31 and segmentation of blood vessels and optic disc in retinal imaging. 32 However, the input size in the original VGG network (224 × 224 pixels) is substantially larger than the size of the patches in this study and thus might cause overfitting. Therefore, we reduced the number of con volutional layers from 13 to six to reduce the complexity of the model, and the finding that model performance in the local test sets was similar to that in the training and validation set suggested that no notable overfitting occurred.
The notable finding of the CNN achieving higher sensitivity compared with that of radiologists in the local test sets supports the potential usefulness of the CNN in assisting radiologists and clinicians. Because the two local test sets and training and validation set shared no common patients, the good performance of the CNN could not be attributed to overfitting. Even in a tertiary medical centre, 7·1% of the pancreatic cancers were missed by radiologist interpretation. This finding high lights the common clinical scenario of pancreatic cancers being present on CT images but missed and only discovered in retrospect when the cancers have become clinically apparent and underscores the limitations in the diagnostic performance of CT for pancreatic cancer. Patient survival dramatically decreases when tumour size exceeds 2 cm, 33 but tumours smaller than 2 cm are often inconspicuous on CT, with approximately 40% of such tumours being missed. 5 Disparities in radiologists' expertise and experience and heavy workload might further adversely affect the accuracy of radiologist interpretation. Given the rising incidence of pancreatic cancer 34 and dire consequences associated with missed or delayed diagnosis, solutions to these limitations are urgently needed. 35 Our results suggest that with further improvements, a CNNbased computeraided diagnostic or detection tool might reduce the miss rate in clinical practice and narrow disparities in resources and expertise.
A tradeoff generally exists between sensitivity and specificity, but an ideal computeraided diagnostic tool for pancreatic cancer needs to have high sensitivity and specificity simultaneously. To that end, this study adopted a unique twolevel analytical approach. If a singlelevel analysis was used, lowering the threshold to increase the sensitivity for detecting pancreatic cancer would Data are sensitivity (95% CI), or n/N (%). Only patients with pancreatic cancer were analysed. Sensitivity by stage in the external dataset could not be assessed because information on cancer stage was not available. CNN=convolutional neural network. *Radiologist report could not be identified in two patients. †Radiologist report could not be identified in 3 patients. ‡Radiologist report could not be identified in three patients. §Radiologist report could not be identified in 5 patients. ¶Radiologist report could not be identified in 3 patients. ||CI and p value could not be calculated because no case was missed by either CNN or radiologist.  invariably lower the specificity. Given the somewhat low incidence of pancreatic cancer (38 per 100 000 population aged over 50 years), screening the population by a test with even 99% specificity would yield a positive predictive value of merely 3·6% (ie, only 3·6% of all individuals diagnosed as having pancreatic cancer actually have pancreatic cancer), causing excessive patient anxiety and unnecessary examinations and treatments. 36 Instead of diagnosing whether the patient had pancreatic cancer (patientlevel analysis) on the basis of a single round of analysis, in this study the images were processed into overlapping patches representing finegrained sub regions and analysed individually (patchlevel analysis). Furthermore, because the patches were overlapping, each subregion was analysed multiple times by the trained CNN, each time with different surrounding foci included for analysis, which thereby increased the chance of spotting foci harbouring cancer (ie, improved the sensitivity in detecting pancreatic cancer). However, just as multiple comparisons increase the chance of type I error and thus the standard for inferring statistical significance needs to be tightened, examining the image multiple times also increased the chance of false positivity (ie, reduced specificity); therefore, a cancer diagnosis should require more than one patch diagnosed as cancerous. In the patientlevel analysis, diagnosing pancreatic cancer required the proportion of patches diagnosed as cancerous to exceed a threshold, which was objectively decided from the training and validation set, to protect against excessive false positives. Our results support the usefulness of this analytical approach.
To ascertain the potential generalisability of CNN in recognising pancreatic cancer, we tested the CNN trained solely using Asian patients with external datasets, which differed substantially in patient race and ethnicity and scanning parameters. Although the sensitivity for tumours smaller than 2 cm seemed unsatisfactory at first glance, DeWitt and colleagues 11 found that the sensitivity of CT with radiologist interpretation was only 53% for pancreatic cancers smaller than 2·5 cm. 11 Therefore, the sensitivity of the CNNbased analysis was at least similar to radiologist interpretation. The lower sensitivity of the CNN in the external test set compared with local test sets might be attributed to the differences in race and ethnicity and scanners and protocols between the training and external test sets, which could present greater challenges in small tumours. A key factor that influences the imaging characteristics of the pancreas and differs between races is the fat content of the pancreas. Higher pancreatic fat content reduces the density of the pancreas on CT (ie, darker pancreas to the naked eye), and previous studies found striking differences in pancreatic fat content between races and ethnicities. 37,38 Although technical differences between different centres might have also contributed to the lower sensitivity of the CNN in the external test set, a stan dardised imaging protocol (the pancreatic protocol 39 ) is widely adopted for the assessment of pancreatic diseases. Therefore, technical variations between institutions were more likely to have been random rather than institutionspecific and less influential compared with differences in the imaging characteristics of the pancreas between races and ethnicities. Whether the performance of CNN can be improved by including in the training dataset images obtained in diverse patient populations using various scanners and protocols should be further investigated.
We used CT images with a multitude of technical variations as test sets to investigate whether CNN has the potential to distinguish patients with and without pancreatic cancer in real clinical practice. In a preliminary report, deep learning differentiated between 156 patients with pancreatic cancer and 300 healthy participants with 94·1% sensitivity and 98·5% specificity, but all images were obtained using a standardised protocol on CT scanners from a single vendor and carefully matched on the basis of imaging parameters to minimise technical variations across images. 27 Therefore, the wide variations resulting from equipment, scanner, and various patient factors that radiologists inevitably encounter in clinical practice were excluded in that study, with unclear generalisability to other datasets. By contrast, this study included images obtained using six CT scanners from four major CT vendors, without selection based on imaging parameters. Therefore, the training and local test data in this study closely resembled that encountered in clinical practice and the results should have better generalisability than do those of the previous study. The excellent performance of our CNN in all local test sets supported the notion that a welltrained CNN based on adequate training data could reliably detect pancreatic cancer and serve as a computeraided diagnostic tool in realworld clinical practice. On the other hand, the decreased accuracy of our CNN in an external test set suggested that important differences in CT images might exist between different races and ethnicities, and such diversity needs to be included in the training data for the trained CNN to have good generalisability across diverse populations.
The finding that the CNNbased analysis achieved a higher sensitivity than radiologists did should be interpreted with caution. Radiologists were provided with key clinical information and the reasons for requesting CT examinations from the clinicians when they interpreted the images, whereas the CNN was not provided with any information other than CT images. The trained CNN examined whether pancreatic cancer existed in individual focused areas (ie, patches) generated from the ROIs segmented by radiologists. Therefore, the major usefulness of the CNN was to assist radiologists in deciding whether a lesion or suspicious area in the pancreas harbours pancreatic cancer. One of the common clinical scenarios in which the CNN is especially useful is when patients present with obstructive jaundice, a typical sign of pancreatic cancer in the pancreatic head, but the CT findings are negative or equivocal. In such circumstances occult pancreatic cancers should be highly suspected even if no apparent mass is noted on CT, given that approximately 40% of pancreatic cancers smaller than 2 cm are missed on CT because of inconspicuous borders with surrounding tissue. 5,10 This study showed that a fine grained examination of the pancreatic head by the CNN in those patients could facilitate detection of occult pancreatic cancers and supplement radiologist interpretation. Because the CNN examined only the CT patches without knowing other information, such as the number of patches generated from each patient or ROI and the patient or ROI from which a certain patch was generated, the high accuracy of the trained CNN was most likely due to an ability to recognise the imaging characteristics of pancreatic cancer. Collectively, our results suggest that the CNN could supplement radiolo gists to reduce the miss rates, rather than outperform or replace radiologists. Further prospective studies in realworld clinical settings are needed to draw definite conclusions on the relative performance of the CNN and radiologist interpretation and how the CNN should be integrated into clinical practice to maximise the benefits. Whether CNNs can distinguish between pancreatic cancer and other pancreatic pathologies such as pan creatitis and other pancreatic tumours must also be studied.
Through our explorative analysis, we found that 50 × 50 pixels, equivalent to 3·5 × 3·5 cm, was the optimal patch size for detecting pancreatic cancers on CT. The performance of CNN is influenced by the relative size of the patch compared with that of the ROI. In this study, the lower AUCs achieved by CNNs based on patches smaller than 50 × 50 pixels might be because the patches were too small to contain enough information on the relationship between the tumour and adjacent tissue, which is necessary for the CNN to learn and diagnose with precision. On the other hand, CNNs based on larger patches yielded lower AUCs with greater variability, probably due to an increased probability of introducing more noise with oversized patches. The optimal patch size can vary depending on the target conditions that are to be analysed by CNN and need to be explored individually.
This study had several strengths. In two local test datasets containing variations in imaging parameters seen in realworld clinical settings, the CNN could enable differentiation between individuals with and without pancreatic cancer, with accuracy consistently exceeding 98%, and had a lower miss rate than radiologist interpretation did. Furthermore, the CNN achieved acceptable performance in an external test dataset that differed in patient race and ethnicity with the training dataset, even for tumours smaller than 2 cm. We also did comprehensive exploratory analyses to identify the optimal patch size for analysing pancreatic imaging by CNN.
This study also had limitations. Because manual labelling of pancreatic images is extremely labourintensive and access to external data is limited, we used a dataset of modest sample size that included only Asian participants from a single institution; therefore, the training dataset might not include the whole spectrum of the CT manifestations of pancreatic cancer. In response to this limitation, we used data augmentation techniques to increase the training data and verified the generalisability of the CNN using an external dataset. 16 We seek to increase the volume and diversity of training and testing images in a future study. Also, because patients with pancreatic cancer and controls in the external test set were from different institutions, technical variations in image acquisition between institutions might be a potential source of confounding. Given the barriers limiting the export of patient data outside of hospitals and the paucity of publicly available data, we could not obtain data from a single North American source to constitute the external test set. However, the excellent performance of the CNN in the local test sets and the significant positive correlation between sensitivity and tumour size in the external test set suggested that the CNN diagnosed the presence of cancer on the basis of differential imaging characteristics rather than technical variations. Had the CNN relied on a subtle technical difference between the two sources in the external test set (eg, a mark placed on the image in one of the centres) for classification, the performance of the CNN would have been perfect and not related to the size of the tumour. Furthermore, because the mark was not present in the local training set, the CNN probably could not have learned to differentiate between pancreatic cancer and control on the basis of the presence or absence of the distinct technical difference.
In conclusion, this study provided a proof of concept that CNN can accurately distinguish pancreatic cancer on portal venous CT images. The CNN model holds promise as a computeraided diagnostic tool to assist radiologists and clinicians in diagnosing pancreatic cancer.

Contributors
This study was conceived by KLL, WCL, and WW, with inputs from all other authors. KLL and TW contributed equally in this work. KLL, PTC, and WCL were involved in data acquisition. TW, YMT, HR, and WW did the analyses with critical inputs from all other authors. PTC and TW were involved in data interpretation and the initial draft of the manuscript. KLL, WCL, WW, and MSW were involved in data interpretation and critical revision of the manuscript. All authors approved the final version of the manuscript.

Declaration of interests
KLL, TW, PTC, YMT, MSW, WCL, and WW report grants from Taiwan Ministry of Science and Technology during the conduct of the study. KLL, TW, PTC, WCL, and WW have a patent pending-differentiation between pancreatic cancer and noncancerous pancreas on contrast enhanced CT by deep learning. HR declares no competing interests.

Data sharing
The datasets from National Taiwan University Hospital were approved for the current study and are not publicly available. The Cancer Imaging Archive-PancreasCT dataset) and Medical Segmentation Decathlon dataset are available online. The experimental and implementation methods are provided in detail in the appendix (pp 3-6). The major components and source codes of this study are available in the repositories Python, Keras, Tensorflow, and GitHub.