{'fraction_non_alphanumeric': 0.028479985157010994, 'fraction_numerical': 0.016883899995361565, 'mean_word_length': 5.280221380716575, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predictive radiomics signature for treatment response to nivolumab in patients with advanced renal cell carcinoma

Introduction: The anti-PD-1 immune checkpoint inhibitor nivolumab is currently approved for the treatment of patients with metastatic renal cell carcinoma (mRCC); approximately 25% of patients respond. We hypothesized that we could identify a biomarker of response using radiomics to train a machine learning classifier to predict nivolumab response outcomes. Methods: Patients with mRCC of different histologies treated with nivolumab in a single institution between 2013 and 2017 were retrospectively identified. Patients were labelled as responders (complete response [CR]/particle response [PR]/durable stable disease [SD]) or non-responders based on investigator tumor assessment using RECIST 1.1 criteria. For each patient, lesions were contoured from pre-treatment and first post-treatment computed tomography (CT) scans. This information was used to train a radial basis function support vector machine classifier to learn a prediction rule to distinguish responders from nonresponders. The classifier was internally validated by a 10-fold nested cross-validation. Results: Thirty-seven patients were identified; 27 (73%) met the inclusion criteria. One hundred and four lesions were contoured from these 27 patients. The median patient age was CUAJ – Original Research Malone et al Biomarkers of response to nivolumab in advanced kidney cancer


Introduction
Immunotherapies have revolutionized the treatment of metastatic renal cell carcinoma (mRCC) and have demonstrated improved overall survival. (1)(2)(3)(4) Despite this advance, not all patients benefit from immune treatments and to date no predictive biomarker has been validated. CheckMate 025 was a landmark randomized phase III trial that assigned patients with previously treated mRCC to the programmed cell death protein-1 (PD-1) inhibitor nivolumab or everolimus. Patients treated with nivolumab had a statistically significant improvement in overall survival of approximately 5 months compared with everolimus. (1) However the objective response rate to nivolumab was 25% suggesting that this benefit was limited to a small proportion of patients. Furthermore, higher levels of PD-L1 expression did not predict for responses. A predictive biomarker would facilitate selection of patients most likely to respond to nivolumab and allow those who are not likely to respond to avoid immune-related adverse events and reduce the substantial treatment related costs.
The application of radiomics technology in medical imaging offers a potential platform to develop a predictive signature. (5,6) Radiomics extracts quantitative information from routine radiologic images to identify characteristic features that could predict tumor biology and immune cell infiltrates. Compared with biopsy-based methods, radiomics affords unique advantages: 1) it captures spatial information encompassing the entire tumor and microenvironment, thereby accounting for intra-tumoral heterogeneity and tumor-host interactions; 2) multiple lesions can be assessed in parallel, thereby accounting for intertumoral heterogeneity; 3) lesions can be assessed longitudinally for changes over time; and 4) unlike biopsies, it is a non-invasive method that presents minimal risk to patients.
The Predictive RadiOMIcs SignaturE for treatment response to nivolumab or PD-1/PD-L1 inhibitors in patients with advanced renal cell carcinoma (PROMISE) was a pilot study that aimed to develop a radiomics signature to predict response to immune checkpoint inhibitors in mRCC. We hypothesized that in patients with mRCC, host immune recognition of tumor cells would create a microenvironment of infiltrating immune effector cells. The

Image acquisition
Contrast-enhanced CT imaging of the thorax, abdomen, pelvis was performed with nonionic intravenous contrast. Regions of interest (ROI) were drawn circumferentially around each index lesion by a study member (H.W-S and S.P). Each target lesion on CT imaging was assessed as a distinct entity, in order to account for inter-tumoral heterogeneity. Responses were dichotomized into responders versus non-responders based on RECIST version 1.1 measurements of the target lesion only. (7) Responders were defined as lesions with complete response, partial response, or stable disease for at least 12 weeks. Non-responders were defined as lesions with stable disease for less than 12 weeks, or progressive disease. The best response was used, such that patients with initial progressive disease, but treated beyond progression with subsequent treatment benefit, were regarded as responders. The decision to differentiate stable disease at 12 weeks was based on a study of response heterogeneity with immune checkpoint inhibitors. (8)

Radiomics analysis
Using computational algorithms, quantitative radiomics features were extracted from each contoured target lesion. These radiomics features represent the defining quantitative characteristics of the lesion. For the algorithms, each image voxel (a value on a regular grid in three-dimensional (3D) space) was quantized to a finite range of 32 grey levels. Additionally, algorithms initially described for two-dimensional (2D) pictures were adapted for 3D volumes, by extrapolating from the 4 principal directions in 2D to the 13 principal directions in 3D. (9) For each tumor volume of interest (VOI) a set of 11 and 142 1 st and 2 nd -order, and two-point correlation, two-point cluster, and minimum spanning tree texture metrics were generated. A set of two-dimensional ROIs was used to construct a VOI that occupied a contiguous range of slices and overlapped from one slice to the next. The 1st-order metrics were composed of 11 image intensity percentiles from each VOI ranging from the minimum value (0%) to the maximum value (100%) broken up into nine 10% intervals.

CUAJ -Original Research
Malone et al Biomarkers of response to nivolumab in advanced kidney cancer 4 © 2021 Canadian Urological Association Characterization of the one dimensional image intensity histogram shape was provided by these metrics.
The intensities within each VOI were grouped into 32 equal-sized bins extending across the range of image intensities from the 1st percentile at the bottom, and the 99th percentile at the top prior to computing the 142 second-order texture metrics. The second order texture features were composed of metrics from four classes calculated from multidimensional histograms: (i) the mean and range of the 13 Haralick features computed from the gray scale co-occurrence matrix (10) taken over all 13 neighbor orientations (9); (ii) five features based on the neighborhood gray tone difference matrix (11); (iii) ten features from the gray level run-length matrix (12); and (iv) the same ten features from the gray level size zone matrix. (13) A detailed, illustrated description of these metrics has been previously published. (9) Machine learning algorithm A support vector machine (SVM) algorithm with a radial basis function (RBF) kernel was used to carry out all machine learning. A classifier was trained to predict pre-treatment whether advanced renal carcinoma tumors treated with nivolumab would be responders (CR/PR/durable SD) or non-responders and distinguish post-treatment between nivolumab responding and non-responding tumors, based on the texture features extracted from the associated VOIs. The MATLAB interface to the LibSVM (library for support vector machines) software library (14) was used to apply the SVM training algorithm to the data, consisting of 104 contoured lesions (lymph nodes: 62; lung metastases: 24; kidney or adrenal metastases) from 27 patients, in these two configurations. It was necessary to tune three hyperparameters governing the behaviour of the classifier for each SVM training run. The first hyperparameter was related to feature selection. 191 input texture features were ranked in order of their association with the response classification using an F-statistic approach. (15) A tunable hyperparameter which represented the fraction of the most highly associated features to keep was applied to select the features that were used. The second hyperparameter was the standard cost parameter common to all versions of SVM, while the third was the width of the Gaussian that makes up the RBF kernel.
The three hyperparameters were tuned using a nested cross-validation scheme while ensuring to keep the assessment of accuracy totally independent. In each of 100 iterations of the outer loop, ten-fold cross-validation was used to hold out 10% of the data for testing, while the remaining 90% was passed to the inner loop. Within the inner loop, a further tenfold cross-validation protocol was employed for each point in a three-dimensional grid covering a range of fractions of the best features to retain, values of the SVM cost parameter, and values of the RBF width. For each grid point searched the inner-loop cross validation result was recorded, the best performing triplet of hyperparameters was utilised to train a classifier using all of the inner loop data. This classifier was then applied to classify the heldout data from the outer loop, and the results were recorded as the s accuracy.
The pre and post-treatment texture features for each series along with their binary

CUAJ -Original Research
Malone et al Biomarkers of response to nivolumab in advanced kidney cancer 5 © 2021 Canadian Urological Association classifications (non-responder: 0; responder: 1) were separately inputted into the second stage supervised machine learning (SML) RBF SVM classifier to train binary discrimination between the responders and non-responders. A nested 10-fold cross-validation protocol was used to assess classifier accuracy. The second stage SML algorithms used were also "as is", the same as were used for previous projects. The SVM algorithms generate a cross-validation (CV) accuracy value for each combination of three metaparameters: fraction, cost, and gamma, yielding a 3-dimensional CV accuracy space. The optimization target was to find the region of the parameter space yielding the highest cross-validation accuracy.

Receiver operating characteristic (ROC) and other post-hoc analyses
A SVM classifier produces a single, continuous number on the real line rather than a dichotomous binary classification as its output. The SVM classifier is only transformed into a classification when a threshold is applied to it. As a result it is possible to adjust the threshold to adjust sensitivity and specificity, thereby creating a ROC curve. Repeating the outer loop of the nested cross-validation protocol 100 times yields 100 such numbers for each tumor, this results in a more fine-grained ROC curve than would be possible without repetitions. Each of the 100 numbers for a particular tumor represents an instance in which it was held out during cross-validation with a different 10% of the data, allowing for a more robust characterization of the accuracy for that case. In the results section below, the mean ± SD accuracies reported are the best from each of the 100 ROC curves, and the sensitivities and specificities are taken from the point on each curve where the best accuracy was found. By combining all the trial data into a single curve the ROC curves presented in the figures are generated.

Statistical analysis
Descriptive statistics were used to summarize baseline demographics. Survival estimates were computed using the Kaplan-Meier (KM) method. The accuracy of the predictive signature from the SVM classifier was determined by 10-fold nested cross-validation, and reported in terms of sensitivity, specificity and AUC. An AUC of 0.5 would signify no discriminatory ability, 0.7-0.8 represents an acceptable level, 0.8-0.9 represents a good level of discrimination and 0.9-1.0 would be excellent. Statistical significance was determined by permutation testing. (16) For permutation testing, the observed accuracy was compared against the distribution of accuracies obtained by random noise, generated by randomly labelling the radiomics signatures as being responders or non-responders using 10,000 permutations. Significance level was set at 0.05.

Patients
Thirty-seven patients were identified between August 21, 2013 and October 6, 2017. Only 27 patients met the inclusion criteria with available imaging. Ten patients were excluded as imaging was either not available (n=3) or the imaging protocol used was incompatible with

CUAJ -Original Research
Malone et al Biomarkers of response to nivolumab in advanced kidney cancer 6 © 2021 Canadian Urological Association the radiomics software (n=7). A final cohort of 27 patients were used for analysis. The median patient age was 56 years, 78% were male, 89% had clear cell histology, 89% had prior nephrectomy and 89% had prior systemic therapy. The selected patients were categorized as being in the good (30%), intermediate (60%) or poor (10%) risk groups as per the International Metastatic RCC Database Consortium (IMDC) scoring system. (17) The lesions selected were lymph nodes (60%), lung metastases (23%) and renal/adrenal metastases (17%). The characteristics of the patient population and ROIs are listed in Table 1.
The median progression and overall survival for the entire population was 10. The final data set of 27 subjects with pre-treatment and immune checkpoint inhibitor post-treatment yielded 52 responding lesions and 52 non-responding lesions that were analyzed using a SML radiomics application pipeline that has been previously validated.

Performance of radiomics signature
Lesion VOI contours were extracted using validated Neuroimaging Informatics Technology Initiative (NIfTI) software libraries. Each series consisted of a specific subject and set of lesion VOIs which were examined using the radiomics analysis software image display to confirm that: 1) the VOI contours were successfully extracted from the NIfTI files and 2) their positions matched those of a lesion. No issues were found with the data set and it was found to be suitable for radiomics texture analysis. The PROMISE contour bounded data were batch processed by the first stage of the radiomics application to generate the set of texture features to be used by the second stage radiomics SML application.

Supervised machine learning
The exhaustive iterative grid over the region of highest cross-validation accuracy in the 3D space yielded a set of ROC curves as represented in Figures 4a and 4b for the pre-treatment and post-treatment groups respectively. For the classifier trained on the baseline CT scans, 69% accuracy was achieved. For the classifier trained on the first post-treatment CT scans, 66% accuracy was achieved. The pre-specified level of sensitivity and specificity that was required for the analysis to be successful was 85%; on this basis, the pre and post-treatment ROC curves do not demonstrate sufficient discriminatory ability to warrant further study.

Discussion
In patients with mRCC treated with Immune Checkpoint Inhibitors (ICI) we were not able to derive a radiomics signature that could predict for response or resistance to ICI. The pre and post-treatment ROC curves are negative results for the radiomics analysis of the PROMISE

CUAJ -Original Research
Malone et al Biomarkers of response to nivolumab in advanced kidney cancer 7 © 2021 Canadian Urological Association data set. The current "as is" supervised machine learning set of algorithms were not able to discriminate significantly better than a binary classifier for a response in a malignant lesion. A reason for this might be that the user-defined texture features cannot significantly discriminate between ICI responder and non-responder classifications.
The patient population in this analysis was representative of the typical mRCC patient cohort that would be treated with ICI. The lesions contoured included lymph nodes, lung metastases and renal metastases, which are typical of the metastases seen in this patient cohort. The OS in the responders were similar to what has been reported in the Checkmate-025 study. However, the PFS was much longer in the PROMISE responders compared to the Checkmate-025 cohort, which could be because without formal tumor measurements physicians were more likely to continue patients beyond progression or at least call progression of disease later in the patients' treatment course.
There are certain limitations to the use of radiomics in general. For example the textures that are sensitive to acquisition modes and reconstruction parameters are not recommended for radiomics applications such as malignant and benign tissue differentiation. (18) Accurate delineation of tumor volumes is crucial for the computation of radiomic features, doing this manually is time-consuming and subject to inter-observer variability. Numerous studies have recommended automatic and semiautomatic methods of tumor volume delineation over manual contouring. (19,20) We tried to account for this by having 2 investigators define the VOI, however this approach may still have had inherent inaccuracies that could have been resolved by an automated process. Although tumor heterogeneity can potentially be quantified using radiomics features extracted from medical imaging, many features are unstable between imaging scans which would mean that a large number of features may be unreliable. (21,22) Datasets need to consist of 10-15 patients per feature to reduce the false discovery rate necessitating the use of larger datasets. (23) Optimal cut-off values of radiomic features are often used to stratify patients into two risk groups for KM survival analysis, however searching for the optimal cutoff values through testing multiple thresholds can increase the likelihood of obtaining statistically significant but spurious results. As the optimum cut-off value can vary in different datasets, single study results may not be reproducible in other patient populations. Independent validation datasets are required to confirm the prognostic value of the same radiomics features to avoid false positive results. (24,25) Recent studies have confirmed the utility of radiomics across numerous cancer types (lung cancer (25)(26)(27)(28)(29)(30)(31)(32); head and neck cancer (25,27,33,34); glioblastoma (35)(36)(37)(38); esophageal cancer (9,(39)(40)(41); rectal cancer (42); hepatocellular carcinoma (43); prostate cancer (44); RCC (45); soft tissue sarcoma (46); and germ cell tumors (47)) using various imaging modalities. In one study, radiomics features for predicting the survival of patients with lung cancer were found to be distinct from those for head and neck cancer, highlighting the need to develop tumor-specific signatures. (27) In general, most studies have explored correlates of survival, gene expression or molecular pathway aberrations. Several studies

CUAJ -Original Research
Malone et al Biomarkers of response to nivolumab in advanced kidney cancer 8 © 2021 Canadian Urological Association investigated predictive signatures for pathological response to neoadjuvant chemoradiotherapy. (9,19,29,(39)(40)(41)(42) At this stage, the role of radiomics is still emerging, and radiomics has not been integrated into clinical practice as signatures have not yet been independently validated and the complexity of deriving the signature may make it difficult to implement outside of centres that do not have the computational algorithms to calculate such signatures.
The current study does have some important limitations. It is a retrospective, single institution analysis which used a selected patient sample without an external validation cohort. The sample size was small in this pilot study because of the time required for manual contouring of primary kidney lesions and metastatic lesions from different sites. A larger sample size would have been prohibitive without automation. We do not know if adding further patients would potentially change the results seen. Radiomics is a technically challenging and time-consuming approach and so is unlikely to be usable outside larger tertiary cancer centres without improvements in automated contouring of ROIs.
The study of radiomics could be continued in mRCC as new methods become available such as additional novel texture feature measures added to the existing standard set which may improve discriminating capability between the ICI non-responder and responder classifications. Furthermore, radiomics in mRCC requires substantially more data with a higher number of patients for the analysis to have a greater chance of success. The use of more advanced supervised machine learning techniques such as convolutional neural networks, which learn high-level features, is required. Unsupervised deep learning, automated volume delineation or variational auto-encoders which would sort the images into different regions of low-dimensional space based on intrinsic image properties are possible technological innovations that may enhance the development of a radiomics signature for mRCC. If these advances can be integrated into such an analysis, then a radiomics signature for mRCC patients treated with immunotherapy may be more feasible. Biomarkers of response to nivolumab in advanced kidney cancer 12 © 2021 Canadian Urological Association Fig. 1A. Kaplan Meier-curve of the progression-free survival (PFS) for the whole study population. Median PFS 10.8 months (range 1.4-48.8).