{'fraction_non_alphanumeric': 0.03597093580325775, 'fraction_numerical': 0.01660811242414564, 'mean_word_length': 5.388421321091558, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Performance of Conventional and Machine-Learning Approaches for the Diagnosis of Tumor Recurrence on MRI after Radiation Therapy of Brain Metastases

Objective: To compare the performance of conventional and machine- learning approaches for the diagnosis of tumor recurrence after radiation therapy of brain metastases. Methods: 184 symptomatic patients with solitary metastatic brain lesions treated with radiation therapy were enrolled in a monocentric retrospective study from June 2013 to May 2018. The diagnosis was tumor recurrence (n=71) and radiation necrosis (n=113) using as reference standard expert-consensus derived from pathology and long-term follow-up. 37 potential predictors were recorded at the time of radiological progression (7-15 months after therapy): 6 clinical features and 31 imaging features including 20 radiomics features derived from standard of care 3D T1-gadolinium sequences. We compared four approaches (A, B, C, D): expert report using MRI sequences without (A) and with delayed- contrast MRI (TRAM) sequences (B), 11 non-Radiomics imaging features alone (C) and a signature combining variables selected using unsupervised machine-learning algorithms (D), training:validation sets: n=144:40 pts). Results: Overall (n=184), approaches B and C (using TRAM sequence alone) reached comparable performances with respective AUCs [95% CI] of 78.7% [72.3%-85.1%] and 76.8% [70.3%-83.3%]. Both significantly outperformed approach A with AUC [95% CI] of 57.4% [50.7%-64.1%] (DeLong’s test, p-value=10-7). In the validation set (n=40), the signature reached an AUC [95% CI] of 92% [87%-97%]. Conclusion: A quantitative analysis of TRAM sequence seems the best approach for the diagnosis of recurrent tumor after radiation therapy. It is parsimonious, objective and less time-consuming than interpreting all sequences. A signature derived from the analysis of standard of care 3D T1-gadolinium sequence showed promising results that warrant prospective validation.


Introduction
Brain metastases are the most frequent brain tumors in adults and represent about 25% of brain masses [1,2]. Stereotactic Radiosurgery (SRS), a well-established treatment option, involves delivering a high dose of focal radiation to the tumor [3]. SRC provides increased local control and survival advantage compared to whole brain radiotherapy [2,4], but has a greater risk of Radiation Necrosis (RN), typically occurring months after the treatment [5]. Another major concern after treatment in Tumor Recurrence (TR). Distinguishing these two conditions is pivotal, as the treatment options differ markedly. While RN is mainly treated initially with dexamethasone and in persistent cases with bevacizumab [6,7], TR is treated with surgery, further radiation, or systemic therapy [3]. appearance on imaging, with an enlarging enhancing region and increased vasogenic edema on post gadolinium T 1 -weighted magnetic resonance imaging (MRI) and T 2 -weighted FLAIR MRI, respectively [3]. In an effort to avoid invasive diagnostic techniques (such as biopsy), more sophisticated modalities such as perfusion MRI, MR spectroscopy, and Positron Emission Tomography (PET) have been studied to help distinguish RN and TR, but there is limited availability of those modalities compared to more conventional imaging methods such as contrast enhanced MRI [8][9][10][11][12][13]. Furthermore, none of these modalities offer sufficient sensitivity or specificity in clinical settings [14][15][16]. Hence, identifying reliable diagnostic parameters, preferably using routine MRI sequences to avoid more invasive procedures such as biopsy or isotopic examination, would benefit a great number of patients and physicians facing this challenge.
Considering the physiological differences between TR and RN, there might be subtle differences in texture appearance in medical images. Computational medical imaging, known as radiomics, involves the analysis and translation of medical images into quantitative data [17], and could identify such potential differences. Promising studies have demonstrated quantifiable textural differences in gray levels and histogram-oriented gradients between RN and TR, which are not obvious to human observers; These include both local-level features (within a tumor niche) and global-level features (global composition of the region of interest) [18][19][20][21]. For instance, a T1/T2 mismatch in lesion margins has been correlated with RN [22]. Mesh-like diffuse enhancement and rim enhancement with feathery indistinct margins have been associated with RN, while TR was associated with focal solid nodules and solid uniform enhancement with distinct margins [23]. Analyzing differences between edges, as well as level and spot patterns within the lesions associated a soap bubble pattern with RN [24,25]. RN was associated with a diffuse pattern characterized by periventricular white matter changes, while TR was associated with hyper/hypo-intensities indicative of hemorrhagic changes on Gd-T1-w, T2-w, and FLAIR MR Modalities [24,26]. The importance of intensity normalization in cases of multi-scanner and multi-site acquisitions have been emphasized to normalize these quantitative features [27,28].
The objective of this study is to compare the performance of conventional and machine-learning approaches using quantitative imaging biomarkers alone or in combination (signature) for the diagnosis of tumor recurrence after radiation therapy of brain metastases. As an ancillary study, we assessed the feasibility of radiomic analysis of routine MR imaging sequences to identify computer-extracted texture differences between RN and TR.

Study design
Our aim was to compare the performance of conventional and machine-learning approaches using quantitative imaging biomarkers alone or in combination (signature) for the diagnosis of tumor recurrence after radiation therapy of brain metastases. As an ancillary study, we used a machine-learning algorithm to combine a subset of radiomic characteristics to build a model which best distinguished RN from TR. To this end, the algorithm was trained on a training cohort (as described below) and validated in a testing set. The ultimate goal of the study was to determine the best non-invasive MRI technique and sequence that can be used on routine MRI to distinguish RN from TR. Figure 1 shows a summary of the study design.

Approaches
We compared four approaches referred as A, B, C, and D: (A) expert report using conventional sequences (3D T1 gadolinium sequence, ADC, CBV); (B) expert report using conventional and delayed-contrast MRI (TRAM) sequences; (C) quantitative imaging biomarkers alone. (D) As an ancillary study, a signature was defined using an unsupervised machine-learning derived combination of 20 additional radiomics features extracted from standard of care 3D T1 gadolinium sequences.

Study population
184 symptomatic patient after 7-15 months from the end of radiotherapy for solitary metastatic brain lesions treated with  This study was approved by the local institutional review board and was in accordance with the declaration of Helsinki. We have sent a letter of information to the attention of patients and all patients were informed of their enrollment in the study, in compliance with the institutional and national guidelines.
Inclusion criteria were: (i) patients who initially responded to SRS (tumor shrinkage), (ii) but months later showed evidence of radiation necrosis or tumor recurrence. In all cases, standard clinical diagnostic MRI including T 2 -weighted FLAIR and T 1 -weighted pre-gadolinium MRI and T 1 -weighted post-gadolinium MRI (DOTAREM (acide gadotérique) guerbet villepinte France) were obtained in this order: an MRI of planning of the radiotherapy which corresponds to the MRI of the diagnosis of the metastasis, then an MRI of the reaugmentation of the contrast (suspicion of radionecrosis or progression) between 7 to 15 months after SRS.
Exclusion criteria were: (i) patients for whom the acquired MRI showed a significant movement artifact; (ii) patients who did not receive a gadolinium injection.
The reference standard for the diagnosis of radiation necrosis was the consensus of experts using imaging and/or biopsy. The initial diagnosis of radiation necrosis or tumor recurrence was made clinically by a committee of experts (two neuro-oncologists, 3 radiotherapists, 2 neurosurgeons). Final diagnosis was made either after 3 years of follow-up, using MRI and the clinical examination or PET MR choline and the clinical examination and findings (164 patients), or using histopathological findings after surgical resection (20 patients).

Imaging parameters and variables
The MRI acquisition parameters were the same on all acquired MRIs (Table 1). Data included at least a contrast (gadoterate meglumine [Dotarem ® , 0.1mmol/kg injections of gadolinium-based contrast agent]) enhanced three-dimensional T 1 -weighed Fast Spoiled Gradient Recalled (FSPGR) acquisition on 3 Tesla MR Scanners (MR 750, Discovery; General Electric Healthcare). Imaging parameters were set as follows: repetition time: 10.2ms; echo time: 3.4ms; field of view, 22cm; voxel size: 0.8mm×0.8mm×1.2mm. Post-contrast 3DT1 sequences only were used as inputs of the radiomics classifier.
Dynamic susceptibility MRI images were acquired using a first pass gadolinium contrast-enhanced T2*W echo-planar image sequence with 50 measurements. Each measurement ranged from 1.5s to 2s long depending on number of slices. Continuous transverse slices throughout the brain were made with each measurement. The number of slices was dependent on patient size. The image acquisition parameters were as follows repetition time: average 2,008ms (range: 1,690-2,250), echo time: 62.4ms, flip angle: 90°, slice thickness: 4mm with a 0.4mm skip; a matrix of 128 x 128 and a flip angle of 90. A series of 40 multisection acquisitions was acquired at 0.2s interval, field of view: 140 or 150, number of excitations: 1, bolus of Paramagnetic contrast media, gadobenate dimeglumine (Dotarem ® ), at a dose of 0.1mmol/kg was injected contrast bolus at 5ml/s followed by 20ml saline flush at 3ml/s into the cephalic vein catheter using an automatic power injector (medrad MR Injection System). A series of 40 multisection acquisitions was acquired at 0.2s interval. The first 8 acquisitions were performed before the contrast agent was injected to establish a pre-contrast baseline. Figure 2 shows representative images of an axial MRI of a brain lesion used in this study.
We recorded 17 variables including 6 clinical variables (age, primary tumor type, systemic immunotherapy treatment, symptom, surgery or no , final diagnosis) and 11 imaging biomarkers reported by the radiologist (edema, mismatch FLAIR/T1, ratioADC lesion/ADC control, ratioCBV corrected lesion/ CBV corrected control, ratioCBV corrected lesion/CBV corrected control , ratioCBV 95% percentile corrected lesion/CBV 95% percentile corrected control, rHP , PSR , radiological diagnostic hypothesis with all sequences without Delayed Contrast Extravasation MRI generate Treatment Response Assessment Maps (TRAM) , TRAM only, radiological diagnostic hypothesis with all sequences with TRAM). Imaging biomarkers were extracted from 4 MRI sequences: (i) First-pass perfusion or DSC-MRI and a Post-processing of DSC-MRI and DWI images were performed using the Olea MRI analysis software (Brain olea sphere ® ). The perfusion and diffusion plugin generates relative color perfusion MAP and ADC maps and the quantitative parameters are extracted as relative cerebral blood volume (rCBV), relative amplitude of the peak (rHP) and the percentage signal recovery (PSR); (ii) diffusion DWI; (iii) three-dimensional T 1 -weighted post-gadolinium (T1 ce w+) (iv) delayed-contrast MRI (TRAM) and (iv) extraction features radiomics with T1 ce w+ acquisition.

Image analysis
Pre-processing: MR images preprocessing included bias field correction using the N4ITK algorithm from the Advanced Normalization Tools (ANTs) library [29], skull-stripping with the Brain Extraction Tool (BET) of the FSL software (FMRIB's Software Library) [30] and Z-score normalization with a scaling factor of 100. No spatial resampling was performed due to data homogeneity.
Tumor segmentation: Segmentation of the volume of interest, including the contrast-enhanced and necrotic regions, was performed semi-automatically using Olea SphereÒ (Olea Medical). Within a region of interest defined by a trained radiologist (5 year of experience), threshold-based grey level contouring and manual correction were used for the segmentation so that the volume of interest was carefully drawn along the tumor enhancement.

Feature extraction
A total of 100 radiomics features were extracted using the Pyradiomics (version 2.1.2) package [31]. An absolute discretization with a bin size of 64 was chosen. Six features classes were considered: Using principal component analysis, 20 uncorrelated imaging features were generated by using these 100 features and used for subsequent analysis.

Model building
In the training set, the machine-learning classifier reaching the highest performance, or the prediction of the diagnosis was selected as the radiomics signature. The establishment of the classification model was based on the scikit-learn library version 0.20.3 [32], which included two steps applied to the training set: (1) selection of the machine-learning (ML) classifier and feature scaling method and (2) optimization of the remaining hyper-parameters. In (1), a nested cross-validation was used given the moderately-sized dataset and 135ML models combining 9 feature scaling methods (no scaler, Standard Scaler, Min Max Scaler, Max Abs Scaler, Robust Scaler, Quantile Transformer-normal, Quantile Transformer-uniform, Power Transformer-yeo-johnson, Normalizer) and 15 classifiers (Logistic regression, Quadratic Discriminant Analysis, Adaboost, Bagging, Extra Trees, Decision Tree, Gradient Boosting, Random Forest, Extra Tree, Ridge, SGD, BernouilliNB, GaussianNB, KNeighbors, MLP) were compared. The nested cross-validation considered a 5-fold cross-validation in the inner loop for hyperparameters tuning and a 5-fold cross-validation in the outer loop for the model performance evaluation. In step (2), only the combined model showing the lowest generalization error was kept and a ten repeated 5-fold cross-validation was performed. In this second step, a grid search method was implemented to optimize the remaining hyper parameters. Mean sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve (AUC) and their associated standard deviations were calculated as performance metrics.

Evaluation of the independent test set and comparison to human performance
All cases and associated clinical reports were reviewed by a certified neuroradiologist (A.S) with 10 years of experience, who was blinded to the final histopathological interpretation and the final diagnosis. The interpretation of the expert was then compared with that of the radiomics signature.
The performance of each sequence was evaluated (ADC alone, CBV alone, mismatch T2flair/T1, TRAM) and the performance of the radiologist was calculated without TRAM (the case in all centers) and with TRAM (the case of reading at Gustave roussy).
To ensure that comparison between the performances of radiomics and the expert mimicked clinical conditions, neuroradiologists analyzed all available sequences of the imaging exam as done in clinical routine (post-contrast 3DT1 sequences and the other sequences acquired).

Statistical Analysis
Statistical analysis was performed using R software (version 3.6.2). Sensitivity, specificity, accuracy and AUC were used to assess the diagnosis performances of the radiomics model. A MacNemar test (p-value level of significance 0.05) was used to assess the differences between the diagnostic performances of the radiomics classifier and of the readers at each step of the image analysis described above. AUCs were compared using DeLong's test for two correlated ROC curves.

Results
Characteristics of the study population are displayed in Table 2 Expert using conventional sequences In the overall population (n=184), the performance of experts was AUC [95% CI] of (57.4% [50.7%-64.1%]).

Expert using conventional sequences and TRAM
The addition of TRAM sequence led to significant improvement (DeLong's test, p-value = 7.5

Signature
In the training set (n=134), the performance of several models combining all available quantitative and qualitative imaging features quantifying morphological and metabolic characteristics was evaluated. There was minimal improvement in performance by using more than 3 variables, presumably due to the significant correlation between several pairs of variables (Figure 3, Supplementary Figure 1) as well as the excellent performance of TRAM alone.
Using an unsupervised random forest algorithm, the machinelearning algorithm ranked the most important variables for the diagnosis (Figure 4-6, Table 3 and Supplementary Figure 2-4). The performance of each variable included in the signature, as well as the optimal threshold, are displayed in Figure 4, 5 and 7.
The final signature reaching the highest performance in the training set was obtained from a random forest using regression methods in 500 trees and evaluating 3 variables (TRAM, Component 4, rHP) in each split.
In the validation set, the signature explained 38  This correlogram displays correlation between imaging features. The size of the circle shows the strength of the correlation. The color code is used to differentiate negative (red) and positive correlations (blue). Unsupervised clustering was used to regroup correlated variables. This shows that several imaging biomarkers are redundant and provide similar information. Of note, the referenced standard (i.e., diagnosis is tumor recurrence is strongly correlated with the TRAM sequence and the interpretation by a trained radiologist (rad_hypfinal_tram) as well as with the primary tumor type (NSCLC).

Correlation between imaging features
The correlogram shows that several imaging features are providing redundant information (Figure 3, Supplementary Figure  1, 3 and 4). For instance, the primary tumor type is also a predictor: a non-small cell lung cancer is associated with higher probability of tumor recurrence. The referenced standard (presence of tumor recurrence) is correlated with the TRAM and the interpretation by a trained radiologist. It shows that increase TRAM values are correlated with higher probability of tumor recurrence.
The correlogram also shows that some radiomics features are not providing an incremental value since they are significantly correlated with conventional imaging biomarkers. For instance, components 2 and 12 are correlated with CBV.
Of note, in the signature, the second most contributing imaging feature was component 4. Radiomics-derived component 4 was the second most contributive feature and was not correlated with any other known imaging biomarker (TRAM, ADC, and CBV). Hence, Component 4 is providing new information. Of note, Component 4 is derived from principal component analysis and the most contributive feature is GLCM features which is a texture feature.

Discussion
As a standard treatment for brain metastatic lesions, SRS provides superior likelihood of local control by providing focal high  dose radiation. The main complication of SRS, radiation-induced injury referred to as RN, can happen in up to 34% of the cases, up to several months following SRS [33,34]. Distinguishing RN with TR is challenging, considering the similarity of initial clinical and imaging manifestations. As both these conditions are associated with significant morbidity and mortality, and their treatments are fundamentally different, early diagnosis is of great clinical importance. Current methods of making the definitive diagnosis are histological findings and/or long term clinical follow up [13,14,35,36]. Histological confirmation necessitates a biopsy from the lesion, an invasive procedure which might not be the optimal choice for many patients. It also raises the problem of how representative the sample is. On the other hand, diagnosing with clinical and imaging follow up requires at least 4-9 months of observation, which significantly limits the opportunity for early treatment [13,14,35,36].
Currently available non-invasive options to distinguish RN and TR include metabolic imaging (MRS, PET, and SPECT) and functional imaging [37]; however, these techniques are of limited availability, are costly, and have not shown efficient sensitivity and specificity in clinical management [14][15][16]. In this study, we investigated the potential of using routine MRI sequence images in a radiomics model to distinguish RN from TP, compared the performance with that of an expert neuroradiologist, and analyzed the outcomes to propose the superior modality to make this differentiation.
The presence of new contrast on MRI after irradiation of a malignant tumor raises the problem of differential diagnosis between RN and TP. Many criteria have been proposed in the literature based on morphological, functional or metabolic criteria [38]. In the follow-up of irradiated metastases, a transient and moderate increase in lesion volume is possible with a good prognosis. Lesion  morphological characteristics (volume, T2/T1 gadolinium ratio) and perfusion analysis, ADC and/or spectroscopy, TRAM plus and radiomics provide useful tools to approach the diagnosis of RN versus TP. We investigated different modalities and presented the best current tools to differentiate RN from TR based on our findings.
We found the TRAM sequence to be the best sequence for distinguishing RN from TP. TRAM sequence performance not only is superior to all conventional sequences currently in use, but also by utilizing the TRAM sequence, the radiologist's performance increased significantly. Furthermore, TRAM sequence outperformed the radiologist utilizing all the other conventional sequences. TRAM sequence is calculated from delayed contrast MRI, and its advantage in differentiating radiation side effects versus persistent tumoral lesions has been shown in other tissues in recent studies. Peker et al. showed that TRAM sequence shows excellent sensitivity (96%) and positive predictive value (99%) in diagnosing radiation effects in lung cancer [39]. Guez et al. investigated the TRAM sequence in radiation-induced brain vascular malformations mimicking cancer, and reported that TRAM can effectively distinguish the radiation effects and active tumors [40]. In line with literature, our findings suggest that TRAM sequence can be implemented clinically to significantly increase the diagnosis power in distinguishing RN from TP, ultimately to help guide the treatment plan.
The use of Radiomics and machine learning has shown that an imaging biomarker characterizing the texture and spatial heterogeneity of the lesion on a conventional T1 sequence allows to differentiate RN from TR with good performance (Component 4). This parameter is not correlated with other imaging biomarkers and provides independent information. This is of great clinical importance, since it provides additional data to support the diagnosis, and guide the appropriate treatment. The combination of these parameters in a signature has generated satisfactory performances. A prospective validation is warranted to validate these findings, given the sample size of our study.
Our study had some limitations. This is a retrospective monocentric study which could limit the generalizability of the results. However, it was done in a referral oncology center, one of the largest radiotherapy departments in Europe, receiving patients from different backgrounds with a large volume of irradiated brain metastases, with a multidisciplinary team expert in post radiation aspects. There are very limited studies analyzing all the MRI sequences and parameters and our approach provides an original and novel concept in differentiating RN and TP, using MRI as a non-invasive method and optimizing it via machine learning.

Conclusion
We investigated the performance of MRI sequences in distinguishing RN from TP. We utilized multiple parameters extracted from these sequences, created a signature to optimize performance, and compared them together and with radiologist readings. The TRAM sequence is the best sequence for diagnosis of RN. Its performance is superior to all conventional sequences currently in use. The radiologist's performance significantly increases by utilizing TRAM sequence as well. We found a new, independent imaging parameter (component 4) using radiomics and machine learning which can differentiate RN from TR with great performance. The combination of these parameters in a signature has generated satisfactory performances which could benefit from a prospective validation.