{'fraction_non_alphanumeric': 0.029974470641237424, 'fraction_numerical': 0.009791259948941283, 'mean_word_length': 4.911931818181818, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Radiomics for Brain Tumor Detection and Classification from Multi-Sequence MRI

Glioma constitutes 80% of malignant primary brain tumors and is usually classified as HGG and LGG. The LGG tumors are less aggressive, with slower growth rate as compared to HGG, and are responsive to therapy. Tumor biopsy being challenging for brain tumor patients, noninvasive imaging techniques like Magnetic Resonance Imaging (MRI) have been extensively employed in diagnosing brain tumors. Therefore automated systems for the detection and prediction of the grade of tumors based on MRI data becomes necessary for assisting doctors in the framework of augmented intelligence. In this paper, we thoroughly investigate the power of Deep ConvNets for classification of brain tumors using multi-sequence MR images. We propose novel ConvNet models, which are trained from scratch, on MRI patches, slices, and multi-planar volumetric slices. The suitability of transfer learning for the task is next studied by applying two existing ConvNets models (VGGNet and ResNet) trained on ImageNet dataset, through fine-tuning of the last few layers. LOPO testing, and testing on the holdout dataset are used to evaluate the performance of the ConvNets. Results demonstrate that the proposed ConvNets achieve better accuracy in all cases where the model is trained on the multi-planar volumetric dataset. Unlike conventional models, it obtains a testing accuracy of 95% for the low/high grade glioma classification problem. A score of 97% is generated for classification of LGG with/without 1p/19q codeletion, without any additional effort towards extraction and selection of features. We study the properties of self-learned kernels/ filters in different layers, through visualization of the intermediate layer outputs. We also compare the results with that of state-of-the-art methods, demonstrating a maximum improvement of 7% on the grading performance of ConvNets and 9% on the prediction of 1p/19q codeletion status.


Introduction
Magnetic Resonance Imaging (MRI) has become the standard non-invasive technique for brain tumor diagnosis over the last few decades, due to its improved soft tissue contrast 1,2 . Gliomas constitute 80% of all malignant brain tumors originating from the glial cells in the central nervous system. Based on the aggressiveness and infiltrative nature of the gliomas the World Health Organization (WHO) broadly classified them into two categories, viz. Low-grade gliomas (LGG), consisting of low-grade and intermediate-grade gliomas (WHO grades II and III), and high-grade gliomas (HGG) or glioblastoma (WHO grade IV) 3 . Diffuse LGG are infiltrative brain neoplasms which include histological classes astrocytomas, oligodendrogliomas, and oligoastrocytomas and World Health Organization (WHO) grade II and III neoplasms 3 . Although LGG patients have better survival than those with HGG, the LGGs are found to typically progress to secondary GBMs and eventual death 4 . In both cases a correct treatment planning (including surgery, radiotherapy, and chemotherapy separately or in combination) becomes necessary, considering that an early and proper detection of the tumor grade can result in good prognosis 5 .
Histological grading, based on stereotactic/surgical biopsy test, is primarily used for the management of gliomas. Typically the highest grade component, among the histopathology samples obtained, is used to predict the overall tumor grade. Gliomas being heterogeneous, sometimes histopathology samples collected from different parts of the same tumor exhibit different grades. Since pathologists are not provided with the entire delineated tumor during examination, it is likely that the highest grade component may be missing in the biopsy sample. This is called the biopsy sampling error [6][7][8] , and can potentially result in wrong clinical management of the disease. Moreover there exist several risk factors in the biopsy test, including bleeding from the tumor and brain due to the biopsy needle; causing severe migraine, stroke, coma and even death. Other associated risks involve infection or seizures 9,10 .
MR imaging, on the other hand, has the advantage of being able to scan the entire tumor in vivo and can demonstrate a strong correlation with histological grade. It is also not susceptible to sampling error, and inter-and intra-observer variability. In this context multi-sequence MRI plays a major role in the detection, diagnosis, and management of brain cancers in a non-invasive MICCAI BraTs 2017 dataset 39 being used for training and testing of models.
In addition to tumor grading, the prediction of 1p/19q codeletion status serves as a crucial molecular biomarker towards prognosis in LGG. It is found to be related to longer survival, particularly for oligodendrogliomas which are more sensitive to chemotherapy. Such noninvasive prediction through MRI can, therefore, lead to avoiding invasive biopsy or surgical procedures. Predicting 1p/19q status in LGG from MR images using ConvNet was reported 40 . The network was first trained on a brain tumor patient database from the Mayo Clinic, containing a total of 159 LGG cases (57 non-deleted and 102 codeleted) and having preoperative postcontrast-T 1 and T 2 images 41 . The model was also trained and tested on 477 2D MRI slices extracted from the 159 patients, with 387 slices being used for training and 90 slices (45 non-deleted and 45 codeleted) during testing. A test accuracy of 87.7% was obtained.
Studies by the TCGA have established that LGGs can be grouped into three robust molecular classes on the basis of IDH1/2 mutations and 1p/19q co-deletion. The variants have been reported to differ with respect to tumor margins and internal homogeneity. The T 2, FLAIR mismatch sign was found to be associated with a survival profile similar to that of the IDH-mutant 1p/19q-non-codeleted glioma subtype 42 , and more favorable to that of the IDH-wild type gliomas (which present outcome similar to WHO grade IV glioblastomas).
In this paper we exhaustively investigate the behaviour and performance of ConvNets, with and without transfer learning, for noninvasive studies of gliomas, involving (A) detection and grade prediction (into low-(Grades II and III) and high-(Grade IV) brain tumors (LGG and HGG), and (B) classification of LGG with/without 1p/19q codeletion, from multi-sequence MRI.
Tumors are typically heterogeneous, depending on cancer subtypes, and contain a mixture of structural and patch-level variability. Prediction of the grade of a tumor may thus be based on either the image patch containing the tumor, or the 2D MRI slice containing the image of the whole brain including the tumor, or the 3D MRI volume encompassing the full image of the head enclosing the tumor. While in the first case only the tumor patch is necessary as input, the other two cases require the ConvNet to learn to localize the ROI (or VOI) followed by its classification. Therefore, the first case needs only classification while the other two cases additionally require detection or localization. Since the performance and complexity of ConvNets depend on the difficulty level of the problem and the type of input data representation, we introduce three kinds viz. i) Patch-based, ii) Slice-based, and iii) Volume-based data, from the original MRI dataset by introducing the sliding window concept, and employ these over the two experiments. Three ConvNet models are developed corresponding to each case, and trained from scratch. We also compare two state-of-the-art ConvNet architectures, viz. VGGNet 43 and ResNet 23 , with parameters pre-trained on ImageNet using transfer learning (via fine-tuning).
The main contributions of this research are listed below.
• Adaptation of deep learning to Radiomics, for the non-invasive prediction of tumor grade followed determination of the 1p/19q status in Low-Grade Gliomas, from multi-sequence MR images of the brain.
• Prediction of the grade of brain tumor without manual segmentation of tumor volume, or manual extraction and/or selection of features.
• Conceptualization of "Augmented intelligence", with the application of deep learning for assisting doctors and radiologists towards decision-making while minimizing human bias and errors.
• Development of novel ConvNet architectures viz. PatchNet, SliceNet, and VolumeNet for tumor detection and grade prediction, based on MRI patches, MRI slices, and multi-planar volumetric MR images, respectively.
• New framework for applying existing pre-trained deep ConvNets models on multi-channel MRI data using transfer learning. The technique can be further extended to tasks of localization and/or segmentation on different MRI data.

Results
The ConvNet models were developed using TensorFlow, with Keras in Python. The experiments were performed on the Intel AI DevCloud platform, having a cluster of Intel Xeon Scalable processors. The quantitative and qualitative evaluation of the results are elaborated below.

Quantitative evaluation
We use (i) leave-one-patient-out (LOPO), and (ii) holdout (or independent) test dataset for model validation. While only one sample is used for testing in the LOPO scheme, at each iteration, the remaining are employed for training the ConvNets. The process iterated over each patient. Although LOPO test scheme is computationally expensive, it allows availability of more data as required for ConvNets training. LOPO testing is robust and well-suited to our application, with results being generated for each individual patient. Therefore, in cases of misclassification, a patient sample may be further investigated. In holdout or  independent testing scheme either a portion of the training data that has never been used for training or a separate test dataset is used during model validation.
Training and validation performance of the three ConvNets were measured using the following two metrics.
Accuracy is the most intuitive performance measure and provides the ratio of correctly predicted observations to the total observations. F 1 Score is the weighted average of Precision and Recall, which are defined as T P T P+FP and T P T P+FN , with T P, T N, FP, and FN indicating the numbers of true positive, true negative, false positive and false negative detections. In the presence of imbalanced data one typically prefers F 1 Score over Accuracy because the former considers both false positives and false negatives during computation.

Case study-A: Classification of low/high grade gliomas
The dataset preparation schemes, discussed in Section Dataset preparation, were used to create the three separate training and testing data sets. The ConvNet models PatchNet, SliceNet, VolumeNet, were trained on the corresponding datasets using Stochastic Gradient Descent (SGD) optimization algorithm with learning rate = 0.001 and momentum = 0.9, using mini-batches of size 32 samples generated from the corresponding training dataset. A small part of the training set (20%) was used for validating the ConvNet model after each training epoch, for parameter selection and detection of overfitting.
Since deep ConvNets entail a large number of free trainable parameters, the effective number of training samples were artificially enhanced using real-time data augmentation -through some linear transformation such as random rotation (0 o − 10 o ), horizontal and vertical shifts, horizontal and vertical flips. Also we used Dropout layer with a dropout rate of 0.5 in the fully connected layers and Batch-Normalization to control the overfitting. After each epoch, the model was validated on the corresponding validation dataset.
Training and validation Accuracy and loss, and F 1 Score on the validation dataset, are presented in Fig. 1 for the three proposed ConvNets (PatchNet, SliceNet, and VolumeNet), trained from scratch, along with that for the two pre-trained ConvNets (VGGNet, and ResNet) fine-tuned on the TCGA-GBM and TCGA-LGG datasets. The plots demonstrate that VolumeNet gives the highest classification performance during training, reaching maximum accuracy on the training set (100%) and the validation set (98%) within just 20 epochs. Although the performance of PatchNet and SliceNet is quite similar on the validation set (PatchNet -90%, SliceNet -92%), it is observed that SliceNet achieves better accuracy (94%) on the training set. The performance of two the pre-trained models (VGGNet and ResNet) exhibit similar results, with both achieving around 85% accuracy on the validation set. All the networks reached a plateau after the 50th epoch. This establishes the superiority of the 3D volumetric level processing of VolumeNet.
LOPO testing results: After training, the networks were evaluated on the holdout test set employing majority voting. Each patch or slice from the test dataset was from a single test patient in the LOPO framework, and was categorized as HGG or LGG. The class with maximum number of slices or patches correctly classified was indicative of the grade of the tumor. In case of equal votes the patient was marked as "ambiguous".
The LOPO testing scores are displayed in Table. 1. VolumeNet is observed to achieve the best LOPO test accuracy (97.19%), with zero "ambiguous" cases as compared to the other four networks. SliceNet is also found to provide good LOPO test accuracy (90.18%). Both the pre-trained models show similar LOPO test accuracy as PatchNet. This is interesting because it demonstrates that with a little fine-tuning one can achieve a test accuracy similar to that by the patch-level ConvNet trained 4/15 from scratch on a specific dataset. Therefore fine-tuning of a few more intermediate layers can lead to very high test scores with little training. Table 2 compares the proposed ConvNets with existing shallow learning models in literature, used for the same application but requiring additional feature extraction and/or selection from manually segmented ROI/VOI, in terms of classification accuracy. Ref. 19  Testing results on holdout dataset: Trained networks were also tested on an independent test dataset (MICCAI BraTS 2017 database) as discussed in Section Brain tumor data. The confusion matrix for each of the networks are shown in Fig. 2. VolumeNet performs the best on the holdout set (accuracy = 95.00%). The other two models PatchNet and SliceNet, which were trained from scratch, also demonstrate good classification performance. On the other hand, the fine-tuned models VGGNet and ResNet perform poorly on the independent test dataset. Comparison is made with two recently reported methods 37,38 , used for the same problem on same or different datasets. The comparison results are given in Table 3. It is observed that VolumeNet performed the best on the holdout dataset, achieving 7% improvement in the accuracy as compared to state-of-the-art method 38 using the same dataset. Note that cross-validation was used in the compared models 37,38 for evaluating performance, with training done on a part of the same data on whose holdout portion testing was evaluated. On the other hand, we trained our models on a different dataset (from TCIA) and tested it on the BraTs dataset. This validates the robustness of our models as compared to existing methods.

Case study-B: Classification of LGG with/without 1p/19q codeletion
We trained the best performing model, i.e. VolumeNet, for the classification of LGG with/without 1p19q codeletion. The Mayo Clinic database used for the task contains T 1C and T 2 MRI sequences for each patient. VolumeNet was trained on the multi-planar volumetric database, preprocesed from the raw 3D brain volume images as described in Section Dataset preparation. Training performance of the network, in terms of training and validation Accuracy & loss, are presented in Fig. 3.
Comparison was made with a state-of-the-art method 40 , which is also based on deep learning and uses same dataset. The test performance on the holdout dataset is reported in Table 4. Here again our model VolumeNet achieves 9% more accuracy than the compared method, over the same dataset. The improvement is due to the incorporation of volumetric information through multi-planar MRI slices.

Runtime analysis
The total time required for training each network for 100 epochs is presented in Table 5, averaged over several runs. This further corroborates that multi-planar and slice level processing can learn to generalize better than a patch level network, although at the expense of higher computational time.

Qualitative evaluation
The ConvNets were next investigated through visual analysis of their intermediate layers. The performance of a ConvNet depends on the convolution kernels, which are the feature extractors from the unsupervised learning process. Visualizing the output of any convolution layer can help determine the description of the learned kernels. Fig. 4 illustrates the intermediate convolution layer outputs (after ReLU activation) of the proposed SliceNet architecture on sample MRI slices from an HGG patient. The visualization of the first convolution layer activations (or feature maps) [ Fig. 4(b)] indicates that the ConvNet has 6/15  learned a variety of filters to detect edges and distinguish between different brain tissues like white matter (WM), gray matter (GM), cerebrospinal fluid (CSF), skull and background. Most importantly, some of the filters could isolate the ROI (or the tumor); on the basis of which the whole MRI slice may be classified. Most of the feature maps generated by the second convolution layer [ Fig. 4(c)] mainly highlight the tumor region and its subregions; like enhancing tumor structures, surrounding cystic/necrotic components and the edema region of the tumor. Thus the filters in the second convolution layer learn to extract deeper features from the tumor by focusing on the ROI (or tumor).
The texture and shape of the tumor get enhanced in the feature maps generated from the third convolution layer [ Fig. 4(d)]. For example, small, distributed, irregular tumor cells get enhanced (one of the most important tumor grading criteria called "CE-Heterogeneity" 44 ). Finally the last layer [ Fig. 4(e)] extracts detailed information about more discriminating features, by combining these to produce a clear distinction between images of different types of tumors.

Discussion
An exhaustive study was made to demonstrate the effectiveness of Convolutional Neural networks for non-invasive, automated detection and grading of brain tumors from multi-sequence MR images. Three novel ConvNet architectures were developed for distinguishing between HGG and LGG. Three level ConvNet architectures were designed to handle images at patch, slice and multi-planar modes. This was followed by exploring transfer learning for the same task, by fine-tuning two existing ConvNet models. The scheme for incorporating volumetric tumor information, using multi-planar MRI slices, achieved the best test accuracy of 97.19% in the LOPO mode and 95% on the holdout dataset for the classification of LGG/HGG (Case study A). In Case study B an accuracy of 97% was obtained for the classification of LGG/HGG (case study A). In case study B an accuracy of 97.00% was obtained for the classification of LGG with/without 1p/19q codeletion status on the holdout dataset.
Visualization of the intermediate layer outputs/feature maps demonstrated the role of kernels/filters in the convolution layers in automatically learning to detect tumor features closely resembling different tumor grading criteria. It was also observed that existing ConvNets, trained on natural images, performed adequately just by fine-tuning their final convolution layer on the MRI dataset. This investigation allows us to conclude that deep ConvNets could be a feasible alternative to surgical biopsy for brain tumors.
Diagnosis from histopathological images is also considered to be the "gold standard" in this domain. HGGs are characterized by the presence of pseudopallisading necrosis (necrotizing cell-devoid region radially surrounded by lined-up tumor cells) and microvascular proliferation (enlarged new blood vessels in the tissue) 45 . The LGGs, on the other hand, exhibit a visual smoothness with cells spread evenly throughout the tissue. Automated glioma grading, through image analysis of the slides, serves to complement the efforts of clinicians in categorizing into low-(Grade II) and high-(Grade III, IV) gliomas (LGG and HGG).

Methods
In this section we provide a brief description of the data preparation at three levels of resolution, followed by an introduction to convolutional neural networks and transfer learning.

Brain tumor data
For the classification of low/high grade gliomas (Case study A), the models were trained on the TCGA-GBM 46 and TCGA-LGG 47 datasets downloaded from The Cancer Imaging Archive (TCIA) 48  The ConvNet model used for classification of 1p/19q codeletion status in LGG (Case study B) was trained on the brain tumor patient database from Mayo Clinic, containing a total of 159 LGG patients (57 non-deleted and 102 codeleted) having preoperative postcontrast-T 1 and T 2 images downloaded from TCIA 41 . A total of 30 samples (15 non-deleted and 10 codeleted) were randomly selected from the data at the beginning, as a test set, and was never shown to the ConvNet during its training. The remaining 129 samples were used for training the model.
Sample images of the two glioma grades (LGG/HGG), and LGG with and without 1p/19q codeletion are shown in Fig. 5(a), (b), respectively. It can be observed from the figure that it is very hard to discriminate in each case, based only on the phenotypes visible to the human eye. Hence abstract features learned by the deep layers of a ConvNet are expected to be helpful in noninvasively differentiating between them. Besides, the use of large public domain datasets can allow more clinical impact as compared to controlled and dedicated prospective image acquisitions.  The datasets were aligned to the same anatomical template, skull-stripped, bias field corrected and interpolated to 1mm 3 voxel resolution.

Dataset preparation
Although the TCGA-GBM and TCGA-LGG datasets consist MRI volumes, we cannot propose a 3D ConvNet model for the classification problem; mainly because the dataset has only 262 HGG and 199 LGG patients data, which is considered as inadequate to train a 3D ConvNet with a huge number of trainable parameters. Another problem with the dataset is its imbalanced class distribution i.e. about 35.72% of the data comes from the LGG class. Therefore we formulate 2D ConvNet models based on the MRI patches (encompassing the tumor region) and slices, followed by a multi-planar slice-based ConvNet model that incorporates the volumetric information as well.
Applying ConvNet directly on the MRI slice could require extensive downsampling, thereby resulting in loss of discriminative details. The tumor can be lying anywhere in the image and can be of any size (scale) or shape. Therefore classifying the tumor grade from patches is easier, because the ConvNet learns to localize only within the extent of the tumor in the image. Thereby the ConvNet needs to learn only the relevant details without getting distracted by irrelevant details. However it may lack spatial and neighborhood details of the tumor, which may adversely influence grade prediction. Although classification based on the 2D slices and patches often achieves good accuracy, the incorporation of volumetric information from the dataset can enable the ConvNet to perform better.
Along these lines, we propose schemes to prepare three different sets viz. (i) patch-based, (ii) slice-based, and (iii) multi-planar volumetric, from the TCIA datasets.

Patch-based dataset
The slice with the largest tumor region is first identified. Keeping this slice in the middle, a set of slices before and after it are considered for extracting 2D patches containing the tumor regions using a bounding-box. This bounding-box is marked, corresponding to each slice, based on the ground truth image. The enclosed image region is then extracted.
We use a set of 20 slices for extracting the patches. In case of MRI volumes from HGG (LGG) patients, four (ten) 2D patches [with a skip over 5 (2) slices] are extracted for each of the MR sequences. Therefore a total of 210 × 4 = 840 HGG and 75 × 10 = 750 LGG patches, with four channels each, constitute this dataset. Although the classes are still not perfectly balanced, this ratio is found to be good enough in the scenario of this enhanced dataset.
In spite of significant dissimilarity visible between contiguous MRI slices at a global level, there may be little difference exhibited at the patch level. Therefore patches extracted from contiguous MRI slices look similar, particularly for LGG cases. Fig. 6 depicts a set of 10 patches extracted from contiguous MR slices of an LGG patient. This can lead to overfitting in the ConvNet. To overcome this problem we introduce a concept of static augmentation by randomly changing the perfect bounding-box coordinates by a small amount (∈ {−5, 5} pixels) before extracting the patch. This results in improved learning and convergence of the network.

Slice-based dataset
Complete 2D slices, with visible tumor region, are extracted from the MRI volume. The slice with the largest tumor region, along with a set of 20 slices before and after it, are extracted from the MRI volume in a sequence similar to that of the patch-based approach. While for HGG patients 4 (with a skip over 5) slices are extracted, in the case of LGG patients 10 (with a skip of 2) slices are used.

Multi-planar volumetric dataset
Here 2D MRI slices are extracted along all three anatomical planes, viz. axial (X-Z axes), coronal (Y -X axes), and sagittal (Y -Z axes), in a manner similar to that described above.

Convolutional neural networks
Convolutional Neural Networks (ConvNets) can automatically learn low-level, mid-level and high-level abstractions from input training data in the form of convolution filter weights, that get updated during the training process by backpropagation. The inputs percolating through the network are the responses of convoluting the images with various filters. These filters act as detectors of simple patterns like lines, edges, corners, from spatially contiguous regions in an image. When arranged in many layers, the filters can automatically detect prevalent patterns while blocking irrelevant regions. Parameter sharing and sparsity of connection are the two main concepts that make ConvNets easier to train with a small number of weights as compared to dense fully connected layers. This reduces the chance of overfitting, and enables learning translation invariant features. Some of the important concepts, in the context of ConvNets are next discussed.

Layers
The fundamental constituents of a ConvNet consist of the input, convolution, activation, pooling and fully-connected layers. The input layer receives a multi-channel brain MRI patch/slice denoted by I ∈ R m×w×h , where m is the number of channels, w and h represent the resolution of the image. The convolutional layer takes the image or feature maps as input, and performs the convolution operation between the input and each of the filters to generate a set of activation maps. The output feature map dimension, from a convolution layer, is calculated as w out /h out = (w in /h in −F+2P) S + 1, where w in and h in are the width and height of the input image, w out and h out are the width and height of the effective output. Here P denotes the input padding, the stride S = 1, and F is the kernel size of the neurons in a particular layer. Output responses of the convolution and fully connected layers pass through some nonlinear activation function, such as a Rectified Linear Unit (ReLU) 50 , for transforming the data. ReLU, defined as f (a) = max(0, a), is a popular activation function for deep neural networks due to its computational efficiency and reduced likelihood of vanishing gradient. The pooling layer follows each convolution layer to typically reduce computational complexity by downsampling of the convoluted response maps. Max pooling enables selection of the maximum feature response in local neighborhoods, and thereby enhances translation invariance. The features learned through a series of convolutional and pooling layers are eventually fed to a fully-connected layer, typically a Multilayer Perceptron. Additional layers like Batch-Normalization 51 reduce initial covariate shift. Dropout 52 is used as regularizer to learn a better representation of the data.

Loss
The cost function for the ConvNets is chosen as binary cross-entropy (for a two-class problem) as where n is the number of samples, y i is the true label of a sample and f i is its predicted label.

Transfer learning
Typically the early layers of a ConvNet learn low-level image features, which are applicable to most vision tasks. The later layers, on the other hand, learn high-level features which are more application-specific. Therefore, shallow fine-tuning of the last few layers is usually sufficient for transfer learning. A common practice is to replace the last fully-connected layer of the pre-trained ConvNet with a new fully-connected layer, having as many neurons as the number of classes in the new target application. The rest of the weights, in the remaining layers, of the pre-trained network are retained. This corresponds to training a linear classifier with the features generated in the preceding layer. However, when the distance between the source and target applications is significant then one may need to induce deeper fine-tuning. This is equivalent to training a shallow neural network with one or more hidden layers. An effective strategy 27 is to initiate fine-tuning from the last layer, and then incrementally include deeper layers in the tuning process until the desired performance is achieved.

ConvNets for Brain tumor Grading
This section introduces the three ConvNet architectures, trained on the three level brain tumor MR data sets, along with a brief description of the fine tuning of existing models.

Three level architectures
We propose three ConvNet architectures, named PatchNet, SliceNet, and VolumeNet, which are trained from scratch on the three datasets prepared as detailed in Section Dataset preparation. This is followed by transfer learning and fine-tuning of these networks. The ConvNet architectures are illustrated in Fig. 7. PatchNet is trained on the patch-based dataset, and provides the probability of a patch belong to HGG or LGG. SliceNet gets trained on the slice-based dataset, and predicts the probability of a slice being from HGG or LGG. Finally VolumeNet is trained on the multi-planar volumetric dataset, and predicts the grade of a tumor from its 3D representation using the multi-planar 3D MRI data. As reported in literature 25 , smaller size convolutional filters produce better regularization due to the smaller number of trainable weights; thereby allowing construction of deeper networks without losing too much information in the layers. We use filters of size (3 × 3) for our ConvNet architectures. A greater number of filters, involving deeper convolution layers, allows for more feature maps to be generated. This compensates for the decrease in size of each feature map caused by "valid" convolution and pooling layers. Due to the complexity of the problem and bigger size of the input image, the SliceNet and VolumeNet architectures are deeper as compared to the PatchNet.

Fine-tuning
Pre-trained VGGNet (16 layers), and ResNet (50 layers) architectures, trained on the ImageNet dataset, are employed for transfer learning. Even though ResNet is deeper than VGGNet, the model size of ResNet is substantially smaller due to the usage of global average pooling rather than fully-connected layers. Transferring from the non-medical to the medical image domain is achieved through fine-tuning of the last convolutional block of each model, along with the fully-connected layer (top-level classifier). Fine-tuning of a trained network is achieved by retraining on the new dataset, while involving very small weight updates. The adopted procedure is outlined below.
• Instantiate the convolutional base of the model and load its pre-trained weights.
• Replace the last fully-connected layer of the pre-trained ConvNet with a new fully-connected layer, having single neuron with sigmoid activation.
• Freeze the layers of the model up to the last convolutional block.
• Finally retrain the last convolution block and the fully-connected layers using Stochastic Gradient Descent (SGD) optimization algorithm with a very slow learning rate.
Since the base models were trained on RGB images, and accept single input with three channels, we train and test them on the slice-based dataset involving three MR sequences (T 1C, T 2, FLAIR). The T 1C sequence was found to perform better than T 1, when used in conjunction with T 2 and FLAIR. Although running either of these two models from scratch is very expensive, particularly on CPU, the concept of fine-tuning just the last few layers could be easily accomplished.