{'fraction_non_alphanumeric': 0.027450257921886514, 'fraction_numerical': 0.009764185703758291, 'mean_word_length': 5.266378066378066, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Comparative Evaluation Of Three Methods Of Automatic Segmentation Of Brain Structures Using 426 Cases

Segmentation of brain structures in a large dataset of magnetic resonance images (MRI) necessitates automatic segmentation instead of manual tracing. Automatic segmentation methods provide a much-needed alternative to manual segmentation which is both labor intensive and time-consuming. Among brain structures, the hippocampus presents a challenging segmentation task due to its irregular shape, small size, and unclear edges. In this work, we use T1-weighted MRI of 426 subjects to validate the approach and compare three automatic segmentation methods: FreeSurfer, LocalInfo, and ABSS. Four evaluation measures are used to assess agreement between automatic and manual segmentation of the hippocampus. ABSS outperformed the others based on the Dice coefficient, precision, Hausdorff distance, ASSD, RMS, similarity, sensitivity, and volume agreement. Moreover, comparison of the segmentation results, acquired using 1.5T and 3T MRI systems, showed that ABSS is more sensitive than the others to the field inhomogeneity of 3T MRI.


INTRODUCTION
Automatic segmentation of brain structures in magnetic resonance imaging (MRI) has been a focus of attention in recent years due to the large datasets that need to be analyzed in different applications. Performing manual segmentation of these structures engages large datasets and is both costly and time consuming. This approach is also limited in its reproducibility but is considered the ground truth for its lesser susceptibility to imaging artifact and noise. Therefore, as automatic segmentation methods are developed, comprehensive comparison studies are needed to evaluate performance of these methods and investigate the advantages and disadvantages of each method as a suitable replacement for the manual approach. This will provide opportunity to refine these methods and adapt them for use in various conditions. The hippocampus is considered as a key structure in studying temporal lobe epilepsy [1] and Alzheimer's disease [2]. In both conditions, the shape and volume of hippocampus has been used as a biomarker for diagnosis. To extract volumetric and shape features from MRI, proper segmentation of this structure is required. Because of its irregular shape, small size, and some poorly defined edges, automatic segmentation of this structure is challenging [3]. Hence such methods must be carefully scrutinized before being validated.
Nowadays, FreeSurfer [4] and HAMMER [5] are well-known tools for automatic segmentation. Prelabeled atlas images are mapped to the subject's images and then refined to label each part of the brain. Alternatively, other methods and software have evolved such as Automatic Brain Structure Segmentation (ABSS) [6] and LocalInfo [7] that are particularly designed to segment subcortical brain structures including the hippocampus. These methods use pattern recognition techniques in addition to registration algorithms to automate the segmentation process. They have outperformed FreeSurfer and HAMMER in a previous study [3,31,32] using MRI studies obtained from 195 cases. A limitation of that study which involved data acquired by both 1.5T and 3T MRI systems was not separately analyzed. Because field inhomogeneity artifact depends on the magnetic field strength [8] and is therefore different in 1.5T and 3T scanners the performance of individual automatic segmentation methods may have been variably affected.
In this work, we use MRI data obtained from 424 cases to evaluate comparative performance of three different automatic segmentation methods in segmenting the hippocampus. Four performance measures are calculated to evaluate the agreement of the automatic and manual segmentation results. We also investigate the effect of magnetic field strength (1.5T vs. 3T) on the performance of these methods.

LITERATURE REVIEW
Volumetric analysis of various brain structures is becoming increasingly common, as they can be used to diagnose diseases and monitor disease progression. The hippocampus is of importance, as it is implicated in several disease states, such as epilepsy and Alzheimer's disease. However, volumetric analysis of the hippocampus remains primarily manual. Automatic extractions are difficult due to poor definition of borders on MRI. Prior knowledge of the regional distribution, shape, atlas templates, among other aspects, are helpful in determining borders and to aid in the process of refining automatic segmentation methodology. Moreover, analysis of large cohorts is mandatory to overcome doubt pertaining to any unanticipated shortcomings of these methods.
Manual segmentation methods require a great deal of user interaction with commitment to all processing steps [9] to avoid transgressing borders inappropriately. With semiautomatic methods, users employ geometric shape and global localization constraints by marking points on the MRI or by drawing contours [9]. These methods can be based on Deformable Models or Atlases allowing for the use of geometric information already complied from real patient datasets to limit user interaction and reduce manual effort [9]. In fully automated methods, the statistical geometric information is obtained automatically from a reference atlas [9]. Some approaches use Single-Atlas, Multi-Atlas, or Probabilistic-Atlas. Others use deformable models or classifiers [9].
Deep convolutional neural networks (CNN) have emerged as an alternative to manual segmentation [33][34][35]. Ataloglou and colleagues [30] developed a CNN-based hippocampal segmentation (Hippodeep) that was compared to manual segmentation and FreeSurfer 6.0 [10]. This study was conducted with 200 healthy patients from seven locations across Canada [10]. The algorithm was shown to produce stable and reproducible data compared to other methods of segmentation. The approach requires additional testing in clinical populations to prove its promise for tracking volumetric changes in longitudinal studies [10].
In a joint international effort to harmonize existing manual segmentation protocols for the hippocampus, the European Alzheimer's Disease Consortium (EADC) and the Alzheimer's Disease Neuroimaging Initiative (ADNI) developed a Harmonized Protocol for Hippocampal Segmentation (HarP) [11]. The method developed is highly stable within and between human tracers and provides higher agreement than local manual segmentation protocols. However, only 20 MRI studies were segmented as this method was also very time consuming [11].
An ADNI dataset was used to evaluate a novel segmentation method based on iterative local linear mapping (ILLM). This method, which uses representative and local structure-persevered feature embedding, eliminates the need for registration [12]. It uses a semi-supervised deep autoencorder (SSDA) to exploit an unsupervised deep autoencorder and the local structurepreserved feature to transform the MR patch [12]. This method was tested with 135 subjects and was shown to be superior to the current patch-based multi-atlas segmentation (PBMAS) approach [12].
Semiautomatic methods, such as Fast Segmentation Through SURface Fairing (FASTSURF), are based on mesh processing techniques which use certain smoothness constraints to reconstruct hippocampal shape from a few manually delineated cross-sections of the hippocampus [13]. It was validated by comparing it to fully manual segmentation methods using different datasets and comparing it to FreeSurfer, an automated segmentation method [13]. FASTSURF results in highly accurate segmentations and require less effort in delineation than fully manual segmentation methods making it a promising tool [13]. Prior study of this method has used datasets of 12, 135 and 80 subjects [13] and deserves further confirmation with a larger cohort in other hands to promote its use.
Another semiautomated segmentation method (AdaBoost), was able to reliably analyze large datasets faster than a manual method [14]. MR images of 161 subjects taken from the ADCS Donepezil/Vitamin E clinical study were used to compare AdaBoost to manual segmentation [14]. Each method detected baseline differences in the hippocampus of patients who retained mild cognitive impairment throughout the clinical trial and those whose impairment progressed to Alzheimer's Disease [14].
An automatic segmentation method developed by Chupin, et al was created by adding hybrid constraints to an existing algorithm to make it fully automatic [15]. This method was based on the Iterated Conditional Modes (ICM) algorithm [15] and used global probabilistic atlas information to automatically initialize deformation and determine the bounding boxes and initial objects. It was better than semiautomatic segmentation and atlas-based segmentation in a small cohort of 16 patients [15]. FreeSurfer has been used as an automatic segmentation method in several papers. Research conducted by the University of Melbourne compared the utility of manual segmentation using two protocols for FreeSurfer and found that, although there was high reliability and agreement between protocols, more refinement was necessary and a dataset larger than one with 27 patients required [16]. FreeSurfer and FSL/FIRST were further compared to manual segmentation in quantifying hippocampal and amygdalar volumes with FreeSurfer found better correlated to manual segmentation [20].Hippocampal segmentation by FreeSurfer-Subfields was also found superior to seven other automated methods in 105 ischemic stroke patients and 59 healthy participants [17]. However, when FreeSurfer was evaluated against manual tracing to determine hippocampal volume in a longitudinal study examining atrophy in middle and early old age, it was found to overestimate volume compared to manual segmentation [21]. Four segmentation methods: FreeSurfer, ANIMAL, nonlinear patch-based method, and nonlinear patchbased with and without error correction were compared using the ADNI dataset; the latter resulted in the most promising results [22]. In the pediatric population, FreeSurfer and FSL-First were compared to manual segmentation for use with the hippocampus and amygdala and neither were shown to have particularly strong agreement with manual segmentation [23].
The MAPS-HBSi method demonstrated superior reproducibility to other segmentation methods including manual segmentation, FreeSurfer, AdaBoost and FSL/FIRST when assessing atrophy rates of the hippocampus [18]. Another supervised algorithm, RUSBoost, showed a better dice index when compared to ADABoost, Random Forest, and FreeSurfer [19].
As witnessed above, there is substantial variance in the results comparing these various automated segmentation methods according to a variety of metrics applied and there is disagreement regarding the ultimate utility of any particular method, such as Freesurfer. Beyond this comes an assessment of subfield analysis in the course of disease or natural development as it may involve the hippocampus or other entity. A recent study analyzed the MRI of individuals aged 6 to30 years [24]. Two semiautomated segmentation methods, Automated Segmentation of Hippocampal Subfields (ASHS) and Advanced Normalization Tools (ANTs), were compared to manual segmentation and showed that ASHS outperformed ANTs. Hippocampal subfield volume was also studied using ASHS in 32 patients addressing normal and early-stage nondementia Parkinson's disease [25]. Automatic segmentation methods of the hippocampus were also used to distinguish between Alzheimer's, mild cognitive impairment and normal aging [26].
. In previous work, the MRI studies of 195 patients were segmented by manual and automated methods, using LocalInfo, FreeSurfer, HAMMER, and ABSS [3]. ABSS was found to generate the most accurate results, followed by LocalInfo and FreeSurfer [3]. HAMMER scored the least accurate and was therefore excluded from this study.
In this work, we analyze a very large cohort using three automatic segmentation methods: ABSS, LocalInfo, and FreeSurfer. The methods are compared and evaluated on their performance in hippocampal segmentation using the MRI studies of 426 patients. Additionally, four performance measures are calculated to evaluate individual strengths against those of manual segmentation.

Preprocessing
For all T1-weighted images, the brain was extracted from the scalp, skull, and dura using automated and manual skull stripping [3]. This procedure is not a prerequisite but improves the results of the automated segmentation methods used in this study. Both right and left hippocampi of each subject were manually segmented using MRIcro by a research assistant trained for this task [3,27] and validated by a neuroscientist.

Automated Segmentation
In this study, we used three methods for automatically segmenting the hippocampi: ABSS, LocalInfo, and FreeSurfer.

ABSS
Automatic Brain Structures Segmentation (ABSS) is a pattern-recognition algorithm that is used for segmentation of brain structures in MR images. The ABSS method involves two stages. First, the shape of the desired brain structure is represented using geometric moment invariants (GMIs) in 8 scales. Each scale uses an artificial neural network (ANN) to approximate shape and signed distance functions of the structure. Input features of the ANN are the GMIs, voxel intensities and coordinates and the output feature is the signed distance function. Second, the outputs of all the ANNs in the first stage are combined and another ANN is designed to classify the voxels of the image based on whether they were inside or outside of the brain structure of interest [6]. In our study, we use this tool to train ANNs based on the shape (voxel intensities and coordinates) and signed distances of the hippocampal regions for the standard atlases or the user's training data. Finally, the subjects are processed and segmented based on the learned patterns of the training data. This package is freely accessible at [28].

LocalInfo
In the Local Information-Based Multiple-Atlas method (LocalInfo), multiple atlases are used to improve the likelihood of approximating an MR image with the desired atlas counterpart, resulting in improved segmentation. The desired atlas is selected for a specific brain region of interest and is utilized for location information extraction and for shape model. During initialization, nonrigid registration is used and a transformation is applied to the label maps. The most similar atlas label map is identified and both principal shapes and mean shapes are extracted. Additionally, a mean distance function for each structure is used for location information extraction. This is followed by a fine-tuning step, which is used to extract details of those brain structures that are not extracted from their primary shapes [7]. We use the LocalInfo method to segment the right and left hippocampi using information-based multiple atlases and affine registration to find the coordinates of each structure in each atlas mask. Finally, fuzzy classification, tissue-type information extraction, optimization of shape parameters, and transformation to label maps through nonrigid registration are employed for the segmentation task.

FreeSurfer
FreeSurfer [4] is an atlas-based segmentation tool designed for automated segmentation of brain structures including subcortical and cortical regions. It is a freely available [29] open source set of automated, robust, and accurate tools used to analyze neuroimaging data. It uses nonlinear alignment of the subject's image to the atlas labels (template matching) and then optimizes the transferred label map based on the subject's brain structures. To generate accurate models of various brain structures from MR images for segmentation, separate models for each structure at each point in space are generated. This enables the heterogeneity within a structure to be accounted for while keeping the distinction between pallidum and caudate, resulting in sharper and more informative distributions that make segmentation clearer. FreeSurfer also considers stereotypical spatial relationships between brain structures, by using the Markov Random Field (MRF) model and extending it to be spatially nonstationary. This allows for the ability to model the likelihood that the amygdala is above the hippocampus separately from the likelihood that it is below it. This creation of more sophisticated models enables the use of manually labeled training set images to bootstrap a whole-brain segmentation procedure.

Segmentation Evaluation
To evaluate the performance of segmentation methods, we calculated eight different performance measures by comparing the resulting label map of the hippocampi with the manually segmented hippocampi as the ground truth.
The Dice coefficient is defined as: where A and M are the voxels labeled as hippocampus by the automated and manual segmentation methods, respectively, and N represents the number of voxels in each set. This measure can have values in the range of 0.0 (i.e., no overlap) to 1.0 (i.e., complete overlap). Precision is calculated as the ratio of the number of voxels truly labeled as the hippocampus, i.e., true positive (TP), to all voxels labeled as the hippocampus, which include the true positive voxels and the voxels labeled falsely as the hippocampus, i.e., false positive (FP) voxels, This measure also ranges from 0.0 (totally inaccurate detection) to 1.0 (fully accurate detection). It should be noted that a precision value of 1.0 does not necessitate a value of one for dice coefficient. As an example, assume half the hippocampus is labeled as hippocampus by a method. For this case, the precision will be 1.0 while the dice coefficient is equal to 1/1.5 = 0.667. A related measurement is similarity, which includes FN, the voxel proportion of the hippocampus which segments as a background, in the denominator.
Sensitivity is calculated using the following equation: The Hausdorff distance is defined as: where and are the coordinates of the points (voxels) on the automated (i.e., A), and manually (i.e., M), segmented structures, respectively. Here, ||.|| denotes a norm function that can be the Euclidian distance. The Hausdorff distance decreases as the overlapping of the two structures increases. The rational absolute value degree (RAVD) is based on the volume of the segmented structures, = − where V is the volume of the underlying structure. This measure is important in evaluating the methods when the biomarker is the volume of the structure. For example, in Alzheimer's Disease studies, where the volume of the hippocampus is one of the measures used to diagnose the disease [2], a segmentation method with lower RAVD value is more desirable. The RMS measure is used to evaluate whether the quantity of segmentation varies or not by giving a distance metric using surface-to-surface geometries.
SA is the set of automatic surface voxels, whereas SR is the set of manual surface voxels. The equation, d 2 (a,SA), is the nearest Euclidian distance from a point on the surface (a) to the surface (SA) squared. A related measure is ASSD, which is the average distance of all the contour points that have been automatically segmented to the closest contour point that has been manually traced. The notations for ASSD are the same as the ones for RMS.

RESULTS AND DISCUSSION
FreeSurfer, ABSS, and LocalInfo were used to automatically segment the hippocampal structure of 426 T1-weighted MRI studies. The eight performance measures discussed in Section 2 were calculated to compare the automatic segmentation results with the manually segmented hippocampi. Table 1 shows performance measures from 6 patients as a representation of the entire dataset. In Fig. 1 and 2, the boxplots of these measures are shown and in Fig. 3 and 4, the scatterplots of the data are shown. Table 2 shows that ABSS results for hippocampal volumes approximate more closely the results from manual segmentation. Table 3 is a standard ANOVA table that shows a p value of < 0.05; the null hypothesis is rejected identifying the three methods of automatic segmentation as having significantly different results.       Table 3: Standard ANOVA table using data from 426 patients and three automatic segmentation methods. The source of the variability is shown in the source column. The sum of squares (SS) from each of the sources is depicted in the third column and the degrees of freedom (df) in the fourth. The fifth is the mean of squares (MS), which is the ratio of the sum of the squares/degrees of freedom. The sixth column is the F-statistic (F), the ratio of the mean squares, and the last column is the pvalue from the cumulative distribution of F.
The ABSS method outperforms the other two with all eight similarity measures showing superiority. Moreover, RAVD is greater than zero for all three methods suggesting the segmented hippocampi are larger than the actual size. In other words, the estimated volumes are biased to larger values. However, the segmented volume by ABSS is the closest to the actual volume while the one by FreeSurfer is the furthest. The segmentation performance of LocalInfo stays between the two other methods but closer to FreeSurfer.
Performance measures are altered by field inhomogeneity induced by the magnet in the 1.5T and 3T MRI. For example, the Dice coefficient and precision values of FreeSurfer improve slightly when using the data of 3T MRI. On the other hand, these performance measures decrease slightly for ABSS when 3T MRI is used. Also, the Hausdorff distance of all three methods increase when using 3T MRI data.