{'fraction_non_alphanumeric': 0.03160988278713734, 'fraction_numerical': 0.012643953114854936, 'mean_word_length': 4.974683544303797, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Classiﬁcation of Echocardiogram View using A Convolutional Neural Network

The standard views in echocardiography capture distinct slices of the heart which can be used to assess cardiac function. Determining the view of a given echocardiogram is the ﬁrst step for analysis. To automate this step, a deep network of the ResNet-18 architecture was used to classify between six standard views. The network parameters were pre-trained with the ImageNet database and prediction quality was assessed with a visualization tool known as gradient-weighted class activation mapping (Grad-CAM). The network was able to distinguish between three parasternal short axis views and three apical views to 99% accuracy. 10-fold cross validation showed a 97%-98% accuracy for the apical view subcategories (which included apical two-, three-, and four- chamber views). Grad-CAM images of these views highlighted features that were similar to those used by experts in manual classiﬁcation. Parasternal short axis subcategories (which included apex level, mitral valve level, and papillary muscle level) had accuracies of 54%-73%. Grad-CAM images illustrate that the network classiﬁes most parasternal short axis views as belonging to the papillary muscle level. Likely more images and incorporating time-dependent features would increase the parasternal short axis view accuracy. Overall, a convolutional neural network can be used to reliably classify echocardiogram views.


Echocardiography
Echocardiography, or ultrasound imaging of the heart, can be used to assess both cardiac structure and function and to detect diseases. [1] In echocardiography, the ribs present an obstacle for imaging, as they attenuate most of the incoming waves, thereby blocking transmission of the beam to the heart. [2] To avoid imaging through the ribcage, sonographers will place the transducer at certain imaging windows. [3] These windows include the suprasternal, subcostal, left parasternal, right parasternal, and apical windows. The suprasternal window sits above the rib cage, while the sub-costal window sits below it. For the left parasternal, right parasternal, and apical windows, the transducer is placed in between the ribs. [4] During an examination, a sonographer will rotate the transducer at different windows in order to get different cross sections of the heart. These cross-sections allow the doctor to visualize different heart structures at different angles. [3] Short axis views from the left parasternal window and long axis views from the apical windows constitute two categories of these cross-sections that will be explored in this paper. While many of the same cardiac structures are shown in both categories, long axis apical views and parasternal short axis views are orthogonal from one another. [5] Long axis views taken from the apical imaging window include the apical two chamber (A2C), apical three chamber (A3C), and apical four chamber views (A4C). All three of these apical views include the left ventricle, left atrium, and mitral valve. The right ventricle, right atrium, and tricuspid valve can be visualized in the A4C view, while the aortic valve is shown in the A3C view. Short axis views from the left parasternal imaging window include the mitral (PSAX -MV level), papillary (PSAX -PM level), and apex (PSAX -AP level) views. All three of these views show the left ventricles and partially the right ventricle walls whereas the mitral valve is included in the PSAX -MV level view and the papillary muscles are shown in the PSAX -PM level view. [3] The typical echocardiogram analysis pipeline starts with view identification. The identified view can then be segmented appropriately to identify specific cardiac structures present in the images. For example, the left ventricle can be segmented from the A4C view. From these segmentations, measurements can be calculated related to cardiac structure (such as left ventricle volumes), indices relevant to cardiac function (such as left ventricle ejection fraction) can be measured, and diseases (such as hypertrophic cardiomyopathy) can be detected. [1] New technologies, such as portable ultrasound devices and cloud storage, have increased the need for automated echocardiogram analyses. For portable devices, automatic analysis is needed for non-expert users, and cloud capabilities have allowed for the storage of large databases where manually intensive analyses aren't a viable option. As the analyses of echocardiograms become automatic, there is an increasing need for automatic view classification, as it is the first step in the analysis pipeline.
Automatic classification of echo views is a challenging problem because of the great variability in these clips. [6,7] Different views can look similar to one another; for example, apical view subcategories (such as A2C, A3C, and A4C) are similarly oval shaped, whereas parasternal short axis (PSAX) subcategories (such as PSAX -AP level, PSAX -MV level, and PSAX -PM level) are similarly circular shaped. Additionally, a single clip may fluctuate between two different views if the transducer was placed close to the border between them. Even echos belonging to the same view can appear differently, as differing patient body composition, sonographer expertise, ultrasound machine, and speckle noise can introduce variations within the image quality and background data.

Deep Learning 1.2.1 Convolutional Neural Network
A possible solution for this challenging problem, is a deep learning approach, specifically a convolutional neural network. Convolutional neural networks (CNNs) are commonly used as the target function for image processing with machine learning. [8] An important element of this target function is convolution which is a mathematical procedure that is commonly used for feature extraction in image processing. [9] For example, a vertical edge filter can be convolved with an input image to extract all the vertical lines in an image. This output image of vertical lines is called an activation map because it highlights the parts of the image activated by the given feature. In a convolutional neural network, the target function learns what kernels (features) are most useful for the given task. While filters can describe horizontal/ vertical lines, blobs or other low dimensional features, deep networks have stacked convolutions so that learned filters describe more high dimensional features. In stacked convolutions, filters are convolved over an input image, and the resulting activation maps are stacked along their depth dimensions. Filters are then convolved along these stacked volumes. These filters describe combinations of lower dimensional features. Therefore, filters in deeper convolutional layers generally describe higher dimensional features. [10] For example, in a network trained to classify cars, features at the lower levels may describe vertical and horizontal edges, where those at convolutional layers towards the middle could describe circles and grid patterns, and finally those towards the end layers may describe wheels and tires. Common convolutional neural networks include the Residual Network (ResNet), [11] which is an architecture created by Microsoft research that classifies natural images extremely well and won the ImageNet challenge in 2015 with only a 3.57% error.

Transfer Learning
In deep networks, there are millions of parameters that need to be optimized in the target function. Training the network to find the most optimal features would theoretically require millions of examples to train. [12] This is an issue especially in the medical image processing domain where the datasets are of limited size. A possible solution is transfer learning, which is a technique where a model developed for task a is reused as the starting point for the model on task b. Usually task a is trained on a huge dataset whereas task b has only a limited sized dataset. The purpose is to apply the knowledge learned from a related task as the initialization for another task of limited data. [12] ImageNet, [13] which is a well-known (and very large) database of natural images, is commonly used as the dataset in "task a" for a smaller image classification "task b".

Grad-CAM
Deep network results can be deceiving, as high accuracies do not always indicate reliable networks. For example, a network could have high classification accuracy but base its decision on "incorrect" features due to a bias in the input dataset. In Selvaraju et al., a network was tasked to identify between nurses and doctors. [14] While it performed with high accuracy on the training set, the net was using "incorrect" features such as face and hair to distinguish between nurses and doctor. This was due to a bias in the dataset, as the majority of nurses were female and the majority of doctors were male. For applications in the medical field, it is important to understand how the network makes decisions, and why, or why not, it produces high results. Gradient-weighted class activation mapping (Grad-CAM) is a relatively new visualization tool that can be used to analyze learned networks. [14] Grad-CAM produces coarse heat-maps that highlight the region of an image recognized as a given class. These class discriminative coarse heat-maps provide a visual explanation for how a model made its decision. In short, Grad-CAM uses the gradients flowing back into a convolutional layer to create its heat maps. The gradient size determines which feature maps most greatly affect the score prediction. The feature maps that most affect the score are weighted more heavily. When all the weighted feature maps in the convolutional layer are combined, only the pixels that have a positive influence on the class are kept since these pixels increase the class prediction.
Depending on the convolutional layer chosen, Grad-CAM can highlight different features of interest. While the lower layers are known to have lower dimensional features and small receptive fields, deeper layers have higher dimensional features. [10] Grad-CAM can be used at different layers to assess both low and high dimensional features.

Previous Works
Previous works that have attempted to automatically classify echocardiogram views can be divided into two main categories. The first approach is used in earlier works and involves manually picked features as input to a machine learning classifier. For example, Khamis et al. [7] extracts a region with important spatio-temporal features using a cuboid detector and uses supervised dictionary learning as the classifier. Using this method, the A2C, A3C, and A4C views were classified with 95% accuracy. Many other papers use a support vector machine (SVM) as their machine learning classifier of choice such as in Refs. [15][16][17][18][19][20]. The reader should refer to Penatti et al. [21] for a review on this approach. Manually chosen features may not be optimal for classification. In the second approach, which is found in newer works, deep networks attempt to learn the "optimal" discriminative features. For example, Zhang, Jeffrey, et al. [1] classified six echo views as part of an analysis pipeline using a VGG network and achieved accuracies of 93%-97% for apical views. Zhang later expanded on this initial study to include twenty-three subcategories which included views containing occlusions. [22] Madani, Ali, et al. [23] also utilized a VGG network to classify fifteen echo views and found that low resolution images were classified by the network better than an expert. They achieved frame accuracies of 87%-95% for the PSAX -PM and PSAX -MV levels (which they included as one group), A2C, A3C, and A4C views. Ostvick et al. [24] classified seven echo views using an original CNN architecture and had access to a very large database of images. They achieved 98%-99% accuracies for A2C and A4C clips. Gao, Xioahong, et al. [25] used both hand crafted and network learned features to achieve an overall accuracy of 92.1%. They fused two networks to classify eight views; one learned spatial information from echo clips while the other applied the feature of acceleration (derived from optical flow images) to incorporate temporal information.
In this study, we automatically classify an echocardiogram database into six common views including three apical views (A2C, A3C, A4C) as well as three parasternal views (PSAX -AP level, PSAX -MV level, and PSAX -PM level) (see Figure 1). The A2C, A3C, and A4C views image the heart lengthwise, while the MV, PM, and AP views image the heart along its cross-section. We use a deep learning approach and chose ResNet as our architecture due to its high accuracy in image classification. Additionally, we utilize transfer learning and initialize the network's weights with those trained on ImageNet. We train our network in a ten-fold cross validation scheme and assess the results using Grad-CAM.

Database
The echocardiogram database was composed of clips collected from Assaf Harofeh Medical Center, Carmel Medical Center, Emek Medical Center, Shaare Zedek Medical Center, Soroka Medical Center, Rambam Medical Center, Hillel Yaffe Medical Center, University of Leipzig, Kaplan hospital, Assaf Harofeh Medical Center, and Hadassah Hospital -Har HaTsofim. The database included healthy and abnormal echocardiograms of varying image quality. Depending on the source, data was saved in either DICOM, AVI, JPEG, or MAT files and was manually labeled by one or two experts. The database consisted of 681 echo clips divided among the six classes as follows: 170 clips or 23,351 frames for the A2C views, 180 clips or 23,844 frames for the A3C views, 163 clips or 23,005 frames for the A4C views, 86 clips or 10,822 frames for the PSAX -PM level view, 41 clips or 5,268 for the PSAX -MV level view, and 41 clips or 5,118 for the PSAX -AP level view. The clips' frames were of size 636X434.

Data Preprocessing
For pre-processing, the periphery information in the images were removed in a procedure similar to that in Zhang, Jeffrey et al. [1] In short, pixels that remained static throughout the entire clip were removed. Additionally, colored pixels from each frame were removed by masking out pixels with large standard deviations (>9) over their color channels. A histogram was used to find the standard deviation value that distinguished between gray and colored pixels. Preprocessing was conducted using MATLAB software. [26]

Model Architecture and Training
The network parameters were initialized by weights pretrained on the ImageNet database. [13] During training, updating occurred on all the parameters in the network. For stochastic optimization, the ADAM optimizer was used. [27] The initial learning rate was 0.001 and had a mini-batch size of 95 frames.
The average accuracy from 10-fold cross validation runs was used to approximate the model's accuracy. Classes were split for each fold so that roughly 90% of a specific class was in the training fold, and 10% was in the validation fold. A weighted cross-entropy loss was used to account for the imbalanced dataset. The weights chosen for each class equaled the number of samples in the largest class divided by the number of samples in a given class and were: A2C) 1.02 A3C) 1 A4C) 1.037 PSAX -PM level) 2.20 PSAX -AP level) 4.536 PSAX -MV level) 4.66 respectively. Frames served as inputs to the network and those from the same video belonged to either training or validation and were not mixed between the two categories. Videos were classified to the label predicted for the majority of its frames. Image on the right is the resulting image Grad-CAM images were used to visualize the features learned by the network. To view features at different layers, the layer and basic block for the Grad-CAM classification were specified (see Figure 3). Additionally, the class label was specified in the Grad-CAM code to observe features for different classes. Figure 3. Labels for convolutional layers in ResNet-18. Specific convolutional layers for ResNet-18 are labeled according to their "Layer" and "Basic block". Each of the four "Layers" consist of four convolutional layers with an equal number of output filters (i.e. 64 output filters in "Layer 1"). The "Basic block" label pairs the convolutional layers according to the presence of skip connections (see enlarged figure on the right for illustration of complete "Layer 1" with inclusion of skip connections).

Data Transformations
Transformations to the training dataset simulated changes in gain, zoom, and rotation. Transformations to each image included: conversion to grayscale, a randomly placed crop of 224 x 224, an adjustment of brightness by a random factor between 0 and 2, a random rotation within the ranges of [-1,1] to 224 × 224. This crop ensured that the frame had the right dimensions for ResNet and that the resulting image included most of the frame information.

Software and Hardware
The pytorch library with python 3 was used for creating and training the networks. The ResNet-18 architecture, pretrained ImageNet weights, and data transformations were accessed using the torchvision package for PyTorch. [28] Code used to update weights from certain layers was based on Ref. [29] The Grad-CAM script was based on code written by Kazuto Nakashima and Utku Ozbulak [30,31] The GPU used was a Tesla K20m which had 5120 MB of memory. [32]

RESULTS
A confusion matrix of the averaged results from the 10-fold cross validations show high accuracies >97% for apical (A2C, A3C, A4C) view classification (see Figure 4). Additionally, the confusion matrix indicates that the network rarely classified (<1%) parasternal short axis subcategories (PSAX -PM level, PSAX -MV level, PSAX -AP level) as apical subcategories. Most incorrectly classified parasternal short axis views were predicted as PSAX -PM level. For a given network from the 10-fold cross validations, Grad-CAM heatmaps highlighted main regions of clinical interest (see Figure 5). Particularly, for the parasternal short axis subcategories, heatmaps were centered around the left ventricle.
For a correctly predicted apical four chamber view, Grad-CAM heatmaps from earlier layers of the net learned lowerlevel features such as edges recognizing the septums, valves, and myocardium. While, from the second and third layers of the net, Grad-CAM heatmaps highlighted regions pertaining to the left ventricle walls, the atrial septum, the valve level, and the atrium. In the deepest layers, Grad-CAM heatmaps highlighted clinically relevant features such as the left ventricle and ventricular septum strongly and more exclusively (see Figure 6a). This correctly classified view was predicted as A3C with .002% probability, A2C with .001% probability, and belonging to one of the parasternal short axis levels with 0% probability. The corresponding Grad-CAM images highlight some relevant features for the different apical views but show an almost inverted heatmap for the parasternal short axis views when given the corresponding labels. By using Grad-CAM in earlier layers, the valve level was highlighted strongly for the correct class, weakly for the A3C view, and very negatively for the A2C view (see Figure 6b).  Heatmaps generated from "Layer 4" for the three parasternal short axis labels show an almost inverted heatmap to the correct apical four chamber view label. The valve-level feature is circled in the heatmaps generated from "Layer 3" for the three apical view labels. The network's predictions for the original frame is included for reference.

Published by Sciedu Press
The few misclassified apical views in the given network were either of poor quality, mislabeled, or had frames which fluctuated between two different views (see Figure 7). For misclassified parasternal short axis videos, Grad-CAM im-ages are centered over the left ventricle for the papillary muscle level, but do not highlight clinically relevant features for either the mitral valve or apical levels (see Figure 8).

Classification of Parasternal Short Axis versus Apical views
The network distinguished between parasternal short axis views and apical views extremely well (with >99% accu-racy). Corresponding to this high accuracy, given Grad-CAM heatmaps showed no overlap between the learned parasternal short axis and apical features (see Figure 6b). The two categories were easily distinguishable likely because the classification was "simpler" to learn as these views are quite different from one another; parasternal short axis views are more circular while apical views are more oval. It is important to note that many previous works do not distinguish between the parasternal short axis subcategories; labeling them as one category, [5,6,16,18,33,34] or not including them at all. [1,7,24] The ease of classification between the parasternal short axis category and other apical views should be taken into account when comparing our results to their reported accuracies.

Classification of Apical views
The trained network was able to classify the three apical views very well (to 98% accuracy) with minimal preprocessing (see Figure 4). These reported accuracies are a reliable estimation of the "true" accuracies as they are not merely a consequence of an easy to classify test set; rather, they are an average of the ten network accuracies from the cross validations. In a given network, all misclassified apical views had reasonable explanations, further illustrating the network's reliability (see Figure 7). Due to the mislabeling, fluctuations, and low-quality videos, the "true" accuracy values for apical views may be even higher than those reported. Misclassified clips that were of low quality and/or mislabeled looked more like their predicted view than their manually labeled view. Those clips would likely be unusable for analyses requiring the view corresponding to their manually given labels. Fluctuating clips, such as that in Figure 7a, could be removed from the database since the frame classifications are split between two different classes. A cut-off could be set to determine the percent of frames needed for clip classification to remove fluctuating videos instead of using a majority vote. More experiments would be necessary to determine the appropriate cutoff.
Grad-CAM images showed that the network recognized class-specific features for apical views similar to those used by an expert (see Figure 5). The heat maps for the apical views were centered a bit above the valves, showing a slight preference for the ventricles as opposed to the atrium. Similarly, in Zhang et al., [22] occluding the left ventricle impaired classification more than occlusion of the left atrium. Figure  6a illustrates that the network recognized clinically relevant landmarks. As is expected in deep networks, features learned towards the end of the architecture were more high level than those learned in the earlier layers. Interestingly, the valve level area was highlighted by the Grad-CAM as an important feature for distinguishing between apical subcategories (see Figure 6b) which is similar to the findings in Khamis et al. [7]

Classification of Parasternal Short Axis views
While classification accuracies of the parasternal short axis subcategories were low in comparison with the apical subcategories (see Figure 4), Grad-CAM images showed that the learned features were centered over the "correct" regions of interest for accurately classified videos (see Figure 5). Theoretically, when given infinite data and the proper network architecture, the network should learn the optimal features which may or may not correspond to those features that we believe to be "correct". With that in mind, we make the reasonable assumption that the "correct" region of interest for parasternal short axis views is the area within the left ventricle as it contains either the mitral valves (for PSAX -MV level), papillary muscles (for PSAX -PM level), or an empty space (for PSAX -AP level). To reiterate, the Grad-Cam generated heat maps for correctly classified parasternal short axis views were centered around this region of interest.
Unlike the apical views, parasternal short axis videos that were misclassified by the network included those of good quality (see Figure 8). Misclassified parasternal short axis views were usually labeled as parasternal papillary muscle level (see Figure 4) and Grad-CAM heat maps suggested that the network recognized features relevant for the parasternal papillary muscle level regardless of the correct class (see Figure 8). The over classification of PSAX -PM level views may be a consequence of our imbalanced dataset; the network was trained on double the amount of PSAX -PM level views (in comparison to PSAX -AP level and PSAX -MV level views). More parasternal papillary level views may have caused the network to learn features pertaining to this class or to classify more frames as this class. Possibly, optimizing the weights for the cross entropy would balance out the effect of the different sized classes, although, the network likely needs to train on more parasternal short axis images. Assuming that clips were evenly distributed among the classified views, other deep networks for echo classification (excluding Gao et al. [25] who incorporated manually extracted features) had a minimum of 13,000 frames per class. [1,23,24] This is greater than the number of frames for all parasternal short axis subcategories in our database.
Since parasternal short axis views resemble one another and are more challenging to visually classify (even manually) than the apical views, it may be necessary to incorporate information from movement instead of relying solely on structure. The current network considered spatial features only, but time may be an important feature for the classification of parasternal short axis views. Gao et al. [25] showed that after incorporating a convolutional network using optical flow-based images (representing acceleration), accuracies for PSAX -MV level images increased from 68.8% to 75%, though accuracy for PSAX -PM level views did not change.
Time dependent features could be incorporated by extracting our network features to use as input to a long short-term memory network (LSTM) over a given clip. Likely movement based features are less important for apical view classification as the network was able to recognize the valve level feature which Khamis et al. [7] found to be an important spatial and temporal feature. Additionally Kumar et al. [17] achieved very high accuracies ( 88%) for parasternal short axis subcategories and very low accuracies for apical subcategories ( 65%) when using motion features as input to a learned dictionary and SVM.

Additional Comparisons to Other Works
A limitation of this study is that Grad-CAM provides a qualitative analysis and its results are only shown for one of the cross validation networks. Nevertheless, the Grad-CAM images support the results of the averaged 10-fold net accuracy and help promote the believability of the network. To our knowledge, we are the only paper of an echo classification network which visualizes within the layers. Other deep network echo classification papers may use t-SNE [1,23] or very general occlusion experiments [23] that do not attempt to explain what distinguishing features the network has learned. As previously stated, these deep network papers (excluding those which include manual features) either ignore the parasternal short axis view [1,24] or include all subcategories as one class. [23] An exception is Zhang et. al, who does include the three parasternal short axis subcategories and achieved 76% accuracy for the PSAX -PM level and PSAX -MV level views (no PSAX-AP level views were in the test set). [22] While these are still not high accuracies, their improved results may be a result of a more varied dataset. For training, they extracted frames from a dataset which contained approximately four times the amount of PSAX -PM level and PSAX -MV level videos. In comparison to papers which use machine learning classifiers, our high accuracies were obtained without incorporating ECG information (unlike in Ref. [17] ) and without using hand crafted features (unlike in Ref. [16] ) or dictionary learning (like in Ref. [7] ) that would make adaptation to other views difficult.

CONCLUSION
In conclusion, a deep learning approach was shown to classify echocardiogram views with minimal preprocessing. Accuracies for apical views (A2C, A3C, A4C) were particularly high. Grad-CAM images were used to analyze the believability of the network. These images illustrated that the network learned features similar to what an expert would use to classify the views. Unlike in previous works, Grad-CAM was used to analyze features in the lower layers, and showed that the network recognized features with anatomical significance. Misclassified apical views were typically mislabeled, of low quality, or fluctuated between views while misclassified parasternal short axis views were typically labeled as parasternal papillary muscle level views. Classification of parasternal short axis views would likely improve with more training images.