{'fraction_non_alphanumeric': 0.033629078420148825, 'fraction_numerical': 0.012664567830566686, 'mean_word_length': 4.904731727925644, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Analysis of functional neural codes of deep learning models

Deep neural networks (DNNs), the agents of deep learning (DL), require a massive number of parallel/sequential operations. This makes it extremely challenging to comprehend DNNs' operations and hinders proper diagnosis. Consequently, DNNs cannot be readily used in high-stakes domains, in which incorrect decisions can lead to catastrophic failures. Therefore, to build more reliable DNNs/DL to be deployed in high-stakes real-world problems, it is imperative that we develop proper analysis tools that will allow us to better understand DNNs' internal operations underlying their decision-making. Here, we used the self-organizing map (SOM) to analyze internal codes of DL models associated with their decision-making. Our analyses suggest that hidden layer activation patterns can be mapped onto a finite number of patterns and are correlated with DL predictions, raising the possibility that they could serve as functional codes of DL models. Encouraged by this observation, we further used SOM to estimate input features coded in hidden layers, analyzed the effects of adversarial inputs to better understand characterized internal representations' evolution and adversarial perturbations' propagation in DL models.


Introduction
Deep learning (DL) can train artificial agents (deep neural networks, DNNs) with examples to perform a wide range of real-world problems [1,2,3], removing the necessity of human experts' detailed instructions. DNNs can outperform humans in multiple domains including computer vision and games even without instructions. However, DNNs rely on large numbers of linear and nonlinear operations, making their operations incomprehensible. Without understanding their operations, we cannot conduct timely predictions of failures or proper diagnosis. Thus, we cannot safely deploy current DNNs in the real world [4,5]. Furthermore, DL is vulnerable to adversarial perturbations that are imperceptible changes crafted to disrupt DL models' decisions-making [6,7,8].
Therefore, to utilize DL's capability in high-stakes domains, it is imperative that we build more transparent and safer DL models. Earlier studies had tackled this necessity and aimed to investigate DNNs' internal operations that determine their decisions/predictions, which fall into two categories. The first line of studies estimated the influence of inputs on DNNs' decisions. Specifically, several studies utilized gradient-based analysis [9,10,11], layerwise relevance propagation assigned individual neurons' relevance using backpropagation-like rules [12], and occlusion analysis measured the influence of input patches [13]. The second line of studies focused on analyzing hidden layer representations (HLRs). The feature visualization [14,15] inverted DL models to identify optimal features that can drive certain hidden layers.
TCAV used examples with and without human-annotated concepts to train a secondary model to learn concept vectors [16]. Once the secondary model is trained, it can evaluate the influence of concept vectors on DL models. Further, a few studies measured direct relationships between HLRs' and DNNs' decisions [17,18].

arXiv:2205.10952v1 [cs.LG] 22 May 2022
As DNNs successively transform representations from a layer to another, HLRs in each layer represent intermediate stages of DNNs' operations, and thus, characterizing them can shed a light on their internal operations underlying their decisions. However, DNNs have a large number of hidden units in each layer, and their responses vary highly (due to diverse input patterns and nonlinear operations in the DL models), making the characterization of HLRs extremely challenging. Then, how do we characterize HLRs without knowing the layer-specific rules? One possibility is to compare HLRs from the same layer with one another to capture HLRs' underlying patterns, each of which could reflect DNNs' internal codes essential to their operations.
With this possibility in mind, we turned to self-organization map (SOM), an unsupervised competitive learning algorithm [19,20], to analyze HLRs of DNNs. We tested 4 internal layers of 3 popular DL models, and our SOM-based analysis suggests that HLRs can be mapped onto a finite number of internal codes linked to labels of inputs, raising the possibility that SOM can identify the functional codes (FnC) of DNNs. Encouraged by this observation, we further analyzed the FnC obtained by SOM. Specifically, we employed model inversion to obtain input features that correspond to the internal codes identified by SOM and probed how perturbations induced by adversarial perturbations propagate through DNNs.

Functional codes of DNNs
In this section, we describe our analyses of DNNs' functional codes (FnC). Section 2.1 discusses the details of our approach, and Section 2.2 describes our analysis of DL models' FnC.

SOM-based analysis of HLRs
We assumed that FnC (i.e. internal codes contributing to DNNs' decision-making) should satisfy two necessary conditions. First, the number of internal codes of DNNs should be finite and significantly smaller than that of potential inputs. In principle, one input is mapped onto a FnC of DNN (in the hidden layer), which is then linked to DNNs' decision. We note that the mapping between inputs and FnC should be many-to-one. If DNNs utilize one-to-one mapping, each input is mapped onto a distinct FnC and then an desired output given during training. This mapping can retrieve the classes of training examples perfectly. However, with this one-to-one mapping, an untrained input will be mapped onto a non-used FnC, whose link to outputs is unknown. That is, untrained inputs are linked to random predictions, making generalization of training impossible. By contrast, if many inputs are mapped onto a single (robust) FnC, DNNs can make proper predictions on untrained inputs by using previously constructed links between the robust FnC and their corresponding answers, as long as the inputs are mapped onto the correct FnC. Conversely, for DNNs to generalize their answers, FnC should reflect input features relevant to desired tasks, which leads to our second condition. Second, if the FnC reflect the essential inputs features, they would be naturally correlated with inputs' labels/classes. In addition, when an input is gradually transformed into DNN's answer, we speculate the correlations to be higher, as the layer goes deeper (closer to the outputs).
We note that HLRs can be considered as noisy FnC in DNNs. As DNNs' operations rely on the weighted summation, the central limit theorem suggests that low-level noises can be effectively averaged out in DNNs. That is, multiple HLRs could correspond to the same FnC. Then, how do we identify proper FnC from HLRs? If HLRs are noisy FnC, we can aggregate similar HLRs and infer functional codes from them via 'effective' averaging. SOM is a supervised competitive learning algorithm that can identify the underlying patterns of high dimensional input data. Specifically, SOM computes functional distance between inputs and finds exemplary inputs. Thus, we hypothesized that the exemplary inputs, identified by SOM, could be DNNs' FnC which the noisy HLRs would code.

Training of SOM
SOM consists of units distributed over 2-dimensional grids. Each unit stores a single weight vector, whose length is the same as that of an input. During training, each input is compared to weights stored in the units, and the best matching unit (BMU) is identified. Then, the weights of BMU and its neighbors are updated according to the learning rule (Eq. 1). This update rule moves the weights of BMU and its neighbors towards the current input, but in different degrees. Notably, SOM has two properties. First, the weights of SOM tend to become similar to the inputs and can represent exemplary inputs after training. Second, neighboring units store similar weights, and consequently the distance between the units reflect the similarity between weights. That is, after training, the similarity between two inputs can be measured with the distance between their corresponding BMUs. We used these properties to analyze HLRs of DNNs.  Figure 1: Schematics of SOM-based analysis. We display ResNet18 as an example. The same approach was applied to ResNet50 and VGG19.
, where s denotes training index; u and v denote the indices of the BMU and the unit to be updated; where α is a learning rate; x is an input; where D = ||x − W u ||; is added for numerical stability.
In the experiments, SOMs were trained using the validation images in the ImagNet [21]. Once they were trained, we estimated BMUs corresponding to the validation images and used them to identify FnC. We used an open source python library QuickSom [22] to construct SOM using square grids and periodic boundary conditions. Table 1 shows the hyperparameters selected for the experiments, and the terminologies are adopted from QuickSom [22]. Table 1: We list the parameters selected for SOM. m and n denote the height and width, respectively. In the experiment, we used a square grid (m = n). σ and α denotes their initial values, and they decay over time. We used 'exp' decay mode supported by Quicksom [22] Parameter value m (height) 10-40 n (width) 10-40 σ 5 α 0.01 decay exp

DL models used in the experiment
In this study, we analyzed HLRs of three DL models (ResNet18, ResNet50 and VGG19 with batch normalization) [23,24]. Specifically, we fed them the ImageNet validation set and measured hidden layer activations to obtain HLRs. As ResNets consist of 4 composite blocks (termed as 'layer' in the original paper), we tested the outputs of these 4 blocks. VGG19 contains 5 max pooling layers that estimate the spatially invariant responses (more than other hidden layers). We tested the outputs from 4 last max pooling layers. In this experiment, we used Timm (an open source library) [25] to build the three models pretrained on ImageNets; the pretrained weights are obtained from the torchvision via Timm's API.
The outputs of ResNets' layers and VGG19's max pooling layers are 3-dimensional tensors (i.e., arrays), since each filter in these layers produces a 2-dimensional feature map. As SOM accepts 1-dimensional vectors, we flatten them into 1-dimensional vectors. Additionally, the naive flattening makes HLRs' dimension quite high, and thus, we shortened their dimension by representing 2-dimensional feature maps with their average activations. For instance, when a hidden layer consists of N m × m convolutional filters, the naive flattening will convert its outputs to a 1-dimensional tensor whose length is N × m × m. Using the average pooling, we shortened its length to N (see Fig. 1). In this study, we used this approximation for the sake of simplicity and discussed the possibility of extending it in Discussion. Further, we normalized the magnitudes of HLRs by dividing their norms.

Analysis of DL models
We used FnC obtained from SOM to study how DL models code their inputs. In the experiments, we first studied correlations between input features and functional codes and then adversarial perturbations' influence on DL models.

Input features encoded in functional codes
Inspired by the notion that DL models can learn input features that are most relevant to their predictions, we speculated FnC could reflect input features essential to DL's operations. To address this possibility, we used model inversion to infer the input features linked to functional codes. Specifically, we minimized the cosine similarity between observed HLRs and SOM's weights by using 'Lucent' [26], an open source Pytorch library developed for model inversion.

Adversarial perturbations on hidden layers
It is well documented that DL models are vulnerable to various types of adversarial perturbations [6,7,8,27,28], but due to DL models' complexity, our knowledge of adversarial perturbations and their influence on DL models remains poor. To gain insights into underlying reasons behind DL models' vulnerabilities to adversarial perturbations, we estimated how adversarial perturbations modulate FnC in DNNs. If adversarial perturbations can induce strong perturbations, HLRs with and without them would be significantly different [29]. With this possibility in mind, we used projected gradient descent (PGD) [30] to create adversarial images from 500 randomly selected ImageNet validation images and compared their HLRs using SOM. In the experiments, the distance between two BMUs, which corresponds to normal and adversarial inputs, was estimated depending on the maximum intensity of perturbations.

Results
Classification can be considered a function that maps data points in high dimension space onto data points in low dimension space. Thus, it seems natural to assume that functional codes of DL models would be finite and mediate information related to input data's classes (i.e., labels). Inspired by the fact that SOM has been used to learn the similarity between high dimensional inputs, we used SOM to examine HLRs' ability to satisfy these two conditions.
SOM consists of units distributed over 2D grids. After training, each unit stores a single weight vector. In the inference mode, SOM compares an input with all units' weight vectors and determines the best matching unit (BMU). One notable property of SOM is that the similarity between inputs can be measured by the distance between the corresponding BMUs. If SOM is trained with HLRs, similar HLRs can be mapped onto the same or neighboring BMUs. That is, if HLRs correspond to a finite number of functional codes, they would be mapped onto a small number of BMUs, each of which encodes crucial information of input data. Therefore, we estimated the density of BMUs and their correlations with the classes of the inputs.

HLRs as neural codes underlying DL models' operations
We analyzed HLRs of 3 DL models using SOM, which are ResNet18, ResNet50 and VGG19. ResNet contains 4 composite layers (CL), and VGG19 contains 5 max-pooling layers (ML). For ResNets18 and 50, the outputs of 4 CLs were chosen as HLRs. For VGG19, the outputs of 4 last MLs were chosen as HLRs. These CL and MLs consist of multiple feature maps, each of which produces 2-dimensional outputs. As such, their outputs are 3-dimensional vectors (or tensors). As SOM accepts 1-dimensional inputs, the 3-dimensional vectors need to be flattened. Given the size of DL models, the dimension of flattened vectors is often quite big. To limit the size of SOM's inputs, we averaged the outputs of each feature map to map the feature maps' outputs onto a single scalar value (Fig. 1). For instance, ResNet18's first CL's output layer produces 64 56-by-56 feature maps, and the dimension of HLRs is reduced to 64 (Fig.1). In the first set of experiments, we fed 50,000 ImageNet validation images to ResNet18.
In the experiments, HLRs were normalized to have unit length, and each SOM was independently trained for a single hidden layer. It should be noted that we updated SOMs after the presentation of each HLR (that is, the batch size is 1) for 5 epochs (25,000 updates). Fig. 2A shows the moving averages of loss during 25,000 updates ( Fig. 2A). The sliding window size was fixed to be 1000 updates. As shown in the figure, the losses in all 4 CLs decreased rapidly in the beginning and hit local minima around 1,000 and 10,000 updates, suggesting that HLRs are homogenous. We observed the equivalent results in ResNet50 (Fig. 2B) and VGG19 (Fig. 2C).
After training SOMs, we again fed ImageNet's validation images to the three models to obtain their corresponding BMUs and noted that some nodes were frequently chosen as BMUs. To quantify this observation, we estimated the densities of BMUs using Gaussian kernels [31] (Fig. 2D-F). As shown in the figures, BMUs in the three models are inhomogeneously distributed, and many of them are located in the vicinity of a couple of 'attractors'. This result indicates that HLRs in the three models could be grouped to some clusters. We observed the equivalent results independent of SOM's size and layers (CL or ML) of the models (Fig. 3 and Supplemental Figs. 1 and 2). Interestingly, some units (shown in blue) are never chosen as BMUs, and their numbers would increase as the size of SOM grows bigger. These results suggest that HLRs can be mapped onto a finite number of internal codes and that they satisfy the first necessary condition for FnC discussed above.
Next, we tested if HLRs could also satisfy the second necessary condition. To this end, we aggregated BMUs depending on the classes of inputs. Fig. 4 shows the density of BMUs of ResNet18 belonging to the same class; see supplemental To further test this possibility, we tested how well BMUs are clustered according to their classes. Specifically, we used k-means clustering and measured the clustering-score using V-measure [32,31] whose evaluation is invariant over permutations of labels of ground truth and predictions. In k-means analysis, we fixed the number of classes to be 1000 (k=1000, the number of ImageNet's classes). If HLRs are perfectly correlated to inputs' classes, the clusters identified by k-means clustering would be consistent with the ground truth labels, and the V-measure score would be close to 1 (by the definition of clustering score). If HLRs are not correlated with inputs' classes, the score would be 0. In the experiments, we repeated the same experiments 5 times for each condition (i.e., the selected layer and model) and estimated the mean and standard deviations from 5 V-scores (Fig. 5A); the standard deviations are too small to be seen in the figures. We note that the V-measure score increases as the layer goes deeper and moves closer to the output layer (Fig. 5A), suggesting that HLRs are indeed correlated with the labels of inputs to DL models (i.e., ImageNet classes), and their correlations become stronger when the layer is closer to the output (classification) layer.
So far, we have used the images, which were used to train SOMs (i.e., the validation set of ImageNet), to test the correlations between HLRs and inputs' classes. To test if this result is generalized to images, which has not been shown to SOM, we randomly selected 10,000 images (10 images per class) from the ImageNet training set to evaluate the correlations between BMUs and input's classes. We used ImageNet training set images, as the ImageNet test set does not contain the ground truth labels. As shown in Fig. 5B, the images of BMUs selected from the training set are also correlated with the inputs' classes, and as the layer goes deeper, the correlations increase. Interestingly, the correlations (i.e., V-score) are stronger in 10,000 examples not shown to SOMs than 50,000 examples used to train SOMs. This can be explained by the fact that DL models are trained with ImageNet training set, and thus HLRs represent 10,000 training examples better than 50,000 validation images, which are not used to train DL models. Conversely, SOM-based analysis can capture the difference in HLRs between ImageNet's training and validation sets.

Utilization of HLRs to probe DL models' operations
Encouraged by the results suggesting that HLRs can satisfy the two necessary conditions for FnC, we used HLRs to probe DL models' operations. In this study, we focused on two topics, 1) the features coded by HLRs and 2) the propagation of adversarial perturbations.

Features encoded by HLRs
Earlier studies inverted DL models to identify features coded in hidden layers. Notably, feature visualization [14] used gradient ascent to determine the optimal images that can maximize the activation of selected hidden layers. Inspired by the study, we inverted HLRs by minimizing the cosine distance (i.e., 1− cosine similarity) between SOM's weights and HLRs of present inputs. The inputs were randomly initialized and iteratively updated to minimize the cosine distance.
In the experiments, we first identified the local attractors of BMUs, which are the centers of BMUs' dense population, and inverted them; see supplemental text for more details. As shown in Fig. 6A, the inverted images of ResNet 18's CL1 contain repetitive patterns with color blobs. In addition, the patterns of inverted images differ between attractors (shown in different columns), suggesting that FnC in CL1 code distinct patterns. Importantly, the inverted images in deeper layers (CL2, 3 and 4) become more complex (Fig. 6B-D), and the inverted images of CL4 contain various visual features recognizable to us (Fig.9D). We repeated ResNet50 and observed the equivalent results in ResNet50' CLs (the right column of Fig. 6).

Effects of adversarial perturbations
DL models' vulnerability to imperceptible adversarial perturbation has been well documented, prompting multiple subsequent studies, but our understanding of adversarial inputs' effective deception of DL models remains limited. To gain insights into the underlying mechanisms of adversarial perturbations, we used BMUs to compare HLRs between normal and adversarial inputs. If HLRs between normal and adversarial inputs are similar to each other, the selected layer is insensitive to adversarial perturbations. By contrast, if the selected layer is sensitive to adversarial inputs, HLRs would be significantly different. As the distance between BMUs in SOM can reflect the similarities between the inputs, we measured the distance between BMUs of normal and adversarial inputs.
In the experiment, we used projected gradient descent (PGD) to create 500 adversarial examples from 500 normal images and estimated the distance between their corresponding BMUs; see supplemental text for more details. It should be noted that each adversarial input is derived from a distinct normal input to compare BMUs' inputs and normal inputs, which would look identical to human eyes. Supplemental Fig. 5 shows examples of adversarial and normal images depending on the maximum magnitude of adversarial manipulation ( ); the higher , the bigger the adversarial perturbation would be to add to each pixel. We fed them to ResNet18, ResNet50 and VGG19 and estimated the distances between the pairs of normal and adversarial inputs depending on the epsilon. We found that the distance between BMUs of normal and adversarial inputs increases, as the epsilon grows (Fig. 7). Interestingly, the degree of growth depends on the layers. In all three models, the distance grows rapidly in the first two layers but grows much slowly in the last two layers (Fig. 7). We used the t-test to examine if the distributions of distances are significantly different between = 0.01 and = 0.04. Notably, they are significantly different (p < 0.002) in CL1/2 and ML1/2 but not significantly different in CL4 and ML4 (p = 0.12, 0.22, 0.47 for ResNet18, ResNet50 and VGG19, respectively). That is, CL1 (ML1) and CL2 (ML2) are more sensitive to the adversarial perturbations' intensity, but CL3 (ML3) and CL4 (ML4) are more sensitive to the patterns than to the pixel intensity. Based on these results, we propose that adversarial attacks can manipulate the late layers to code incorrect semantic meanings by exploiting the sensitivity of the earlier layers to imperceptible perturbations.

Discussion
Hidden layers create DL models' internal codes in response to inputs, but it is challenging to characterize them due to their high variability and dimensions. In this study, we used SOM to study the properties of hidden layer activations (i.e., HLRs). In our analysis, HLRs are mapped onto a finite number of functional codes and are correlated with inputs' classes and models' predictions. Our analysis further suggested that SOM-based analysis allows us to better understand DL models' operating principles. Below, we discuss the potential utilization and extension of HLR-based analysis.

Real-time monitoring of DL models
The result shown in Fig. 7 suggested that BMUs' locations are sensitive to adversarial perturbations. With such sensitivity, BMUs can be used to detect adversarial inputs. Cantareira et al. [29] used U-MAP analysis [33] to show that hidden layer activations (HLRs) start to diverge at one of the hidden layers. Before the layer, HLRs of normal and adversarial inputs are indistinguishable (via U-MAP analysis), but after the layer, HLRs of the adversarial inputs become significantly different from HLRs of normal inputs. Based on SOMs' sensitivity to adversarial perturbations, we expect that BMUs across hidden layers could be used to detect adversarial perturbations. Specifically, the normal inputs could activate fixed sequences of BMUs across layers, but the adversarial inputs could activate abnormal sequences of , clustering score of BMUs of 10,000 images randomly drawn from the ImageNet training set. For each condition, we repeated the same experiment 5 times with different random seeds. That is, we trained 5 SOMs and estimated V-score for each SOM. Then, the mean and standard deviations were estimated from 5 V-scores under the same condition (i.e., the same model and layer). We found that the V-scores were remarkably homogeneous, making standard deviations invisible in the figures. BMUs. These abnormal sequences may be detected by anomaly detectors. We also speculate that the out-of-distribution inputs can activate abnormal sequences of BMUs, and inspecting the sequences of BMUs can reveal whether the present inputs come from the designated domains or not. We plan to study the possibility of building SOM-based real-time monitoring systems in the future.

Limitation of our analysis and future direction
To focus on testing SOM's capability to analyze DNN's FnC, we used the average pooling to reduce HLRs' dimensions in this study. With this dimension reduction, we tested the capability of SOM-based analysis with limited computing power: training SOMs took around 5 minutes or less with NVIDIA's RTX 3080. We note that this approximation removes all spatial information of the input features, which play crucial roles in our visual perception. For instance, eyes are located above mouths in faces, and such a simple spatial relationship can be used to check if the object is indeed a real face. We propose two potential ways to extend our analysis that can capture spatial information.
First, it is possible to preserve all spatial information by flattening hidden layer outputs without average pooling. However, this can significantly increase the length of HLRs, which would demand much more computing power. Instead, a more plausible option would be to train SOMs for spatial regions (receptive fields of convolutional filters). As the input can be split into multiple regions, multiple SOMs can exist for each hidden layer, from which we can obtain multiple internal codes from hidden layers. Then, these codes can be correlated with one another to further analyze DL models' operations.
Second, we trained SOMs with individual layers to seek characteristic HLRs in individual layers, but the relationships between active neurons across layers could capture important internal codes. An earlier study actually revealed that DL models could develop internal wiring depending on the inputs [34]. This approach also requires a lot of computing power due to a large number of neurons in DNNs, but SOMs can effectively solve this requirement. Further, as individual hidden layers have different receptive field sizes, sampling HLRs across layers can provide an effective way to capture spatial correlations between global and local features.
Before closing, we would like to point out the possibility of misusing FnC. FnC obtained by our analysis could be used to infer DL models' architectures and training methods, which may allow the adversaries to exploit them. However, to obtain FnC, the adversaries need access to DL models' hidden layers, and thus, the risk of misusing FnC is low.