{'fraction_non_alphanumeric': 0.03605687845615619, 'fraction_numerical': 0.013147500282135198, 'mean_word_length': 5.020040760869565, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Graph attentive feature aggregation for text-independent speaker verification

The objective of this paper is to combine multiple frame-level features into a single utterance-level representation considering pairwise relationship. For this purpose, we propose a novel graph attentive feature aggregation module by interpreting each frame-level feature as a node of a graph. The inter-relationship between all possible pairs of features, typically exploited indirectly, can be directly modeled using a graph. The module comprises a graph attention layer and a graph pooling layer followed by a readout operation. The graph attention layer first models the non-Euclidean data manifold between different nodes. Then, the graph pooling layer discards less informative nodes considering the significance of the nodes. Finally, the readout operation combines the remaining nodes into a single representation. We employ two recent systems, SE-ResNet and RawNet2, with different input features and architectures and demonstrate that the proposed feature aggregation module consistently shows a relative improvement over 10%, compared to the baseline.


INTRODUCTION
Speaker verification (SV) can be used for various authentication scenarios where the identity of a given speech is compared to that of the claimed speaker. In SV, speaker representations are derived by first extracting the frame-level features and then aggregating them. The network extracting the frame-level features is referred to as trunk network (e.g., convolutional neural network (CNN) and x-vector [1][2][3][4][5][6]). After extracting the frame-level features, various techniques including gated recurrent network (GRU), learnable dictionary encoding (LDE) are used on top of the trunk network to aggregate framelevel features into a single utterance-level feature [7][8][9][10][11][12][13][14][15][16][17][18]. The condensed utterance-level representation ultimately represents the entire character of an utterance, unlike speech recognition where each frame-level representation is of worth. Hence, feature aggregation plays an essential role and a number of studies in the recent literature have focused on developing feature aggregation methods.
A group of studies employed recurrent layers for feature aggregation. These studies modeled the frame-level representations in sequential order [7][8][9][10][11]. Another group of research integrated framelevel features using LDE [16][17][18]. Similar to the recurrent layers, the LDE layer aggregates features over time whereas it further utilizes statistics. Sequential modeling, a foundation of the above two approaches, has been used in SV based on its success in language processing and speech recognition, where the sequential order could † Equally contributed. * Corresponding author. be a prior knowledge. However, recent studies found that sequential information may not be the key, especially in text-independent SV (TI-SV) [14,19].
Other groups of researches exploited attention-based approaches- [12][13][14][15], which exclusively emphasize important features regardless of their sequence and have recently become the mainstream in TI-SV. These approaches assign an attention weight in the form of a scalar to each frame-level feature and then perform weighted summation (also with statistics as in [14]). Each attention weight for a feature is derived via a dot product between a feature and a projection vector. A softmax non-linearity is applied to attention weights to exclusively emphasize which frame is more important, however, they are compared indirectly without modeling their relationships.
The field of graph neural networks is recently getting attention by incorporating the advantages of graph structures and deep neural networks [20,21]. Interpreting high-dimensional representations as nodes of a graph, graph neural networks can model the non-Euclidean data manifold within nodes, including inter-node relationship. Particularly, recent architectures such as graph convolutional network [22] and graph attention network (GAT) [23] combined with various graph pooling layers [24][25][26] have demonstrated competitive performance even in image and audio domains [27][28][29]. Inspired by the recent success of graph neural networks, we argue that the feature aggregation could be further improved by explicitly comparing and modeling the inter-relationship between framelevel features. The feature aggregation is a task to express characteristics relative to other features and condense them into a single representation. Hence, the inter-relationship between features extends the ability of its expressiveness and could be the vital information itself. To this end, we propose a novel feature aggregation module to model inter-relationship by interpreting each frame-level feature as a node of a graph. The proposed module leverages the correlation of all possible pairs of nodes to obtain an utterance-level representation. In other words, entire frame-level features are aggregated considering their inter-relationships in addition to their intrinsic characteristics by utilizing a graph. Specifically, the aggregation involves a GAT, a graph pooling layer, and a readout operation [23,24]. The GAT assigns different weights to each node pairs (edges) and models their relationships. Then, the graph pooling layer discards less informative nodes. Finally, a single utterance-level representation is derived using the readout operation.
The effectiveness of the proposed feature aggregation framework is validated using two strong baselines, SE-ResNet [30] and RawNet2 [11], comprising different input features and architectures. Experimental results on the VoxCeleb datasets [31,32] demonstrate more than 10% consistent improvement over the corresponding baseline. Using the proposed graph aggregation module, we could compose more lightweight, yet better performing models compared to self-attentive pooling and GRU-based aggregations. Proposed graph attentive feature aggregation module. After extracting frame-level features using a trunk network, a graph is constructed using each sequence element as a node. Each node can have a dimensionality equal to either the number of filters (raw waveform, 1D-CNN) or the number of filters × the number of frequency bins (mel-filterbank, 2D-CNN). Graph attention network first aggregates nodes features, and then the graph pooling reduces the number of nodes. The readout function integrates nodes into a single representation (best viewed in color).

GRAPH ATTENTIVE FEATURE AGGREGATION
In this section, we introduce the graph attentive feature aggregation module. The proposed module is located after the extraction of frame-level features using a trunk network (RawNet2 or SE-ResNet). The proposed approach consists of three components: i) graph attention layer, ii) graph pooling layer, and iii) readout. We first describe how we formulate a graph from a feature map and perform GAT to integrate the overall features. Then, we explain how the graph pooling layer discards less informative nodes and the readout combines the results into a single representative feature. The overall scheme is illustrated in Figure 1 using RawNet2 (the scheme is identical for SE-ResNet except the dimensionality of each node).

Graph attention layer
We first formulate a graph using frame-level features extracted from a trunk network. Specifically, we interpret each sequence element as a node of a graph; for RawNet2, each node has a dimensionality equal to the number of filters and for SE-ResNet, the dimensionality equals to the number of filters × the number of frequency bins. Note that we design a fully-connected (complete) graph where all possible edges exist between pairs of nodes. Nodes can be directly compared considering all combinations/relationships because all nodes have bi-directional edges. Hence, entire frame-level features are aggregated considering their inter-relationships as well as intrinsic characteristics of their own by utilizing the graph attention layer.
Let G be a complete graph comprising N nodes with the F dimensional features of each node. A set of nodes in G is defined as x ∈ R N ×F and each nodes are represented as row vectors x1, x2, · · · , xN . In the GAT layer, x is first projected into F dimensional space using a matrix multiplication with W ∈ R F ×F resulting in n ∈ R N ×F (Equation (1)). Then, we calculate attention scores where an attention score aij between i-th node n i and j-th node n j (n i , n j ∈ R F , i = j) is calculated through Equation (2) and (3). We derive eij, by first concatenating two nodes n i and n j and then projecting to a scalar value through a dot product with γ ∈ R 2F ×1 , a learnable parameter, followed by a Leaky ReLU non-linearity [33]. Using the calculated attention scores, the GAT performs self-attention on node n i as in Equation (4). The GAT is Table 1. SE-ResNet architecture applying the proposed graph aggregation method. ReLU and batch normalization layers are omitted. Squeeze and excitation [34] is performed after all convolutional blocks. The numbers denoted in brackets refer to [size× size, filters] and those in the output refer to (filters, frequency, time). The output of the last fully-connected layer is used as the speaker embedding.
applied to every node x transforming n to n and then the output n is fed to a graph pooling layer. The GAT process can be described as follows:

Graph pooling layer and readout
We utilize a graph pooling layer followed by the readout operation to obtain an utterance-level feature. Similar to pooling layers in convolutional neural networks, graph pooling can reduce the size of features to enable high-level feature encoding and receptive field enlargement. Specifically, the graph pooling layer reduces the original graph into a sub-graph by removing less informative nodes as depicted in Figure 1. For the graph pooling, we exploit gPool introduced in the Graph U-Net architecture [24]. There are various other graph pooling methods, but many of them assume the hierarchical architecture of the graph. As it is difficult to define a hierarchical relationship between frame-level features in audio data, we adopt gPool, which does not require the graph to have a hierarchical structure. gPool consists of the following processes: projection, top-K node selection, and a gating mechanism. A set of nodes in the graph pooling layer l is represented as n l . Note that the adjacency matrix is omitted because we utilize a complete graph which is different from the original paper of [24]. In the projection stage, gPool obtains score vector y by estimating the scalar projection values of each node, using a trainable projection vector p ∈ R F . In the top-K node selection, the operation of node ranking (rank) outputs indices of the K-largest values (index) based on calculated score vector y. The score vector of selected nodes,ỹ, can be obtained by applying sigmoid to each element in the extracted scalar projection vector. In the gate stage, element-wise product is conducted between the pooled nodesñ l and the selected node scores y.
Formally, gPool can be denoted as follows: y = n l p l / p l , index = rank(y, K), y = sigmoid(y(index)), n l = n l (index, :) After the graph pooling, we conduct the readout operation, which combines the processed nodes into a single node. We explore various types of readout operation where the best result is obtained using the summation of nodes: Here, U is the aggregated utterance-level feature and K nodes n k are integrated to n. The results of various readout mechanisms can be found in Section 4.

Datasets
All experiments are performed using the VoxCeleb1&2 datasets [31,32]. We train the model using the development subset of VoxCeleb2 that includes the utterances from 5,994 speakers. Then, the evaluation is performed using the original trial that uses VoxCeleb1's evaluation subset.

Implementation details
Both baselines using a mel-filterbank and a raw waveform are implemented based on the PyTorch framework [38]. Graphs are implemented using Deep Graph Library [39]. We adjust the number of attention heads and use 16 heads in RawNet2 and 32 heads in SE-ResNet based on empirical results.

SE-ResNet.
Our mel-filterbank baseline is most similar to the architecture of [30], but overall details are adjusted. We use 40dimensional mel-filterbank features extracted with 1,024 point FFT and a hamming window of width 25ms and step 10ms. The melfilterbank baseline is optimized using Adam optimizer which uses a learning rate of 0.001, and the learning rate is decreased by 5% in each epoch. We use two types of loss functions, additive angular margin softmax (AAM-softmax) [40] and angular prototypical loss (AP) [41] to consider inter-class as well as intra-class covariance. We employ a margin of 0.3 and a scale of 30 for AAM-softmax. For AP, we use a mini-batch size of 200, where each mini-batch contains two utterances per speaker. The system is trained for 100 epochs. Our overall architecture applying the proposed method is presented in Table 1.
RawNet2. RawNet2 [11] is an end-to-end system that is fed by raw waveforms directly without preprocessing techniques. For minibatch construction, utterances are either cropped or duplicated (concatenated) into 59,049 samples (≈ 3.69s) in the training phase, following [11]. In the evaluation phase, no adjustments are made to the length. We modify several details from RawNet2 [11] in the process of adjusting the GAT as follows: i) exclude sinc-conv layer, as it slows down training time in spite of showing similar performance, ii) replace softmax with additive margin softmax (AM-softmax) [42], iii) reduce the dimensions by half for the last fully-connected layer before speaker embedding (1024 dimensions to 512 dimensions).

RESULTS
Application of GAT. In Table 2, we first address the effectiveness of the graph attention layer on our two baselines: SE-ResNet and RawNet2. Here, the graph pooling is temporarily excluded and summation readout operation is applied. Adoption of the graph attention layer reduced the EER relatively by 6% and 10%, respectively. In addition, we confirm that the number of parameters also decreases (by almost 25% in the case of RawNet2 (13.2M to 9.9M)).
Graph pooling. In Table 3, we explore the effect of applying the graph pooling layer between the graph attention layer and the readout operation with diverse pooling ratios. The denoted percentage of pooling refers to the remaining ratios of nodes; higher ratios mean that fewer nodes have been removed. Top 80% pooling removing 20% of irrelevant nodes shows the best results where the relative improvement compared to the baseline becomes 11.6% and 13.3%. Through a comparison of the three different ratios of pooling, we find that discarding too many nodes worsens the performance (11% pool) but removing a few of the most irrelevant nodes improves the performance (80% pool). We thus conclude that the discriminative information is well combined into the majority of nodes (80% in our case) via the GAT layer.
Comparison of readout operations. In Table 4, we further compare several readout operations using the RawNet2 baseline. We find that the proposed graph-based aggregation shows consistent effectiveness, regardless of the adopted readout approach because all node aggregation methods performed better than GRU. Among various methods, summation achieved the best result.
Comparison with state-of-the-art systems. In Table 5, we compare the proposed system with other recent state-of-the-art systems in the literature that adopt various feature aggregation methods. None of the systems apply data augmentation techniques. In both input fea-tures, the proposed models using the proposed graph aggregation module demonstrated the best performance, showing the effectiveness of the proposed approach. Given that we did not use data augmentation methods, there is room to improve the system using various data augmentation.
Additional experiments. Table 6 addresses two additional experiments using the RawNet2 baseline. First, we explore whether adopting a GRU after the GAT is beneficial and show the result in the first row. To maintain the overall complexity of the model, we omitted the last residual block and then placed the GAT and the GRU in sequence. However, this worsens the performance, as presented in the first row. In our analysis, this shows that applying GRU directly to the GAT's output is not effective, because the GAT's output is not sequential. Second, we explore using two GAT layers with different pooling architectures proposed in [25] and denote the result in second and third rows. Both architectures adopt two GAT layers in sequence. The global architecture concatenates two GAT layers' output and feeds it to the graph pooling layer followed by the readout, whereas hierarchical readout performs graph pooling and readout after each GAT layer and then adds them element-wisely. Through experiments, we found that both modified pooling architectures did not bring further improvements.

CONCLUSIONS
In this paper, we proposed a graph feature aggregation method for TI-SV. Utilizing a GAT, graph pooling layer, and readout operation, we directly modeled the inter-relationship between entire framelevel features, which is partially or indirectly utilized in the existing methods. As this is the first work employing graph neural networks for feature aggregation, we also explored various configurations to optimize the system. Consistent improvements over the baselines with different aggregation modules demonstrate the effectiveness of the proposed approach.