{'fraction_non_alphanumeric': 0.023915362731152204, 'fraction_numerical': 0.010579658605974396, 'mean_word_length': 4.962014134275618, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Knowing What to Listen to: Early Attention for Deep Speech Representation Learning

Deep learning techniques have considerably improved speech processing in recent years. Speech representations extracted by deep learning models are being used in a wide range of tasks such as speech recognition, speaker recognition, and speech emotion recognition. Attention models play an important role in improving deep learning models. However current attention mechanisms are unable to attend to fine-grained information items. In this paper we propose the novel Fine-grained Early Frequency Attention (FEFA) for speech signals. This model is capable of focusing on information items as small as frequency bins. We evaluate the proposed model on two popular tasks of speaker recognition and speech emotion recognition. Two widely used public datasets, VoxCeleb and IEMOCAP, are used for our experiments. The model is implemented on top of several prominent deep models as backbone networks to evaluate its impact on performance compared to the original networks and other related work. Our experiments show that by adding FEFA to different CNN architectures, performance is consistently improved by substantial margins, even setting a new state-of-the-art for the speaker recognition task. We also tested our model against different levels of added noise showing improvements in robustness and less sensitivity compared to the backbone networks.


Introduction
Deep speech representation learning has been the subject of a large number of past works. Many techniques have been developed and employed for extracting representations from speech for related tasks such as speaker recognition (SR) and speech emotion recognition (SER) using deep learning. A significant number of these deep learning models have been based on Convolutional Neural Networks (CNN) for SR (Hajavi and Etemad 2019;Okabe, Koshinaka, and Shinoda 2018;Xie et al. 2019a;Chung, Nagrani, and Zisserman 2018;Nagrani, Chung, and Zisserman 2017) and SER (Albanie et al. 2018;Gideon, McInnis, and Provost 2019;Wang et al. 2020;Ghosh et al. 2016). The most common approach to training CNN models for speech-related tasks is to use time-frequency inputs such as spectrograms derived from raw audio signals. Given sufficient data, such deep learning models enable the extraction of better speech representations compared to other methods such as i-Vectors (Nagrani, Chung, and Zisserman 2017;Ghosh et al. 2016).
Attention mechanisms have been shown to have a positive impact on extracting effective deep representations from input data, for instance speech signals. Considerable improvements in accuracy of emotion recognition models (Tarantino, Garner, and Lazaridis 2019;Wang et al. 2020) and speaker recognition models (Zeinali et al. 2019;Bian, Chen, and Xu 2019;Okabe, Koshinaka, and Shinoda 2018) are some of the examples that demonstrate the potential benefits of using attention mechanisms for representation learning.
Attention models uphold a memory-query paradigm, where the memory is a set of information items such as CNN embeddings of a region of the spectral representation in speech-related tasks (Bian, Chen, and Xu 2019;Bhattacharya, Alam, and Kenny 2017), or a part of the utterance embedded by a recurrent cell in a recurrent neural network (RNN) (Zhang et al. 2019;Wang et al. 2020). The query is derived from a hidden state of the model from either the same modality or a different one (Xu et al. 2015;Bahdanau, Cho, and Bengio 2015). The majority of attention models used in speech-related tasks, use features extracted from utterances using a deep neural network as the information items or memory, and the last hidden layer of the model as the query (Xu et al. 2015). The general purpose of an attention model in generating deep representations of speech signals is to focus on each information item individually.
The information items considered in an attention model define the granularity of what the model can focus on. The spectral representation of an utterance enables deep learning models to consider fine-grained features such as frequency bins in very short time-frames. However, typical attention models used on audio signals utilize an embedding obtained from a CNN model as the memory and the final embedding of the model as query. Using embeddings obtained from CNNs, limits the granularity of the attention models to large regions of the spectral representation. On the other hand, improving the granularity of CNN embeddings of an utterance leads to very large attention models which are harder to train and prone to over-fitting. While there have been a number of studies investigating various attention models using CNN embeddings utterances (Bhattacharya, Alam, and Kenny 2017;Bian, Chen, and Xu 2019;You et al. 2019;Safari and Hernando 2019), very limited number of studies aim to use more fine-grained attention models on spectral representation of the utterance.
In this paper, we address the challenge of improving granularity of attention models by introducing a fine-grained attention mechanism for audio signals. This mechanism enables deep learning models to focus on individual frequency bins of a spectrogram without the drawbacks of having very complex models that typically involve large number of parameters. The aim of this model is to attend to each frequency bin in the spectrogram representation in order to boost the contribution of most salient bins. This mechanism also helps reduce the importance of bins with no useful information leading to more accurate representations, which can also lead to more robustness with respect to existing noise in the input audio. The performance of the proposed attention mechanism has been tested using a select set of most prominent CNN architectures on two tasks of SR and SER. The experimental results show that deploying the finegrained frequency attention mechanism improves the performance of all the benchmark networks substantially while being less impacted by added noise.
Our contributions in this paper are as follows: • We introduce a novel attention mechanism for speech representation learning.
• We test our method on two different speech-related problem domains, namely speaker recognition and affective computing, using two large and widely used datasets, demonstrating considerable performance gains for both tasks.
• By simply adding our fine-grained frequency attention method to the existing state-of-the-art model for speaker recognition, we set a new state-of-the-art for speaker recognition in the wild.
• By testing our model against different levels of synthetic noise, we show an improvement in robustness compared to other models.
The rest of this paper is organized as follows. First, we discuss the related work in the area of speech representation learning followed by particular approaches that have used attention mechanisms for this purpose. Next, we present the proposed attention mechanism. In the following section, we discuss the experiments along with implementation details. Next, we provide the results of our work. And finally, we summarize and conclude the paper.

Speech Representation Learning
Speech representation, or utterance embedding, has been an area of research for decades. Classical signal processing techniques such as Gaussian Mixture Models, Hidden Markov Models, and Universal Background Models, were used in many speech related tasks to obtain a proper representation of utterances. Comprehensive reviews of prior work that have used such conventional methods for SR and SER can be found in (Hansen and Hasan 2015;El Ayadi, Kamel, and Karray 2011).
Solutions based on artificial neural networks (ANN) have been widely used in speech-related tasks. In some of the earlier work in this area, speech representations extracted from audio signals using ANNs were fed to conventional classifiers for SR (Farrell, Mammone, and Assaleh 1994) and SER (Nicholson, Takahashi, and Nakatsu 2000).
More recently, deep neural networks (DNN) have been used for learning effective representations of utterances (Variani et al. 2014;Bhattacharya, Alam, and Kenny 2017). Most recent works on extracting deep speech representations for SR have explored the impacts of different deep learning architectures on the quality of these representations. Most prominent works include using CNN architectures such as ResNets for speech representation learning prior to identification (Xie et al. 2019a;Bian, Chen, and Xu 2019;Hajavi and Etemad 2019). Among other speech-related tasks such as SER, DNN models have also been very successful for speech representation learning. Most recent studies of SER focus on improving the accuracy of the deep learning models by modifying and combining different architectures. Some of the considerable attempts include using the combination of CNN and RNNs such as long short-term memory (LSTM) networks (Xie et al. 2019b;Latif et al. 2019;Wang et al. 2020).

Attention-based Speech Representation Learning
The performance of deep learning models has improved significantly by attention models in many cases (Bahdanau, Cho, and Bengio 2015;Xu et al. 2015 (Simonyan and Zisserman 2014). The study done in (Bian, Chen, and Xu 2019) used CNN-based self-attention models to attend to features extracted from a deep learning model with an architecture similar to ResNet (He et al. 2016;Zhao et al. 2020). A novel gated attention model was proposed in (You et al. 2019) to attend to features extracted by a modified version of CNN, namely gated-CNN. The proposed models in (Zhang et al. 2019;Wang et al. 2020;Tarantino, Garner, and Lazaridis 2019), utilized attention models to focus on differences between two sets of features extracted from the enrollment utterance and the questioned utterance using RNN. In the common approach taken in these studies, the attention models were added to the end of deep learning pipelines. The addition of attention models in this way has shown to improve the accuracy of baseline models against in-the-wild datasets in each of these studies.
A different approach was taken in (Okabe, Koshinaka, and Shinoda 2018) and (Zhu et al. 2018). The attention models used in these studies replaced the statistical pooling layer of an X-Vector model. The proposed models utilized TDNN to extract frame-level features from utterances. Attention models were then used to aggregate the features into an utterance-level embedding. The model proposed in (Zhu et al. 2018) was evaluated against the NistSRE16 evaluation set (National Institute of Standards and Technology 2016) and the proposed model in (Okabe, Koshinaka, and Shinoda 2018) was evaluated against the VoxCeleb 1 test set (Nagrani, Chung, and Zisserman 2017). Both models showed substantial improvements compared to their baseline models.
The majority of the aforementioned studies have used the features obtained from DNNs as the memory component of the attention model. The queries of the attention models were also originated from the last hidden layer of the model from which the utterance-level embeddings are retrieved. Generally, DNNs learn to extract a low-dimensional latent representation from the input data without necessarily preserving localization with respect to the input information items. Thus, while the use of the last hidden layer of a DNN for extracting the query of an attention mechanism can be advantageous due to its reduced number of parameters, high levels of granularity and a localized relationship with respect to the input may not be achieved.
Compared to the methods proposed in previous studies, the fine-grained attention model proposed in this paper does not require embeddings obtained from DNN models, and can operate on spectrograms extracted from raw audio signals. Hence, the granularity of the attention model can be improved to attend to frequency-level features. While different attention mechanisms depend on the specific architectures and models, our proposed fine-grained frequency attention mechanism can be used along with various models and architectures. As proven in the experiment section, by adding the frequency attention to multiple CNN-based architectures, a substantial improvement is achieved on both tasks of SER and SR.

Fine-grained Frequency Attention
The fundamental paradigm of a general attention mechanism is the memory-query system. Considering audio signals, the memory typically consists of a set of information items, namely DNN embeddings, and the query is acquired from the hidden state of the overall model. The memory is saved in the form of key-value tuples (key i , value i ). The first element of the tuple key i helps with the calculation of the probability factor p i , which indicates the impact of the item over the query.
Equation 1 represents a general attention in which a multilayer perceptron (MLP) is used to determine the probability p i . The matrix W is a set of trainable weights integrated by an MLP that carries the impact of Query in determining the probability of the item. The final output of the attention mechanism with respect to a query, is the expected value of items with regards to the variable value i (see Equation 2).
While typical attention mechanisms may allow the information items to be as fine-grained as possible, the complexity of the attention model itself grows considerably with improving the granularity of the memory set. Through our proposed Fine-grained Early Frequency Attention (FEFA) model, we tackle this issue by changing the source of the query to any hidden layer of the deep model. This is in contrast to the other attention mechanisms where the last hidden layer is used as the source of the query. We also change the structure of the memory to contain frequency bins provided by the spectrogram representation as the information items.
The spectrogram representation of the speech signal is the most commonly used feature set among deep learning models that exploit CNN architectures. While the number of frequency bins may vary between studies, the overall approach in calculating and using spectrogram representations are quite similar. The spectrogram representation of an utterance is obtained by using Short-time Fourier transform (STFT) (see Equation 3). The symbol x(t) serves as the signal amplitude at a given time t. W (t − τ ) is the window function applied over the signal to enforce the time window of the STFT as well as to extract the phase information of the signal. ω represents the frequency band around which the STFT is performed. (3) After calculating the STFT of the signal for a given frequency bin ω, the squared magnitude of the result is then used as the spectrogram representation (see Equation 4). A typical value used for the time window in speech-related tasks is 25ms. Hence we can drop the variable τ with the default value of 25ms from the formula to simplify the equation as follows: The final spectrogram of the speech signal is obtained by repeating the process for a select number of frequency bands. The selection of frequency bands are given as a hyper-parameter in the form of a set of filters called filterbank. Each value acquired by function Spec(x(t), ω i ) represents the frequency information of the signal with regards to the filter ω i at a given time in a 25ms time-window. Having the frequency bins as the construction blocks of the spectrogram representations, every individual bin can be considered the smallest item carrying information. For the FEFA model we utilize the frequency bins as information items to serve as the memory component for the attention mechanism.
One of the main challenges that prevent attention mechanisms from increasing the granularity of their memory set, is the source of the query. In typical attention mechanisms the query is originated from the last hidden layer of the deep learning model. The complexity of such a model will increase considerably with regards to improvements in the granularity of the attention mechanism. Having spectrograms to serve as the memory component of the attention mechanism, the complexity of the attention mechanism will make the model very hard to train and generalize.
Each layer of a given deep learning model operates over a feature set. The feature set associated with each hidden layer of the DNN is capable of serving as the target modality for extracting queries for the attention model. In the proposed FEFA model (illustrated in Figure 1 (a)), we utilize the hidden layers earlier in the DNN model as the new source of query.
The internal architecture of the FEFA module is shown in Figure 1 (b). The spectrogram representation of the utterance is squeezed into a single vector using an average pooling operation. Then, an MLP module is utilized as the kernel of the proposed FEFA model to calculate the probability of each frequency bin in the enhanced spectral representation of the utterance (See Equation 5). Accordingly, the index of each frequency bin in the initial feature space (spectrogram representation), serves as the key for the information item.
An attention map is then created by calculating the expected value of each frequency bin using the probability obtained through the MLP module (See Equation 6). The attention map acquired from the attention module is then multiplied by the original spectrogram representation of the utterance resulting in an enhanced representation of the utterance to be used by the DNN.
The FEFA model does not require any pre-processing or feature extraction in addition to the STFT calculation. Hence the model is compatible with various deep learning architectures that use spectrogram representation of utterances as input. Later on in the experiments section, we demonstrate that by adding the FEFA model to various architectures such as ResNet, VGG, and SEResNet, considerable performance improvements are achieved.
The memory and computational complexities of the FEFA model are respectively linear and quadratic with regards to the number of frequency bins (nfft) used in calculating the spectrogram representation (See Equations 7 and 8). Hence adding multiple layers of FEFA throughout the pipeline of the DNN does not increase the computational complexity of the model drastically.

Single-layer vs. Multi-layer
The memory set of the FEFA model is not limited to the spectrogram representation of the utterance. Considering the flexible mechanism of the attention module, the embeddings of each hidden layer of the DNN can be utilized as the memory set. In order to achieve this, the embeddings of the hidden layer are first passed through a channel-wise average pool to imitate a single-channel spectrogram image. The resulting matrix is then passed through the FEFA module with the same procedure. The employed channel-wise average pooling mechanism , along with the query extraction mechanism, enable the FEFA module to be used between any two layers throughout the DNN pipeline.

Experiments
The proposed FEFA model has been evaluated on two tasks of SR and SER. The FEFA model can be used with different DNN architectures as backbone networks that take the spectrogram representation of the utterances as the input. Hence we have used a select number of prominent CNN architectures commonly used in these tasks as our benchmarks. In the following subsections we introduce the datasets used in our experiments, implementation details regarding FEFA, as well as the details of the backbone networks used to add our attention mechanism onto.

Datasets
We utilize two widely used datasets for experiments in two different speech representation learning areas (SR and SER), namely VoxCeleb and IEMOCAP. VoxCeleb: For the SR task we perform our evaluations using the large and widely used in-the-wild VoxCeleb dataset (Chung, Nagrani, and Zisserman 2018). The Vox-Celeb dataset includes voices from more than 6,000 individuals. The utterances are captured from uncontrolled conditions such as interviews published in open-source media. The VoxCeleb dataset is available in two versions, Vox-Celeb1 which is used more commonly for evaluation and VexCeleb2 which is used solely for training purposes. In this experiment we follow the common practice and use the Vox-Celeb2 dataset with nearly 1.2 million utterances for training our model and VoxCeleb1 test set for evaluation.
IEMOCAP: We also evaluate the FEFA model using the IEMOCAP dataset (Busso et al. 2008) for the task of SER. The IEMOCAP dataset is a multi-modal emotion recognition dataset including speech recordings, videos, and motion capture. The dataset contains 12 hours of prompted and improvised dialogue performed by 10 actors. The audio recordings of the dataset are divided into short utterances each containing one sentence. Each utterance is then scored by several people to determine the category of emotion conveyed by the utterance. In our experiments we have selected 4 emotion categories of Sadness, Happiness, Angry, and Neutral, for a total of 6 thousand utterances. The selection of these 4 emotion categories is to comply with the common practice of SER established by majority of studies using this dataset.

Implementation Details
Data Preparation: For data preparation, we extract spectrogram representations of the utterances resulting in spectrogram images of size 257 × T . We use 257 frequency bins to be able to better compare our results to the state-of-the-art in SR. We follow the same practice in the SER task to maintain contingency throughout the experiments.
FEFA Details: We utilize a single layer locally connected MLP as the kernel of our attention model. We chose a simple kernel to minimize the impact of the latent scores learnt by more complex networks, and instead focus on the impact of using early attention over frequency-bands on speech representation learning. The number of nodes used in this kernel was set equal to the number of frequency bins in the spectral representations (257 in our case). The kernel is trained using the Adam optimizer and back-propagation.
Backbone Networks: In order to assess the impact of the FEFA model on different deep learning networks, we use two of the latest state-of-the-art models which are based on VGGNet (Nagrani, Chung, and Zisserman 2017) and ResNet (Xie et al. 2019a). We have also implemented a novel thin-SEResNet model by combining the state-of-theart ResNet model with the SE blocks proposed in (Hu, Shen, and Sun 2018).
For each of the three backbone networks, three versions were implemented: • The model without any FEFA enhancing; • The model with one layer of FEFA enhancement; • The model with multiple layers of FEFA enhancement distributed among layers of the DNN pipeline where the dimensions of the hidden representation has changed.
The first model used in our experiments is the VGG-based model proposed in (Nagrani, Chung, and Zisserman 2017), which consists of 5 convolution layers accompanied by 3 maxpooling layers. The utterance-level aggregation is done using a global-average-pooling and the final embedding is acquired using a fully connected layer with ReLU activation.
The second network that we use in this experiment is the ResNet-based model proposed in (Xie et al. 2019a). This model consists of 35 convolution layers used in the form of residual blocks. The shortcuts integrated in residual blocks help the model convey the learning gradients throughout the pipeline of the model more easily, which in turn aids the model to learn faster and more efficiently. This also enables the model to provide better queries for the FEFA module. Complete details about the hyper-parameters and implementation details can be found in (Xie et al. 2019a).
The final network used in our experiments is SERes-Net. Similar to the ResNet model, the SEResNet consists of residual blocks. The formation of blocks and number of parameters used in the SEResNet is similar to a ResNet with an addition of a Squeeze-and-Excitation (SE) module (Hu, Shen, and Sun 2018). The SE module uses a global pooling layer to extract channel information inside the residual blocks. The channel information is then projected onto a latent space using 2 FCNs, a ReLU activation function, and a Sigmoid activation function. The resulting representation is then multiplied across channels of the ID block.
Training: For training the backbone networks with the added FEFA, we used a recent technique to adjust the learning rate throughout the process. The cyclical learning rate proposed in (Smith 2017) helps the model to achieve a better convergence by changing the learning rate periodically and preventing the model from getting trapped in local minima.

Results
For evaluating the networks in the SR domain, we use 2 commonly used metrics namely Equal error rate (EER) and identification accuracy (Acc). The EER is the error threshold of the model in which the number of false positive errors is equal to the number of false negative errors. Table 1 presents the results, as well as the performance gain achieved by using our proposed FEFA model. The first section of the table is dedicated to the typical attention models of self-attention and soft-attention. The results show that FEFA models outperform the typical attention models by a large margin. The  FEFA model also surpasses the state-of-the-art values improving the performance by 3.1% on EER and 6.0% on Acc, achieving a new state-of-the-art for SR. For the backbone models of VGG and Thin-ResNet, we refer to the reported values in the reference papers. As there are no published studies of SEResNet architectures for SR, the implementation and evaluation of this backbone model is done in the scope of this experiment. The results of all the backbone models show the positive impact of the FEFA module on the models. The consistent improvement of all the backbone networks also proves compatibility of the FEFA module with different architectures. As discussed, to evaluate the generalizability of our FEFA model, we also perform experiments on SER. In these experiments, we employ the commonly used classification accuracy as the evaluation metric. To comply with the common practice in using the IEMOCAP speech emotion dataset, we perform a k-fold cross validation for evaluating our models. Given that in many recent works for emotion recognition from speech, VGG-and ResNet-based architectures have frequently been used for speech representation learning (Yenigalla et al. 2018;Kim et al. 2017), we utilize the same backbone networks for evaluating the impact of our proposed FEFA approach. This also enables us to compare our results to the SR task performed earlier and provide a more consistent analysis of the results. Table 2 shows the results of evaluating the FEFA model for the task of SER. It is evident by the results that adding the FEFA module has a very positive impact on the performance of the backbone models in predicting emotion classes.
Interestingly, while using multiple layers of the FEFA module in the deep models considerably improves the performance compared to the plain backbone networks (no FEFA added), the performance is consistently less prominent than using a single layer of FEFA. A possible reason could be that due to the 2D shape of convolutional kernels, some features from the time axis are convolved with the frequency axis. Given that temporal information have already been considered while performing the average pooling inside the FEFA module, including these features in the frequency axis may have reduced the contribution of the frequency information in the final attention map.

Robustness to Noise
Given the inherent function of the FEFA too focus on the most salient frequency bins prior to being processed by the model, we anticipate that DNN+FEFA architectures will be more robust to noise compared to their backbone DNN counterparts. In order to test this hypothesis, we evaluate the performance of our model solution against different levels of noise. To do so, a controlled level of synthetic noise is added to the test utterances for the speaker recognition task. The model with the best performance (the previous stateof-the-art), Thin-ResNet + GhostVlad, is selected and tested with the noisy utterances with and without FEFA.
The added noise is selected from Gaussian (Figure 2 (a)) and uniform (Figure 2 (b)) distributions. The second column in Figure 2 depicts the effect of noise added to the spectral representation of utterances, comparing it to the clean utterance. The last column depicts the attended areas by the FEFA model. The highlighted areas in the spectral representation show the individual frequency bins with the highest contribution. Hence by focusing on these frequency bins, the FEFA model decreases the effect of artifact noise in other areas of the spectrogram on the final learned representation.
The results of this test are reported in Table 3. The test is performed using synthetic noise resulting in 3 signal-to-Clean Noisy a) b) Figure 2: Robustness test by adding synthetic noise to the utterances. (a) the noise selected from a Gaussian distribution. (b) the noise selected from a uniform distribution noise ratios (SNR) of 20db, 50db, and 100db. As shown by the results, while the performance of the backbone network is considerably affected by the added noise, the model with the FEFA mechanism stays relatively more stable.

Discussion and Comparison to Other Forms of Attention
As shown in the first and second rows of Table 1, CNN models plus general forms of attention such as self-attention (Bian, Chen, and Xu 2019) and soft-attention (Okabe, Koshinaka, and Shinoda 2018) do not perform as well as our FEFA model integrated into similar backbone networks. Our approach shows a clear enhancement performance over the classical attention mechanisms as such attention models attend to parts of the latent representation that correspond to large areas in the input utterance spectrogram. Hence they fail to focus on very small frequency-level features that are often crucial in speech-related tasks. While a number of attempts have been made to achieve different levels of granularity with attention mechanisms, existing attention models do not achieve a fine-grained solution. The area attention model proposed in  achieves varying degrees of granularity by creating different combinations of neighboring information items. However, the information items used in the combinations are embeddings already extracted by the DNN model, limiting the level of granularity based on the resolution of the latent representation achieved by the DNN.
Another attempt for a frequency-based attention model was proposed by (Yadav and Rai 2020). Their attention model adopted from image recognition can utilize any hidden layer of a deep network as the source of query. The attention model proposed in their work uses the latent representations obtained from different layers of the CNN as the memory set. This rules out the possibility of a localized attention map with respect to the input. Their model also uses a shared weight CNN layer as the kernel of the attention model. In this approach, and others that similarly employ CNN layers for the kernel of the attention model, information items go through non-linear operations preventing the model from maintaining a one-to-one relation between the attention map and the information items. Therefore as this approach may be successful for some applications, it fails in others where the contribution of each separate information item is important. Generally, the intuition behind many attention models (in speech-related tasks or otherwise) is to focus on different parts of some latent representation of the input to inform better classification. In these models, the representations are generally learned irrespective of important known information items in the input. Speech depends on frequency content to convey information. In fact, humans have evolved to understand different facts about the source of speech (e.g. identity, intent, emotions, etc.) based on factors such as tone, pitch, and others (Hansen and Hasan 2015). By learning to exploit specific frequency bins in the input that may contain effective task-related information, DNNs can learn to pay more attention to those particular bins to achieve better performance.

Summary and Future Work
In this paper, a novel attention mechanism is proposed that allows deep learning models to focus on fine-grained information items, namely frequency bins without, increasing the complexity of the model. The proposed FEFA model uses the spectrogram representation of the model as the input and provides a better representation of the spectrogram by attending to each frequency bin individually. We evaluated our attention mechanism on two tasks of speaker recognition and speech emotion recognition. The comparison between models enhanced by FEFA and the original backbone networks shows consistent improvement in the performance of deep learning models in both tasks.
Our analysis shows that using multiple layers of the FEFA module does not have as much positive impact as a single layer. A possible future route is to study the factors contributing to this effect. The intended outcome of such study would be to design a solution to benefit both features from frequency axis and time axis from the latent layers.
The current version of FEFA model utilizes simple average pooling mechanisms and MLP as the internal components of the attention mechanism. Another possible future route is to improve the internal architecture of the FFA module using more complex neural networks and different temporal pooling operations.