{'fraction_non_alphanumeric': 0.03529650073428875, 'fraction_numerical': 0.013774244188990733, 'mean_word_length': 4.81336473358846, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Detecting Replay Attacks Using Multi-Channel Audio: A Neural Network-Based Method

With the rapidly growing number of security-sensitive systems that use voice as the primary input, it becomes increasingly important to address these systems' potential vulnerability to replay attacks. Previous efforts to address this concern have focused primarily on single-channel audio. In this paper, we introduce a novel neural network-based replay attack detection model that further leverages spatial information of multi-channel audio and is able to significantly improve the replay attack detection performance.


I. INTRODUCTION
In recent years, an increasing number of security-sensitive voice-controlled systems such as virtual assistants have been introduced. While these systems are typically equipped with a speaker verification model, their vulnerabilities to multiple types of replay attacks have become a new security concern, e.g., an attacker can play a pre-recorded or synthesized speech sample to spoof the speaker verification system [1], [2], [3], [4], [5]. Therefore, developing an effective countermeasure to distinguish between genuine and replayed samples has become a recent research focus [6], [7], [8], [9]. While there have been many prior efforts in this area [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23]), they only focus on detecting replay attacks based on single-channel input and therefore only leverage the temporal and spectral features. However, we identified three reasons why a countermeasure designed using multi-channel audio input could provide improved performance. First, multi-channel audio captured by a microphone array contains spatial information, which can contain useful cues to help distinguish genuine and replayed samples [24], [25]. Second, it is relatively easy for an attacker to manipulate the temporal and spectral features to fool an anti-spoofing module [26] by simply modifying the replayed signal, while spatial features (e.g., time difference of arrival (TDoA)) are harder to manipulate and hence are more reliable. Third, multi-channel speech recognition techniques have been extensively studied and adopted [27], [28] and most modern far-field speech recognition systems are already equipped with microphone arrays, which makes it easy to obtain multichannel audio.
Only few prior studies focused on multi-channel voice anti-spoofing. In [25], the authors propose VoiceLive, which captures TDoA changes in a sequence of phoneme sounds to the two microphones and uses such unique TDoA dynamics to distinguish between replayed and genuine samples. The limitation is that this method requires the microphones to be placed very close (1-6cm) to the mouth. In [29], the authors use the "pop noise" caused by breathing to identify a live speaker based on two-channel input, where one channel is used to filter the pop noise and another is used as a reference. The limitation is that the pop noise effect disappears over larger distances and thus the method is only applicable to close-field situations. In [30], [31], the authors use generalized cross-correlation (GCC) of the non-speech sections of a stereo signal to detect the replay attack. The idea is that loudspeakers tend to generate electromagnetic noise during the non-speech section, and therefore the cross-correlation between the twochannel signals will be higher for replayed samples than genuine samples in the non-speech section. The limitation is that in order to make the electromagnetic noise detectable, a suitable background noise level and high-fidelity microphone is required, which is not always met in realistic settings.
In summary, previous multi-channel replay attack detection methods: 1) have only been designed for two-channel audio input while modern microphone arrays usually have more microphones and contain richer spatial information; 2) rely on hand-crafted features and calibrating the features for a different microphone array system or a different environment can be difficult; and 3) have relatively few applicable scenarios due to the close-field or low SNR requirement.
In order to overcome the above-mentioned limitations, in this paper, we propose a novel neural network-based replay attack detection model that has the following advantages. First, the proposed model is completely data-driven, i.e., no manual spectral or spatial feature engineering is needed, and the model can be used for inputs of any number of channels without knowing microphone array specifics (such as the array geometry) whenever training data is available. For the same reason, the proposed model can adapt to different environments using the training data and, therefore, there are no explicit constraints on usage scenarios. Second, all components (i.e., beamformer, feature extraction, and classification) are part of a neural network framework, which makes it easy to train them using existing massive optimizing methods and combine them with other neural-based countermeasure models. This work is the first neural-based multi-channel replay attack detector. We perform experiments using the recently collected ReMASC corpus [24] that contains genuine and replayed samples recorded in a variety of environments using different microphone array systems. We find that by leveraging the multi-channel audio input, a significant performance improvement (up to 46.6%) can be achieved compared to a single-channel input model. One classic microphone array signal processing technique is the filter-and-sum beamformer [27]. For a multi-channel audio with C channels x = (x 1 , x 2 , ..., x C ), the filter-andsum beamformer filters each audio channel x c using an Npoint FIR filter h c with a delay (or advance) of a steering time difference of arrival (TDOA) τ c to conduct the time alignment, and then sum the output of each channel together to obtain the output y: The filter-and-sum beamformer can be decomposed into two sub-processes: 1) estimating the τ t and 2) finding the optimal filter h. The first sub-process can be done by using a separate time-delay estimation module. But in order to implement the entire model in a neural network framework, we follow the method described in [28] to implicitly absorb the steering delay into the filter parameters and use a bank of P filters for each channel to capture different τ c : where: and where * denotes the convolution operation. In the remainder of this paper, we refer to h as a front-end filter to distinguish it from other filters in the neural network.
Usually, an optimal filter h is designed using a separate optimization objective (e.g., minimum variance distortionless response (MVDR) [32] or multichannel Wiener filtering (MWF) [33]), which is usually different from the objective of the actual learning task (e.g., word error rate for the speech recognition task). While this is acceptable for tasks such as speech recognition since low speech distortion and noise level are likely to improve the recognition accuracy, it might lead to the opposite effect for the replay attack detection task, because the filters might remove the useful cues in noisy or high-frequency components that the replay attack detector relies on. Therefore, a better strategy is to jointly optimize the beamformer with the replay attack detector. Note that our network design will also consider that replay attack detection is a sample-level classification task, i.e., there is only one label for each audio sample (either genuine or replayed) and thus using part of the input can be sufficient and will significantly lower the computational overhead.
We design the network based on the architecture presented in [28], [34], which was previously used for the speech recognition task. As shown in Figure 1, first, in order to lower the computational overhead, we use only the first 500ms of each audio and perform a non-overlapped framing of 20ms (frame length M =882 at a sample rate of 44.1kHz) and feed each frame x to the next layer. Then, we conduct the filter-and-sum beamforming as described in Equations 2 and 3 to obtain a 2-D time-frequency representation y for each frame. We use filter length N =630 and test different values for the filter number P in our experiments. Note that the filter-and-sum beamformer differs from the simpler delay-and-sum beamformer in that an independent weight is applied to each of the channels before summing them. Therefore, the filters h for each channel do not share the weight. Similar convolution layer designs have been widely adopted in image processing tasks to process the threechannel RGB input, but not for the purpose of beamforming.
To lower the computational overhead, we do not apply padding for this convolution layer. After that, we conduct a global maxpooling in time and apply a ReLU [35] nonlinear activation function to the beamformer output and get z ∈ R 1×P and feed it to a frequency convolution layer that consists of 256 1×8 filters with a max-pooling of size 3. The pooled output is then fed to a 256-dimensional fully-connected layer. The output o ∈ R 256 is the representation of this frame. We conduct the same above operations to all frames and feed the sequence of o to three stacked LSTM [36] layers, each with 832 hidden units and feed the output of the last frame to a single fullyconnected layer to obtain the prediction. The entire model is end-to-end from audio waveform to prediction and is trained jointly using a unified objective of minimizing the weighted cross-entropy loss. We re-weight the cross-entropy loss for each class using the normalized reciprocal of the sample number of the class in the training set to avoid the class-imbalance problem. To summarize, the proposed model has the following advantages: 1) It is completely data-driven, hence no manual feature engineering is needed, and the model can be used for inputs of any number of channels without knowing the microphone array information such as array geometry whenever training data is available. 2) All components (beamformer, feature extraction, and classification) are in the neural network framework, which makes it easy to train using existing massive optimizing methods. The intermediate tensor y is a standard time-frequency representation, so it is easy to further improve the model by combining with other advanced neural network-based replay attack detectors. 3) By taking advantage of the fact that replay attack detection does not necessarily need to use the entire sequence, a few strategies such as non-overlap framing and convolution without padding are used to speed up computation.

III. EXPERIMENTATION A. Dataset
We previously collected the ReMASC (Realistic Replay Attack Microphone Array Speech) corpus [24] to facilitate experiments on microphone arrays. The ReMASC corpus differs from other publicly available voice anti-spoofing datasets (e.g., the RedDots replayed dataset [37]) as follows: First, the ReMASC corpus contains recordings by a variety of microphone array systems instead of a single microphone. The microphone array geometry and the corresponding recording settings are shown in Figure 2 and Table I, respectively. Therefore, the ReMASC dataset is particularly well-suited for multi-channel voice anti-spoofing research.
Second, instead of using audio simulation tools [28], we recorded the ReMASC corpus in a variety of realistic usage scenarios and settings. Specifically, the data set contains recordings from 50 subjects of both genders and with different ages and accents. The recordings have been obtained in four different environments (two indoor, one outdoor, and one moving vehicle scenario) with varying types and levels of noise, and consisting of 132 voice commands. The distance between speaker and device varies from 0.5m to 6m, the dataset for the indoor environments uses different placements of the loudspeaker and the microphone array. The replayed recordings are played using a variety of loudspeakers. Therefore, the data has a large degree of variety. More details about the ReMASC corpus can be found in [24].

B. Data Splitting and Training Scheme
The ReMASC corpus is split into the core set (6331 genuine and 17175 replayed samples) and the evaluation set (2118 genuine and 14162 replayed samples). Both sets are speaker-independent and therefore strictly non-overlapping. In addition, there are additional synthetic speech data in the evaluation set. This configuration allows us to test the generalization capability of an anti-spoofing model.
In this study, we use the core set for training and development, and the evaluation set for testing. Specifically, we use 90% of the core set to train the model and use the development set to choose the batch size and initial learning rate as well as to implement the early stopping strategy (i.e., stop training when the evaluation metric on the development set stops improving). We use a learning rate decay with warm-up period strategy, i.e., the learning rate starts at the initial learning rate and linearly increases to 10 times larger in the first 20 epochs (the warm-up period), and then drops by half every 20 epochs until the equal error rate (EER) stops improving on the development set or the max epoch of 100 is reached. We select the batch size of 64 and the initial learning rate of 1e-5 through a grid search. We use ADAM optimizer with a l2norm regularizer [38], [39] with the weight decay coefficient of 1e-3. All experiments are repeated three times with different random seeds and the mean value is reported. We use EER as our evaluation metric. Since the four microphone arrays were mounted on a stand and recorded simultaneously during the data collection, the data volume recorded by each of the 4 microphone arrays is roughly 1/4 of the total data volume (D1 has less data due to hardware crashes in the data collection). Since the microphone arrays use completely different hardware and geometry, for our experiments, we train and evaluate the machine learning model separately for each microphone array by only using data collected by that array.

C. Models
We compare the following models in our experiment: 1) NN-Single: This model is exactly the same as the proposed model described in Section II except that only the first channel is fed to the neural network. By comparing this model with the proposed multi-channel model, we want to observe if multi-channel audio processing can indeed lower the error rate.
2) NN-Dummy Multichannel: This model is exactly the same as the proposed model in Section II, but here we feed multi-channel input into the neural network. The difference is that we feed replications of the first channel data as "dummy" multi-channel input to the neural network. The reason why we include this model in our experiment is that only comparing NN-Single with the proposed multi-channel model is not absolutely fair: when the number of input channels increases, the neural network architecture also changes. Specifically, the number of the total front-end filters (i.e., h) linearly increases with the input channel numbers. Therefore, it is possible that the performance difference between NN-Single and the proposed multi-channel model is actually due to the increasing number of front-end filters instead of the spatial information in the multi-channel input. In contrast, this NN-Dummy Multichannel model has exactly the same neural network architecture and number of parameters as the proposed multi-channel model, and therefore the performance difference between the two models can only be attributed to input difference, and we can observe if the spatial information in the multi-channel input can lower the error rate.
3) NN-Multichannel: This is the proposed model described in Section II. For each microphone array, all channels are fed into the neural network. Multichannel Improvement -6.6% -32.8% -40.2% -25.1%

1) Model Comparison:
The results of the model comparison are shown in Table II. In this section and III-D2, we use a fixed P of 64. Our key findings are as follows. First, the NN-Multichannel model clearly outperforms the NN-Single model, with an average EER improvement of 26.2%. For the microphone arrays recorded at 44100Hz (D1, D2, D3), we observe that the more channels the microphone array has, the larger the relative EER improvement by leveraging the multichannel input (e.g., the 2-mic D1 has a 6.6% lower EER, while the 6-mic D3 has a 40.2% lower EER). In contrast, the NN-Dummy Multichannel performs about the same as NN-Single. Since NN-Multichannel and NN-Dummy Multichannel have exactly the same neural network architecture and the only difference is the input, this demonstrates that the performance improvement of NN-Multichannel is not due to its larger number of front-end filters, but due to effectively leveraging the spatial information in the multichannel input. 2) Impact of the Number of Input Channels: In the previous section, we found that using all channels can significantly outperform using a single channel of the same audio. In this section, we further investigate the impact of the number of input channels. We gradually add the input channels from one channel to the total available channels of each microphone array and measure the EER. The order with which we add the microphones is (microphone indexed as shown in Figure 2): D1: 1-2; D2: 1-4-2-3; D3: 1-4-2-5-3-6; D4: 1-4-2-5-3-6-7. The rule is that we add the microphone furthest from the previously added microphone. For device 2, we further test a different order of 1-2-3-4. As shown in Table III, we have the following findings: First, while the results show that EER drops with an increasing number of microphones, indicating that more microphones help improve the defense performance, the best performance of D3 (46.6% multi-channel improvement) and D4 (30.2% multi-channel improvement) is not achieved when all microphones are used, but when only 5 and 6 microphones are used, respectively. We believe a possible reason is that too many channels may increase the risk of model over-fitting. Second, we observe that the performance improves most significantly when the second microphone (the microphone that has the longest distance from the first microphone) is added for D1, D3, and D4, and that the performance gradually saturates with more microphones. Nevertheless, D2 gets the most significant performance improvement when the third microphone is added, and if we switch the order to 1-2-3-4, we find that the most significant performance improvement is achieved when the second microphone is added. This indicates that it is not always optimal to use a microphone array with a larger dimension. We intend to explore these phenomena further in our future research. 3) Impact of the Number of Front-end Filters: In this experiment, we explore the impact of the number P of frontend filters per channel. As shown in Table IV, we find that the model performance does improve with the increasing number of front-end filters, but saturates around P = 64. The result is as expected since more front-end filters can help extract spatial and special features from the raw waveform.

IV. CONCLUSIONS
In this paper, we introduce a novel neural network-based replay attack detection model that leverages both the spectral and spatial information in the multi-channel audio and is able to significantly improve the replay attack detection performance. Compared to previous efforts, the proposed model supports arbitrary number of input channels and is completely datadriven in a neural network framework, which will make it easy to combine the proposed method with other neural-based anti-spoofing countermeasures.