{'fraction_non_alphanumeric': 0.02823074444086368, 'fraction_numerical': 0.0068320979697067356, 'mean_word_length': 4.738165680473373, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Communication-Cost Aware Microphone Selection For Neural Speech Enhancement with Ad-hoc Microphone Arrays

When performing multi-channel speech enhancement with a wireless acoustic sensor network, streaming information from all sensors can be prohibitive in terms of communication costs. However, not all sensors will be necessary to achieve good performance, which presents an opportunity to reduce communication costs. We propose a data-driven technique to leverage these opportunities by jointly learning a speech enhancement and data-request neural network. Our model is trained with a task-performance/communication-cost trade off. While working within the trade off, our method can intelligently stream from more microphones in lower SNR scenes and fewer microphones in higher SNR scenes. We evaluate the model in a complex echoic acoustic scene with moving sources and show that it matches the performance of a baseline model while streaming less data.


INTRODUCTION
Complex auditory scenes may contain dozens of mobile devices with microphones such as smart-phones, smartwatches, laptops, etc. Such ad-hoc arrays are typically referred to as wireless acoustic sensor networks (WASNs). It is often desirable to leverage all microphones within a WASN for tasks like multi-channel speech enhancement. This is of great interest thanks to applications such as hands-free teleconferencing. However, WASNs present unique complications due to the communication expense of continuously streaming from many devices and their unruly geometry.
When communication cost is not an issue, both beamforming and deep learning techniques have proven successful for multi-channel speech enhancement. These range from techniques that combine single-channel enhancers with beamformers [1,2] to fully learned models that leverage spatial features [3,4]. Most recently, some works have operated without explicitly encoding any spatial information [5].
To reduce communication costs, several beamforming models have been proposed. They define a task-performance vs communication-cost trade off to leverage the fact that some microphones will not be necessary for good performance [6,7,8]. By using a microphone selection algorithm in conjunction with a hand-tuned stopping criterion these algorithms decide whether to process data from a microphone without having seen it. In some cases, this selection step can be informed by sensor geometry.
However, these approaches are not amenable to being used with deep learning models due to the non-differentiability of selection. Motivated by this, we design a neural network that can learn in a data-driven fashion whether to process data from an unseen microphone, given a performance/cost trade off. We circumvent non-differentiability by formulating a specialized attention based mechanism that allows the model to learn a data-request mechanism jointly with a speech-enhancement model. Our mechanism is based on the adaptive compute mechanism explored in [9,10,11] and the multi-channel models in [12,13].
The novelty of our method is two-fold. First, using a learning-based mechanism for microphone selection and second, learning that mechanism jointly with the enhancement network. To evaluate our model we construct a challenging multi-channel speech enhancement task using an ad-hoc array of microphones inside an echoic scene with moving sources. Our experiments show that the performance/cost trade off can be easily selected and that the model works within the trade off to learn an adaptive data-request criterion capable of requesting more microphones in more complex scenes and fewer microphone in simpler scenes. For baselines we compare to models that select a fixed number of microphones and to a model that processes all microphones. We use short-time objective intelligibility [14] to measure speech enhancement performance and seconds of audio streamed to measure communication costs. The code is available here.

METHODS
The task of multi-channel speech enhancement is to recover some clean speech signal s from a set of M noisy reverberant mixtures x i . Each mixture is captured by a stationary microphone and is modeled by x i = s i + n i , i ∈ {1, · · · , M }, where both s i and n i are the result of convolving a room impulse response with an anechoic signal. Thus, a multi-channel speech enhancement model takes as input [x 1 , · · · , x M ] and returns an estimate of the clean speech,ŝ.

Network Architecture
To develop a multi-channel speech enhancement model that is communication-cost aware we develop a specialized data request mechanism. This mechanism allows the model to repeatedly request data from additional microphones until some learned condition is met. We construct this request mechanism by reinterpreting and repurposing the adaptive compute time mechanism presented in [9,10,11]. The proposed network architecture is depicted in Fig. 1. It consist of three components: a multi-channel encoder, a data request mechanism, and a single-channel decoder. The encoder processes microphones one at a time as requested by the request mechanism whose output is aggregated and fed to the decoder.

Encoder
The encoder operates on each microphone individually. It processes temporal chunks of single-channel mixtures as the request mechanism asks for them. It first computes the Shorttime Fourier Transform (STFT), and passes the magnitude through a series of 1D convolutional layers. For the i th microphone the encoder outputs Here, h represents the effective hop of all encoder convolutions and f represents the effective receptive-field. These constants are used in Alg. 1.

Data-Request Mechanism
The data-request network iteratively requests to stream from microphones based on the results of the aggregation and scoring modules. These modules are trained with a performance/cost trade off that influences how many requests are made. Specifically, if the model has already processed k channels, an attention layer aggregates across channels to produce an internal representation which is assigned a score. If the score is within the model's budget then it will request data from an additional microphone. The score is order invariant with respect to the k aggregated channels. Once the request budget is exhausted, the aggregated representation and scores are passed through a final layer to the decoder.
Aggregation is accomplished via a multi-head attention mechanism with internal dimension d, query Q[t] 1×d , keys K[t] k×d , and values V[t] k×d . Unlike traditional selfattention we use a single query computed from the most recently requested microphone to collect information across all previously requested microphones. The operations are given by . This output represents aggregated information when using the k requested mics at time t. The second step is a position-wise feed-forward network used to further process h[t] (k) [15]. Both steps are followed by dropout, a residual connection, and layer normalization. In Alg. 1 this entire set of operations is represented by AttentionLayer(z[t] (0:k) ).
The scoring and request module takes h[t] (k) , and calculates a score s k = S(h[t] (k) ) to determine if the model should request to stream data from another microphone. The scoring network outputs a scalar in the range [0, 1]. As microphones are requested and scored, the scores are accumulated in the variable c. If c is greater than the request budget 1 − , this signifies that we have seen enough data to cease requesting microphones. Otherwise, the model requests another microphone, computes h[t] (k+1) , and assigns a new score. The scores are constrained to sum to 1. If a score pushes the sum over one it is trimmed and referred to as the remainder r. Assuming N microphones requests are made, the final hidden state is computed as Once the request budget has been exhausted and its remainder r calculated, we pass h[t] through a dense layer, to obtain an input for the decoder. The value p = N + r is saved for use in the loss function. Thus, this reformulation of adaptive compute attempts to examine as little data as possible by weighting an internal representation of the multi-channel scene. The complete algorithm is shown in Alg. 1.

Decoder
The decoder operates on the aggregated outputs of the request mechanism. It predicts real and imaginary masks, denoted as M r and M i , respectively, via a series of 1D convolutional layers. The masks are applied to the STFT, computed in the encoder, to produce the clean speech estimate S = M r Re(X) + M i Im(X). The STFT domain estimate S is converted to the time domain representationŝ via the inverse STFT.

Loss Function
The full network is trained in an end-to-end fashion using two losses. The first loss L p (p) = T t=0 p[t] encourages the model to request less data. Intuitively, since p[t] represents the leftover request budget, minimizing it encourages the model to exhaust its budget sooner and thus request data from fewer microphones. The second loss measures speech enhancement performance between the true clean speech spectrogram S and the estimated spectrogram S. It is defined as L s (S, S) = | S| α − |S| α 1 where α = .3 is applied element-wise and intended to provide more weight to low energy frequencies.
The full loss is then L = L s (S, S) + λL p (p) where λ is a request penalty that specifies the performance/cost trade off.

Dataset Generation
To generate multi-microphone scenes we modified the Microsoft Scalable Noisy Speech Dataset [16] to use the pyroomacoustics [17] implementation of the image source method. To simulate a scene we select a speech sample, a noise sample, and rescale them to a random SNR in [− 10,10] dB. The rescaled samples are placed in a simulated 3-D room with width, depth, and height distributed uniformly in the interval [10, 15) meters, 10 th order reflections and an absorption coefficient of .35. Within this room, microphone locations are distributed uniformly at random. Both the speech and noise source move with their starting and ending location also selected uniformly at random. We generate distinct testing and training sets which do not have any speech or noise overlap. All scenes last two seconds.

Baseline Model
For comparison we construct two baselines. The first baseline uses the same architecture as the proposed model but has the request penalty λ set to 0 and therefore does not incur a penalty for requesting additional microphones. The second baseline has an identical encoder and decoder but does not perform adaptive data requests and instead processes a fixed number of microphones. All models have the same number of parameters.

EXPERIMENTS AND RESULTS
To evaluate the efficacy of our selection mechanism we train models with microphone request penalties ranging from 0 to 1e − 4. The penalty of zero serves as our first baseline and verifies that requesting more data is beneficial. Then, we select the 5e − 5 penalty model and compare it to our second baseline model which requests a fixed amount of data. This experiment studies whether the selected trade off is flexible and adapts to scene complexity.

Training Details
The model encoder receives two second clips of 16kHz audio and computes STFTs with a 512 point window, a hop of 128, and a Hann window. The magnitude of the STFT is passed through a series of two 1D convolutional layers each with 257 filters of size 5 with a hop of 1. The request network has a hidden size of 256, a multi-head attention mechanism with 4 heads and uses = 10 −4 . The decoder has four 1D convolutional layers with the same parameters as the encoder. The final mask uses a sigmoid activation and is applied on both the real and imaginary parts of the reference channel. We train the models with the Adam optimizer (lr = 10 −3 ), a batch size Request Penalty VS Audio Requested Fig. 2. Effect of different request penalties on the average amount of data requested by our proposed models. As the request penalty increases, less data is requested.

Performance Evaluation
To evaluate the speech enhancement performance of the model we use the short-time objective intelligibility metric (STOI) [14]. We chose STOI due its simplicity though other measures could be used as well. This is contrasted with the communication costs represented in seconds where costs are tracked as the average number of seconds of audio the model requested before making an enhancement prediction. Since all scenes last two seconds, a model will view at least two seconds of audio. If the model requests ten microphones, then this corresponds to it viewing 20 seconds of audio. We track this data cost since the array is ad-hoc and getting information from a specific microphone would require transmitting it. Measuring the communication costs in seconds of audio streamed avoids any confusion about audio representation such as sampling rate, bit depth, or codec.

Performance-Cost Trade Off
In this experiment we compare the average amount of data transmitted on the test set by models which where trained with different request penalties. Intuitively, requesting to stream from all microphones all the time can get you the best performance. However, some microphones are not useful and do not greatly affect performance. We verify this in Fig. 2, which shows request penalty on the x-axis and the average seconds of audio transmitted on the y-axis. As intended, when the request penalty is increased, the model requests less data. We highlight the model with no request penalty on the far left of the plot. Since this model requests to stream from nearly all microphones nearly all the time, we know that in this task there is a performance/cost trade off to be made.

Adaptability of Selected Trade Off
Here, we seek to evaluate whether the performance/cost trade off is learned in an intelligent manner. Given a communication budget, the simplest solution is to request a number of mics that matches that trade off for all scenes. However, this ignores the fact that some scenes are easier and require far fewer microphones and others are challenging requiring more microphones. In this experiment, we compare our proposed model with a request penalty of 5 × 10 −5 to a baseline model that always requests 18 seconds of audio. This fixedbudget was chosen to match the performance of our proposed model in the most complex scenes. We deploy both models in scenes with a variety of SNRs, and display the amount of data requested in Fig. 3 and the STOI scores in Fig. 4. The models achieve similar STOI scores though our proposed model does so while requesting less data.

CONCLUSION
We proposed a method for reducing data streaming costs while performing multi-channel speech enhancement with an ad-hoc microphone array. Our model jointly learns speech enhancement and data-request sub-networks that are trained with a task-performance/communication-cost trade off. Within the trade off, our model learns to scale with complexity, requesting less data in easier scenes and more data in harder scenes. We overcome the non-differentiability of communication cost by constructing a mechanism that penalizes excess data use. We evaluated our model on a challenging multi-channel speech enhancement task with moving sources, reverb, and a variety of speakers and noises. The proposed model displayed performance comparable to our baseline model despite using fewer microphones. We hope this work motivates future research in communication cost aware neural networks.