{'fraction_non_alphanumeric': 0.028388432736991646, 'fraction_numerical': 0.013363511593749247, 'mean_word_length': 4.599568558716462, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Efficient acoustic feature transformation in mismatched environments using a Guided-GAN

We propose a new framework to improve automatic speech recognition (ASR) systems in resource-scarce environments using a generative adversarial network (GAN) operating on acoustic input features. The GAN is used to enhance the features of mismatched data prior to decoding, or can optionally be used to fine-tune the acoustic model. We achieve improvements that are comparable to multi-style training (MTR), but at a lower computational cost. With less than one hour of data, an ASR system trained on good quality data, and evaluated on mismatched audio is improved by between 11.5% and 19.7% relative word error rate (WER). Experiments demonstrate that the framework can be very useful in under-resourced environments where training data and computational resources are limited. The GAN does not require parallel training data, because it utilises a baseline acoustic model to provide an additional loss term that guides the generator to create acoustic features that are better classified by the baseline.


Introduction
Automatic speech recognition (ASR) systems generally struggle to perform well when there is a mismatch between the data that was used to train the ASR system and new data that is decoded. This is especially true in under-resourced environments, where it is often prohibitively expensive to collect enough transcribed training data to cover even the most common application environments.
For some under-resourced languages, there are ASR systems that achieve low word errors rates (WERs) on audio similar to what the systems were trained with. However, when such a system is used to decode mismatched data, the WER is significantly higher. Some of the challenging aspects that lead to mismatched conditions include background noise, reverberation, encoding artifacts and microphone distortion. In cases where it is possible to collect a small dataset of training data that better matches the test conditions, it is usually very expensive to retrain an entire system to include the new data. This effect is magnified when new mismatched data regularly needs to be decoded. When computational resources are limited, it is not feasible to regularly retrain models or have multiple models that run in parallel.
In order to be useful in many different, constantly changing environments, a good baseline ASR system requires an efficient technique to adapt to the changing environments. An attractive solution is feature compensation because you can replace the adaptation technique without changing the acoustic model [1]. The features of noisy audio are changed or enhanced to increase the accuracy of the acoustic model [2]. Methods that are used as a front-end processing technique include masking time and frequency components [3], speech enhancement [4], deep denoising autoencoders [5] and generative adversarial network (GAN) based feature enhancement [6,7]. An alternative to feature compensation is model compensation, where the acoustic model is adapted in some way. Model compensation usually provides better performance gains than feature compensation, but at a much higher computational cost [1].
Multi-style training (MTR) is one of the most popular and well-established techniques used to address mismatch by allowing the acoustic model to learn robust representations of the data. MTR aims to transform the training data to be more representative of the testing data. New training datasets are created from an existing set by adding a series of MTR styles. These can include changing the speed and volume [8], speech style [9] or sampling rate [10], adding time and frequency distortions [11] or background noise, and simulating reverberation [12]. The styles are typically chosen without knowledge of the testing conditions and must still be able to handle a wide variety of mismatch. In addition, the number of styles that are added must be considered because the computational cost of training an ASR system increases significantly with each style that is added.
Recently, a number of GAN-based feature enhancement techniques have been proposed to improve ASR robustness to noise and reverberation [6,7,13]. A GAN framework uses two networks that compete in a min-max game and improve by learning from each other [14]. A generator network learns to create new samples using random noise as input. Another network, called the discriminator, is used to determine if the sample came from the true data distribution or the generator. The output of the discriminator is maximised for real samples and minimised for generated samples. The generator is trained using the output from the discriminator to improve the quality of the samples it generates. In ASR, the generator uses speech features or embeddings as input and aims to produce new features that are more robust to noise or reverberation [6,7]. GAN-based feature enhancement techniques can achieve very good performance improvements when a clean trained baseline ASR system is used [6,7,13].
In this work, we investigate the use of GANs to transform acoustic features of mismatched audio in underresourced environments. We want to improve the performance on a new mismatched test set with a limited computational budget, provided that an ASR system trained on good quality data already exists. Our GAN is trained to transform acoustic features of mismatched audio to be better classified by the acoustic model. Our technique can be applied to most acoustic features including Mel-frequency cepstral coefficients (MFCCs), filter-banks and speaker-adapted transforms like feature-space maximum likelihood linear regression (fMLLR). For the purposes of this paper, we focus specifically on WAV49-encoded audio [15,16], a full-rate GSM codec with a compression ratio of 10:1, often used by South African call centres to store large volumes of telephone calls. Call centre speech analytics is an important application of ASR, and due to the large volumes of audio generated by call centres, some form of compression is often encountered. This is why we specifically focus on noisy speech (also typical of call centre data) encoded with WAV49.

Related work
GANs can improve both deep neural network hidden Markov model (DNN-HMM) [13] and end-to-end ASR systems [6,7] by creating indistinguishable representations between clean and noisy audio samples. A clean corpus is augmented by adding noise or room impulse responses to create a new noisy corpus. GANs used for DNN-HMM ASR systems normally operate on acoustic features (log-power spectra, MFCCs) [13], and on log-Mel filterbank spectra or embeddings when used for end-to-end systems [6,7]. The generator maps the noisy features to the same representation as the corresponding clean features. The discriminator provides feedback to the generator using the clean and noisy features as input. An L1 or mean squared error loss between clean and noisy samples is minimised which assists the generator with the mapping [6,7]. This only works when the noisy set is created from the clean set, because the loss requires both samples to be of the same frames in a recording. All these approaches rely on the assumption that you know what the test conditions are (noise or reverberation). They are only as effective as the conditions you can create using a clean dataset. Still, GANs such as these that are used to improve features or embeddings have been shown to outperform data augmentation and DNN-based enhancement techniques [6,13].
These approaches are designed primarily for noise robustness and dereverberation, not necessarily mismatched audio. It is extremely difficult to accurately produce a noisy dataset using only clean audio if the conditions of the testing data are unknown. Sometimes it is possible to collect a small training dataset with matched conditions to the new test set. In this scenario, the previous GAN-based enhancement techniques cannot be used, because a parallel training set (clean version) does not exist.
An approach different from feature enhancement is to use a GAN to fine-tune an end-to-end ASR system [17]. The entire ASR model is treated as a generator network. The ASR system is trained normally until convergence occurs. After training, a discriminator network is tasked to determine if the transcriptions came from the ASR system or are ground truth. Since the ASR model is already trained, the output is very similar to the ground truth, which makes the task very difficult for the discriminator. The GAN can improve output transcriptions of an ASR model that has already converged. Adding the GAN-based fine-tuning technique yields consistent WER improvements [17].
Previous work focused on noisy and reverberant speech, with little attention paid to compressed audio. None of the GAN-based enhancement techniques are designed to work with a newly transcribed dataset that does not have a corresponding clean version. We are not aware of any research that utilises GANs for acoustic feature transformation in under-resourced ASR or call centre environments.

Guided-GAN
We use a GAN to improve an ASR system on mismatched (noisy and compressed) audio. Our approach can use any GAN loss function, but changes the loss in order to utilise an existing ASR system to guide the GAN training process. We refer to this architecture as a Guided-GAN.
Two datasets are used during GAN training. The first set can be a clean dataset or the same data used to train the acoustic model (known conditions). The second set, which is used as input for the generator, is a noisy or mismatched dataset that does not come from the same distribution as the clean set. Figure 1 shows a diagram of the Guided-GAN training process. Acoustic features of the mismatched audio are given to the generator as input. The generator then creates a new vector of the same dimension, which is used as input to both a pre-trained DNN acoustic model and the discriminator. The negative log-likelihood (NLL) loss of the acoustic model is added to the generator's loss to assist it in generating realistic samples of the correct class. The input to the discriminator is clean (or have known conditions) and generated features for which the output is maximised and minimised respectively. The goal of the GAN is to reduce the performance difference between data with known conditions and a new mismatched test set, for which a small transcribed training set is available. When the system is used to evaluate a test set, only the generator network is used to transform the noisy features before handing them over to the downstream acoustic model.

Loss functions
Any GAN loss function can be extended by adding the guiding term to the generator's loss. We use three different loss functions and select the one that is best suited for our approach. The first is an adapted version of a spectral normalisation GAN (SN-GAN) [18] in which we extend the generator loss function with an additional term. Specifically, we use the labels of the noisy sample to calculate the NLL loss of the cleaned sample. This value is added to the loss function to guide the generator to create features that are more likely to be classified correctly by the target acoustic model. An SN-GAN uses the same loss functions as a Wasserstein GAN (WGAN), but uses spectral normalisation instead of weight clipping in the discriminator to satisfy the Lipschitz constraint [18]. The standard WGAN loss for the generator is given by: where p g is the model distribution implicitly defined byx = G(x) withx the noisy input features, and D(x) is the output of the discriminator that can be a positive or negative value. We add the NLL loss to Eq. (1): where λ is a hyperparameter, p am is the probability defined by the trained acoustic model, andỹ is the senone class label of the noisy features. The WGAN loss function for the discriminator is given by: where p d is the real distribution defined by the clean samples x. Spectral normalisation is used in the discriminator to ensure that the Lipschitz constraint is satisfied [18]. We also compare the SN-GAN loss functions to the standard non-saturating GAN (NS-GAN) [14] and the WGAN with gradient penalty (WGAN-GP) [19]. The NS-GAN loss function we use for the generator is: with a corresponding discriminator loss function: The WGAN-GP loss function for the generator is the same as for the SN-GAN in Eq. 1. The Lipschitz constraint is satisfied in the discriminator by adding a gradient penalty term to Eq. 3: where λ gp is a hyperparameter controlling the gradient penalty, and α is a uniform random number between zero and one.

Network architectures
We compare two different network architectures for the task. The first architecture is based on the encoderdecoder architecture of a speech enhancement GAN (SEGAN) [20]. The original network operated on raw audio and added random noise between the encoder and decoder networks. Our implementation uses fMLLR features as input and we do not add any noise. This architecture is described in Section 3.2.1.
The second architecture we use improves the efficiency of the first by reducing the number of layers and removing the downsampling and upsampling processes. The parameters are fewer and allows for much faster training. This architecture is referred to as a fully-convolutional network because the output dimension will always equal the input dimension for any given input dimension. We later show in Section 5.1 that the performance of the two networks is similar in terms of the final WER for a system. The fully-convolutional network architecture is described in Section 3.2.2.

Encoder-decoder architecture
A diagram of the generator and discriminator architectures for the SEGAN inspired networks are shown in Figures 2 and 3. We do not add any random noise or dropout to the generator during training or evaluation, which allows for a deterministic network. We found that the results were more consistent this way. The encoder network (shown in grey blocks) uses five convolutional layers with a stride of two for downsampling. The first two layers have a kernel size of seven, then two layers with a kernel size of five and the last layer has a kernel size of three. All layers in the generator, except for the output layer, use ReLU activation functions. The decoder network (shown in orange blocks) has five transposed convolutional layers with a stride of two for upsampling. The first and second layer uses a kernel size of four and five, respectively, followed by three layers with a kernel size of six. The choices for the kernel sizes were based on experimental results and architectural reasons (to upsample to the same dimension as the corresponding layer in the encoder). Skip connections are used to splice the feature maps of the encoder layers to the corresponding decoder layers. This is done because the encoder network extracts high-dimensional features and would struggle to produce an accurate output if it only upsamples these high-dimensional features. By splicing the feature maps of the encoder network, the features before the encoder operated on them can assist the generator to produce a better output. The high-dimensional features then assist the layers in the decoder to improve the upsampling process by removing noise from the original features.
The discriminator network that works with the SEGAN-inspired generator has eight convolutional layers followed by a fully-connected output layer. The first layer uses a kernel size of 41, the rest of the layers use a kernel size of 13. The network downsamples the features with max-pooling layers. All convolutional layers use leaky ReLU activation functions with a negative slope of 0.2. Dropout is used only in the first two layers with a probability of 0.3. The fully-connected output layer uses spectral normalisation and has a sigmoid activation function.

Fully-convolutional architecture
The fully-convolutional generator architecture is shown in Figure 4. We use a kernel size of five for all layers, with zero padding to ensure that the dimension stays the same. The first four layers use leaky ReLU activation functions with a negative slope of 0.2. The last layer creates the final output without any activation function. We do not use dropout or add random noise at any stage in the generator.
The discriminator, shown in Figure 5, is a deep convolutional neural network with max-pooling layers and a fully-connected output layer. All layers, except the output, use leaky ReLU activation functions with a negative slope of 0.2. Dropout is used in the first three layers with a probability of 0.25. The output layer is a fully-connected layer with spectral normalisation and a sigmoid activation function. The sigmoid function is used to bound the output of the discriminator to ensure that the magnitude of the NLL loss and discriminator loss are comparable (ratio is controlled by the hyperparameter λ).

Performance metric
We use two different metrics for hyperparameter selection and reporting system performance. The most accurate and objective metric to select the best hyperparameters for the GAN is the WER on a development set. Measuring the WER requires decoding, which takes significant time and resources. An approach that is computationally much cheaper and correlates well with WER (correlation is shown in Appendix A), is to measure the senone error rate (SeER) of the baseline DNN acoustic model when given the generated features as input. SeER is measured before decoding, and is the error rate of tied context-dependent states. The hyperparameters of the GAN are tuned to get the lowest SeER on the development set. We also use early stopping to select the generator which had the lowest SeER. To evaluate and compare the final system performance of different models, we use WER.

Experimental setup
In this section, we describe the experimental setup that we use. We first describe the acoustic model and how it is trained, followed by the training and optimisation procedure for the GAN. We then describe our MTR setup that is used as a competitor for the GAN. Finally, we describe how a controlled environment is created using the LibriSpeech corpus [21]. The toolkit used for experimentation is publicly available 1 .

Acoustic model
We use the baseline DNN acoustic model architecture of the Pytorch-Kaldi toolkit for our experiments [22]. The model is a 5 hidden layer multi-layer perceptron (MLP) with 1 024 units per layer; it uses ReLU activation functions, dropout and batch normalisation. We use fMLLR transformed input features with a context window of 11 frames. The network is trained with the SGD optimiser with a learning rate scheduler that halves the learning rate when the relative improvement 2 on the development set is less than 0.001. All networks are trained until convergence. The hyperparameters are optimised to minimise WER on the development set. The best batch size and learning rate combination is determined by performing a grid search. When scoring the development set, we select the best combination of language-model weight and word insertion penalty. We report WER on a held-out test set using the hyperparameters determined on the development set. On the standard LibriSpeech 100-hour training set [21], our results (9.18% WER on test-clean) are comparable with state-of-the-art results (using the 100-hour training set and the small language model, tg-small) [22,21,23].

GAN training and optimisation
The GAN uses the same fMLLR features that is given to the acoustic model as input for both the generator and discriminator networks. The GAN is trained until the SeER converges on the development set. In initial experiments, the batch size and learning rates (independent learning rates for generator and discriminator) had the biggest impact on the SeER. We find the best combination of batch size and learning rates in a grid search. After the search is complete, we then expand the search in that area to find better results when varying the acoustic model λ. When changing the λ, we also change the batch size and learning rates again. We use the Adam optimiser with equal updates to the generator and discriminator. After the GAN is trained, the DNN acoustic model is evaluated with the generator operating on the features. In a second step, we also measure how fine-tuning the acoustic model can further reduce the WER. During fine-tuning, the pre-trained model is trained further using the cleaned features for a few iterations, until the SeER converges.

Multi-style training
An MTR model is trained to serve as a competitor for the GAN model. We apply speed and volume perturbation to the datasets to create a more diverse training set. The perturbations are applied to the same datasets we use as 'noisy' set for GAN training each time to ensure a fair comparison.
The speed and volume of the training data are changed with 10% and 20%, respectively [24]. The speed is either increased or decreased with equal probability -approximately half of the utterances have a slower speed compared to the original set and the other half have a faster speed. Volume is handled in a similar way. All audio processing is done using Sox 3 . Different combinations of speed and volume perturbation were tested to see the combined performance impact on the development set [24]. We only report on the results for the best setup on each dataset.

Controlled environment using the LibriSpeech corpus
A controlled environment is created to develop a GAN and make certain design choices. We use the LibriSpeech corpus for this, because it is a freely available corpus with separated clean and noisy recordings. Because we are studying a resource-scarce environment, we limit our training data to 100 hours and only use the small language model (tg-small).
The 100-hour clean subset of the LibriSpeech corpus is used as 'clean' set for GAN training and to develop a baseline model. An encoded set is created by downsampling the audio to 8kHz and applying WAV49 encoding to the clean set. After encoding, the file is upsampled back to 16kHz to train a wide-band model. This is done because we want to use a GAN as a front-end to an existing 16kHz system. The upsampling process used does not attempt to interpolate values to add high-frequency information that was lost during downsampling. Only the existing lower frequency components are retained. We also create a development and test set using the dev-clean and test-clean subsets of the LibriSpeech corpus. In under-resourced environments, it can very expensive to collect many hours of training data. To show the ability of a Guided-GAN to work with limited training data, we create datasets ranging from 1 hour to 100 hours using the train-other-500 subset of the LibriSpeech corpus. We also encode the data with WAV49 encoding. Random speakers are selected to get a more diverse training set. The largest set (100 hours) contains 250 speakers. The datasets we use for the second experiment are shown in Table 2.

Experiments
In this section, we utilise our Guided-GAN approach to improve the decoding accuracy of a baseline acoustic model and compare the results to an MTR system. We first compare three different loss functions and two GAN network architectures on a WAV49-encoded LibriSpeech corpus. WAV49 encoding is used to demonstrate the ability of the GAN to improve audio features of encoded recordings. Compression and low sampling rates are typical of low-resource datasets, which is why we encoded the LibriSpeech corpus. We then reduce the amount of training data available to the GAN and demonstrate its effectiveness with limited data. We compare the GAN to an MTR system using the same training data. Finally, we evaluate the Guided-GAN on a proprietary South African Call Centre corpus to demonstrate its ability to work on real-world mismatch datasets.

Clean WAV49-encoded LibriSpeech corpus
Our first experiment uses the LibriSpeech corpus that is encoded with WAV49 encoding. The datasets used in this experiment are shown in Table 1. A baseline acoustic model is trained using the train-clean-100 subset. We improve this model by using the encoded training dataset (train-clean-e-100) and retraining a new system. We also train an MTR system with added speed perturbation.
A GAN is trained using the train-clean-100 subset as 'clean' set and the train-clean-e-100 set as 'mismatched' set. We compare the three GAN loss functions and two network architectures described in Sections 3.1 and 3.2, respectively. The first GAN we train uses the baseline train-clean-100 acoustic model for guidance. This model was trained using 16kHz unencoded audio which requires the GAN to compensate for sampling rate  11.02 ± 0.03 -11.10 ± 0.02 train-clean-e-100 + s (MTR) 10.93 ± 0.08 -11.00 ± 0.07 train-clean-100 NS-GAN (E-D) 13.45 ± 0.12 11.14 ± 0.05 --WGAN-GP (E-D) 13.52 ± 0.04 11.14 ± 0.02 --SN-GAN (E-D) 12.80 ± 0.01 11.04 ± 0.03 --SN-GAN (F-C) 13.41 ± 0.04 10.93 ± 0.03 13.59 ± 0.03 11.04 ± 0.02 train-clean-e-100 SN-GAN (E-D) 11.29 ± 0.04 10.81 ± 0.04 --SN-GAN (F-C) 11.36 ± 0.02 10.79 ± 0.03 11.58 ± 0.11 10.94 ± 0.02 differences and encoding. Another GAN is trained using the train-clean-e-100 acoustic model, which already used 8kHz encoded audio for training. In this case, the GAN does not have to compensate for sampling rate differences or encoding because the acoustic model was already trained on the same type of audio. Table 3 shows the results on the dev-clean-e and test-clean-e subsets. The top section shows the results for the baseline, encoded and MTR system that used speed perturbation (train-clean-e-100 + s) 4 . The GANs are evaluated before and after fine-tuning (indicated by 'FT' in the column heading). The GANs using the encoder-decoder network architecture are labeled with 'E-D' and the networks using the fully-convolutional architecture with 'F-C'. The middle section shows the results for the GANs using different loss functions and the train-clean-100 acoustic model for guidance. The GANs in the bottom section use the train-clean-e-100 acoustic model for guidance.
All four GANs using the baseline acoustic model significantly improved the performance. Fine-tuning further reduced the WER to be comparable with MTR. The SN-GAN loss function performed better than the NS-GAN and WGAN-GP for this task. The two network architectures, encoder-decoder and fully-convolutional, performed very similar. Both GAN models using the train-clean-e-100 acoustic model without fine-tuning performs worse than the model used to train them. When fine-tuning this model, the performance is slightly better than the MTR system.
The two GAN models using the fully-convolutional architecture are also evaluated on the test-clean-e set to confirm the results. The results are similar to those on the development set, although the average WER of the MTR model was slightly lower than the train-clean-100 GAN. These results confirm that the Guided-GAN is effective to recover most of the lost performance due to the reduced sampling rate and WAV49 encoding. The results of the GAN is comparable to MTR.

Simulated resource-scarce environment using the LibriSpeech corpus
We use the train-clean-100 baseline model trained in Section 5.1 to train Guided-GANs using the noisy datasets shown in Table 2. For each GAN, random utterances from the train-clean-100 set is used as the The WER of the clean baseline model on the standard test-clean set is 9.18%, and increases to 42.52% when it is evaluated on a noisy WAV49 encoded set (test-other-e). Noise and encoding is highly detrimental to the performance of this model. This large generalisation gap is due to a mismatched training and test set. The two major differences between the training and testing data is that the test set is encoded using WAV49 encoding and contains various forms of noise (additive noise, poor quality recordings, etc.). We improve this model by adding a Guided-GAN front-end. We train our GAN using different amounts of training data and also fine-tune the acoustic models each time. Table 4 shows the WER on the dev/test-other-e sets. Using only one hour of noisy data for GAN training, improved the baseline by 20.2% relative WER. Adding additional noisy data reduced the WER further, however, the returns were diminishing.
The Guided-GAN can significantly improve a clean baseline acoustic model using very little matched training data. In resource-scarce environments, where it is sometimes difficult to collect many hours of transcribed data, a Guided-GAN can be a useful tool to adapt an existing system. An additional advantage of the architecture is that the GAN is quite fast to train, especially on small datasets.
Different GANs using the train-other-e-100 set are compared to MTR systems in Table 5. The top section shows the results for the two baseline models trained with the train-clean-100 and train-clean-e-100 sets, also used by the GANs for training. The second section shows the results for models that improve the baseline results by using MTR and/or encoded noisy training data. We show results for the MTR models that use speed and volume perturbation in a single set (train-other-e-100 + sv) and two sets separately (train-other-e-100 + s + v). Two GANs are trained using the train-clean-100 and train-clean-e-100 acoustic models for guidance. We fine-tune the acoustic models (indicated with an 'FT' label) with the generator operating on the features. We also fine-tune the acoustic model using the MTR datasets to further improve the WER.
Applying the GAN to the train-clean-100 and train-clean-e-100 models improved the WER of the baseline acoustic models by 20.7% and 6.0%, respectively (relative improvements). Fine-tuning the acoustic models using the train-other-e-100 set further reduced the WER by 18.2% and 7.7%, respectively. Fine-tuning the train-clean-e-100 acoustic model using the speed-and volume-perturbed datasets yielded the best performance of all GANs. The results of the Guided-GANs are very comparable to MTR in terms of WER. In Section 5.3, we compare the training time of the models to see the true benefit of the GAN.

Comparing training times
We compare the training time of Guided-GANs using different amounts of training data to MTR systems. Table 6 shows the training time in minutes and test WER of each model.
Training a GAN requires more memory than training a standard MLP acoustic model because the GAN uses three networks during training (generator, discriminator and acoustic model). Although it uses more memory, the training time is less than training an MTR model because the GAN converges much faster than an acoustic model. The GAN also uses much larger batch sizes during training, which also speeds it up significantly.
The training time is measured using a desktop computer running an Intel i7 8700k CPU with an Asus RTX 3080 GPU. The system has 32 GB RAM running at 3 200 MHz. The baseline and MTR models were trained for 24 epochs. The GANs using the 100-hour training set were trained for 20 epochs. The GANs using less data were trained for more epochs. All networks converged before the end of training was reached. We always use early stopping on the validation SeER to select the best GAN network, but for this experiment, we compare the training time until the end of training.
Training a Guided-GAN using the fully-convolutional architecture and fine-tuning is about three times faster than training an MTR model using two datasets and four and a half times faster than one using three sets. The MTR systems perform very similarly to the GAN in terms of WER. Fine-tuning the GAN with the MTR datasets is still two times faster than training an MTR model using the same sets. The GAN is also extremely efficient with limited training data. A relative WER improvement of 20.2% on the train-clean-100 model is achieved with only 10 minutes of GAN training, including fine-tuning time. With 10 hours of training data, the relative WER improvement increases to 27.9% with a GAN trained in 52 minutes.
This shows that the Guided-GAN is not only a viable replacement for MTR in resource-scarce environments, it is also an efficient technique to improve a good baseline system on mismatched data. The only requirement is that a small, transcribed dataset with conditions similar to the test set is available.

Call centre dataset
To evaluate the effectiveness of a Guided-GAN on a real-world resource-scarce dataset, we use a proprietary South African Call Centre (SACC) corpus. A baseline system is trained using 48.8 hours of audio recorded in  a specific call centre in South Africa (Baseline train). The lexicon and language model are specific to this call centre and are used by all models. We use the same MLP acoustic model and training procedure described in Section 4.1. This baseline model achieves 28.41% WER on a development set (Baseline dev) containing calls from the same call centre not included in the training data. This is considered matched data, since the calls originate from the same call centre.
Small datasets from eight different call centres are available as mismatched sets, shown in Table 7. The data in these sets are extremely limited, ranging from 11 minutes to 75 minutes. For each call centre, a train, development and test set are available. The datasets have different sampling rates, bitrates and encoding types, shown in Table 7 for each set. We select three call centre datasets to train individual GAN models on and train another GAN using all eight datasets (labeled 'Mixed centres'). This mixed set is used to train a GAN with more conditions (from all call centres) to show the ability of the GAN to generalise to a number of different environments. The data between call centres differ in terms of: recording environments, noise conditions, dialects and accents, jargon, noise levels, encoding, etc.
The Baseline train set is used as the 'clean' or 'matched' set for GAN training and the training set of the new call centres as 'mismatched' set. The hyperparameters are optimised to minimise the WER on the development set. After the best hyperparameters are selected, a new GAN is trained with the training and development set to expand the training data. This is done because the datasets are extremely small and additional data is very beneficial.
The WER results for the GANs compared to the baseline model are shown in Table 8. The performance of the baseline model on the new mismatched datasets are extremely poor. All WERs on the test sets are above 50% for the baseline model. Applying the Guided-GAN as a front-end to the baseline model consistently improves the results on all sets. The improvements on the development set ranged from 5.45% to 24.74%. The best test set improvement was found on Call centre 3, with only 9.5 minutes of training data, the relative WER improvement was 19.70%.

Analysis
In this section, we investigate the improvements made by a Guided-GAN. We compare the errors made between the baseline and the GAN in Section 6.1, and visualise the improvements on a frame level in Section 6.2.

Error comparison
We compare the errors made by the train-clean-100 baseline and Guided-GAN by using the train-clean-100 acoustic model. Table 9 shows a summary of the errors made by the two models. The GAN reduced the number of insertions, deletions and substitutions, which resulted in a lower WER. On the frame level, the GAN improved the SeER by 14.9% relatively. This shows that a Guided-GAN improves the acoustic model on all types of errors and not just a specific type.

Feature transformation
To visualise the transformation that a Guided-GAN makes on a frame level, we plot the fMLLR features of two clean, encoded and generated samples. We use the features of a clean sample and the WAV49-encoded version of the same sample. We show two examples in Figure 6. The encoded and generated features on the left were classified incorrectly by the acoustic model. In the figure on the right, the encoded features were classified incorrectly, but the generated features were classified correctly.
On average, the encoded features correlate slightly better with the clean features than the generated features with the clean features. This indicates that the generator does not aim to recover the original features, but instead changes the features in some way so that the acoustic model better classifies them. This is further confirmed by using a trained generator on another acoustic model that was not used for guidance (different initialisation seed used for the acoustic model). If the features were closer to the original features, we would expect that the new acoustic model will also be improved, but this is not the case. Instead, the performance is worse than when using the model without a GAN. The Guided-GAN can only be used to improve the model   that was used for guidance during training because the generator only learns how to improve the classification accuracy of that specific model.

Discussion
Feature transformation using a GAN can significantly reduce the WER of a baseline model when there is a mismatch between the training and testing audio. Similar improvements are observed as when using MTR, at less cost: training a GAN front-end takes much less time than retraining an ASR system using MTR. The GAN can be used with or without fine-tuning, with a fine-tuned acoustic model producing improved accuracy but at an additional computational cost. Fine-tuning the GAN with MTR datasets provided further improvements.
The acoustic model used to calculate the acoustic model loss for the generator during training is very important. The train-clean-100 baseline never used encoded audio for training, which meant that it had no idea how to handle encoding. This requires the generator to learn a transformation that can compensate for sampling rate differences and remove encoding artifacts from the features. Since WAV49 is a lossy encoding, the task is very difficult, and the large gain observed from the train-clean-100 baseline when only the GAN was added (14.95% absolute WER) is very promising.
This work is aimed primarily at under-resourced environments where computational capabilities are limited. The GAN is fast to train and can be plugged into existing systems without other modifications. This is very useful if a strong baseline model has already been developed, but there is a desire to use it on new data from a mismatched environment. Using MTR or other enhancement techniques that require manual creation of a noisy or better matched dataset are limited to the conditions that can be artificially created. It is extremely difficult to accurately create a noisy dataset that is well matched to a new set of conditions. The performance of a model using a dataset that was manually created is far worse than one with perfectly matched conditions [24]. When a small dataset with matched conditions is available, current GAN techniques cannot be used to reduce the mismatch, because a corresponding clean version does not exist (they require both the clean and noisy version to minimise the L1 or mean squared error loss). The Guided-GAN does not have this limitation because it utilises a baseline acoustic model to provide guidance during training instead of an L1 or mean squared error loss.

Conclusion
Our GAN framework can significantly improve a baseline ASR system that is evaluated on mismatched data. The Guided-GAN solves many of the shortcomings of existing GAN or enhancement systems. Current techniques require parallel clean and noisy training data. It is extremely difficult to accurately reproduce a training set that is matched to unknown noise conditions. The developed GAN can utilise a small matched dataset to adapt an existing baseline system trained on good quality data to also perform well on the mismatched data.
The practical relevance of a Guided-GAN is realised when a strong commercial-grade ASR system is required to decode new mismatched data. In commercial applications, it is very expensive to retrain or tweak an ASR system that has already been optimised for a given set of conditions. The strong ASR system can be re-used on the new dataset by using a Guided-GAN. A small training set from the target environment can be collected and manually transcribed without much difficulty. The Guided-GAN can then use this training set to adapt the strong ASR system to improve its performance on the new dataset.

A Appendix: Correlation between WER and SeER
To confirm the correlation between WER and SeER, we measure both during GAN training on the LibriSpeech corpus. Ten points are selected where we measure the SeER and perform decoding using the MLP acoustic model. The data points were selected ranging from the early stages in training up to a much later point where the network has converged. We do not fine-tune the acoustic model in this experiment. Figure A.1 shows the WER versus SeER during GAN training. A strong correlation exists between WER and SeER. We also evaluated eleven different types of GAN models after training. A graph of WER versus SeER for these models after training is shown in Figure A.2. The models in this graph used different network architectures, hyperparameters and loss functions. The correlation also holds here, despite the setups being very different from one another.