{'fraction_non_alphanumeric': 0.03582052165200238, 'fraction_numerical': 0.01405892851551492, 'mean_word_length': 4.610047426664325, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adversarial Meta Sampling for Multilingual Low-Resource Speech Recognition

Low-resource automatic speech recognition (ASR) is challenging, as the low-resource target language data cannot well train an ASR model. To solve this issue, meta-learning formulates ASR for each source language into many small ASR tasks and meta-learns a model initialization on all tasks from different source languages to access fast adaptation on unseen target languages. However, for different source languages, the quantity and difficulty vary greatly because of their different data scales and diverse phonological systems, which leads to task-quantity and task-difficulty imbalance issues and thus a failure of multilingual meta-learning ASR (MML-ASR). In this work, we solve this problem by developing a novel adversarial meta sampling (AMS) approach to improve MML-ASR. When sampling tasks in MML-ASR, AMS adaptively determines the task sampling probability for each source language. Specifically, for each source language, if the query loss is large, it means that its tasks are not well sampled to train ASR model in terms of its quantity and difficulty and thus should be sampled more frequently for extra learning. Inspired by this fact, we feed the historical task query loss of all source language domain into a network to learn a task sampling policy for adversarially increasing the current query loss of MML-ASR. Thus, the learnt task sampling policy can master the learning situation of each language and thus predicts good task sampling probability for each language for more effective learning. Finally, experiment results on two multilingual datasets show significant performance improvement when applying our AMS on MML-ASR, and also demonstrate the applicability of AMS to other low-resource speech tasks and transfer learning ASR approaches. Our codes are available at: https://github.com/iamxiaoyubei/AMS.


Introduction
Automatic Speech Recognition (ASR) has attracted a lot of attention recently and achieved significant improvements (Chan et al. 2016;Graves et al. 2006;Pratap et al. 2019) brought by the success of deep neural networks. However, building an end-to-end deep ASR model often requires huge transcribed training data, which is impractical for the low-resource languages due to the scarcity of audio data and the huge labor resources consumed in transcription.
To solve this issue, many works are devoted to develop low-resource ASR approaches. The representative methods in this line are transfer learning ASR (TL-ASR) (Hu et al. 2019;Kunze et al. 2017), multilingual transfer learning ASR (MTL-ASR) (Adams et al. 2019;Cho et al. 2018;Tong, Garner, and Bourlard 2017) and multilingual meta-learning ASR (MML-ASR) (Hsu, Chen, and yi Lee 2020) that all aim to learn an ASR model initialization from source languages such that the initialization can quickly adapt to target language via finetuning on a few data. Among them, TL-ASR considers one source language and regards the pretrained ASR model on the source data as a model initialization. But as shown in Fig. 1 (a), the learnt initialization by TL-ASR often overfits the source language and cannot quickly adapt to a different target language. To resolve this issue, MTL-ASR and MML-ASR consider multiple source languages. Inspired by multi-task learning, they both sample partial data from each language domain to construct a small speech recognition task. Then for each sampled task, MTL-ASR directly trains its model on this task, while MML-ASR adapts its ASR model to the validation data of the task via fine-tuning on a few training data of the task and minimizes the validation loss. In this way, the learnt initializations by MTL-ASR and MML-ASR can usually fast adapt to the target low-resource language, as both MTL-ASR and MML-ASR learn the common knowledge from all tasks from different language domains which facilitates learning target languages.
However, when sampling tasks from these language domains, MTL-ASR and MML-ASR often ignore the underlying task imbalance issues which could result in unsatisfactory performance. First, different kinds of languages have different training data scales so the underlying task quantity for each language domain varies greatly, which leads to the task-quantity imbalance. Second, as different languages have diverse phonological systems, the tasks drawn from different language domains have various recognition difficulties, causing the task-difficulty imbalance. In this way, for MTL-ASR and MML-ASR, both uniform sampling that uniformly samples tasks from each language domain and more advanced task-quantity-balanced sampling whose sampling rate for each domain positively relies on its task quantity (data scale) cannot handle the task imbalance issue. Uniform sampling neither considers the task-quantity imbalance nor the task-difficulty imbalance, while task-quantity- (a) Initialization learnt in TL-ASR overfits the only one source language. Initializations in MTL-ASR (b) and MML-ASR (c) are close to the optimal model of the large-scaled language and departure from the small-scaled languages. (d) Initialization learnt by our AMS has a more balanced distance to all languages because of our adaptive sampling to handle the task imbalance. balanced sampling directly ignores task-difficulty imbalance. So the learnt initializations by MTL-ASR and MML-ASR are biased, and cannot fast adapt to the target language. For instance, as shown in Fig. 1 (b) and (c), when using taskquantity-balanced sampling in MTL-ASR and MML-ASR, their learnt initializations are often close to the optimal model of the large-scaled language and departure from those of the small-scaled languages. So the learnt initializations are often far from the optimal models of target languages which usually locates around all source languages, and cannot be fast adapted to target languages via fine-tuning on low-resource training data.
Contributions. To resolve the above task imbalance issue, we develop a novel adversarial meta sampling (AMS) method for multilingual low-resource speech recognition. This AMS can effectively help both MML-ASR and MLT-ASR handle task imbalance problem and boost their performance. Considering the superior performance of MML-ASR over MLT-ASR, in this work we spend more efforts to introduce AMS on MML-ASR. Specifically, we observe that the query losses of tasks from each language domain can well measure both task-quantity imbalance and task-difficulty imbalance. For each language domain, if its tasks are not well sampled in terms of its task quantity and task difficulty, then its tasks will have relatively large query losses. It means that the language domain with large task query loss requires more training. So we design a policy network to increase the query loss of MML-ASR model through adversarial learning for sampling from proper language domain. Our policy network incorporates LSTM (Hochreiter and Schmidhuber 1997) structure and attention mechanism to adaptively predict the most befitting task sampling probability for each language domain by using the long-term information in LSTM and the current query losses at each training iteration. Through such an online and adversarial manner, the sampling policy is dynamically changed along with the training state of the MML-ASR. In this way, the language domain that are not well sampled in terms of its task quantity and task difficulty will be sampled more in the later training iterations, making the MML-ASR model learn a more balanced initialization for better adaptation to the target languages as shown in Fig. 1 (d).
Moreover, we validate our method on several datasets with diverse languages selected from Mozilla Common Voice Corpus (Ardila et al. 2020) and the public IARPA BABEL dataset (Gales et al. 2014). The experimental results demonstrate that our AMS significantly improves the performance over the existing approaches on low-resource ASR, especially under the realistic task-imbalance scenarios. Furthermore, we conduct experiments on speech classification and speech translation, which proves that our AMS can be easily generalized to improve other low-resource speech tasks.

Related Work
Transfer learning ASR. To alleviate the need for labeled data, recent works utilize unsupervised pre-training and semi-supervised methods to exploit unlabeled data, e.g. wav2vec (Schneider et al. 2019), predictive coding (Chung and Glass 2020), self-training (Kahn, Lee, and Hannun 2020) and weak distillation (Li et al. 2019). But they still require substantial unlabeled data which is unavailable for some minority languages. To solve this issue, transfer learning is explored via using other source languages to improve the performance of low-resource languages (Kunze et al. 2017), which requires that the source and target languages are similar and the source language has sufficiently large data. Moreover, multilingual transfer learning ASR (Dalmia et al. 2018;Watanabe, Hori, and Hershey 2017;Toshniwal et al. 2018) is developed using different languages to learn language-independent representations for performance improvement under the low-resource setting. Meta-learning ASR. Meta-learning approaches (Zhou et al. 2019(Zhou et al. , 2020 can meta-learn a model initialization from training tasks with fast adaptation ability to new tasks with only a few data and thus is suitable to handle low-resource data learning problems. Especially, Hsu et al. ) and Winata et al. (Winata et al. 2020) adopted MAML (Finn et al. 2017) for low-resource ASR and code-switched ASR and both achieved promising results. But these method ignores task imbalance in real-world scenarios and equally utilizes the meta-knowledge across all the languages, which leads to performance degradation. To alleviate quantity imbalance, Wang et al. (Wang, Tsvetkov, and Neubig 2020) improves differentiable data selection by optimizing a scorer with the average loss from different languages to balance the usage of data in multilingual model training. Besides the language quantity, our AMS also considers the language difficulty and learns the sampling policy in an adversarial manner. Adversarial learning ASR. Inspired by domain adversarial training (Ganin et al. 2016), recent works introduced adversarial learning into ASR to learn robust features invariant to noise conditions (Shinohara 2016) and accents (Sun et al. 2018b). Besides, some researchers use a domain-adversarial classification objective over many languages on multilingual ASR framework to force the shared layers to learn languageindependent representations (Yi et al. 2018). Differently, our proposed method explores adversarial learning to solve the task imbalance problem in multilingual meta-learning ASR and can learn to adaptively sample the meta-training tasks for effectively training low-resource ASR models.

Preliminaries
Here we briefly introduce the ASR model and its metalearning version which are used in our method.
Multilingual Speech Recognition ASR model. We first introduce the joint CTC-attention based end-to-end ASR architecture (Kim, Hori, and Watanabe 2017;Hori et al. 2017) because of its effectiveness and efficiency. It consists of a Seq2Seq network for frames alignment and symbols recognition, and a connectionist temporal classification (CTC) module (Graves et al. 2006) to encourage the alignments to be monotonic. For the seq2seq model, it contains an encoder, a decoder, and an attention unit. CTC is on the top of the encoder and is jointly trained with the Seq2Seq model. Then ASR network combines these two components and minimizes L(θ) = λ ctc L ctc + (1 − λ ctc ) L seq2seq . Multilingual ASR model. To overcome the challenges brought by different sub-word units, lexicon and word inventories between different languages, we take the union over all the language-specific token sets and train a single model on a mixture dataset which combines all the source language data (Watanabe, Hori, and Hershey 2017;Toshniwal et al. 2018). Given N languages with training sets

Multilingual Meta-learning ASR
We train a multilingual meta-learning ASR (MML-ASR) model on all languages to pursue the few-shot learning ability to handle the low resource recognition problems. Specially, we use f (θ) to denote a multilingual ASR model parameterized by θ and adopt D source = {D k source } K k=1 to denote K kinds of source languages. To apply meta learning, e.g. MAML (Finn, Abbeel, and Levine 2017) and Reptile (Nichol, Achiam, and Schulman 2018), for the k-th kind of language, we sample partial data D k task (a few sentences) from D k source to construct a small recognition task T i k . Then we split D k task into support data D k support and query data D k query . Accordingly, for each language we can sample many tasks denoted by where n k is the task quantity of the k-th kind of language. Let V be the total number of examples in the k-th kind of language and w be the number of examples per task, then n k could be calculated by combination number C w V . For brevity, let T be the all task }. Now MML-ASR can be formulated as: where T i is sampled from T , D support , and D query respectively denote the support and query data in task T i . This model can be understood as that given a common model parameter θ for all language (or tasks), for a sampled task T i , we adapt the parameter θ to this specific task via running one gradient descent on its support data and obtain the task specific model parameter θ Ti = θ − α∇ θ L D support (θ). Then we evaluate the effectiveness of θ Ti on the query data D query of task T i and use this query loss, denoted as Q Ti = L Dquery (θ Ti ), to guide the learning of the model parameter θ. This process actually requires the learnt common model parameter θ to be close to the optimal model of all tasks T i in task set T such that taking only one gradient step on a small-sized dataset D support can achieve satisfactory performance on the query data D query . This mechanism gives the few shot learning ability of model parameter θ.
After training, given a new speech recognition task with a few training data, we adapt the model parameter θ to this task by a few gradient descent steps and obtain a task specific model for test. So this MML-ASR model can well handle the low resource speech recognition problem.
This MML-ASR is inspired by the prior meta-learning ASR framework (Hsu, Chen, and yi Lee 2020). Hsu et al. (Hsu, Chen, and yi Lee 2020) first used a shared backbone to extract common features for all languages and then adopted different network branches to learn language-specific features, while MML-ASR here uses an entire shared model for all languages to simplify operations and make full use of information from different languages.

Adversarial Meta Sampling Motivation
As introduced in Sec. 3, for each meta-training iteration, we need to sample a task T i from the task set T denotes the tasks in the k-th language domain. In real-world scenarios, different languages have a diverse geographic location, phonology, phonetic inventory, language family, and orthography, and their datasets vary greatly in size. So when sampling task T i from all languages task set T to train our MML-ASR model, there are two severe issues. The first one is that the underlying task quantity (n 1 , · · · , n K ) of each language fluctuates over a wide range, which means that the task sets T k (k = 1, · · · , K) are imbalanced in terms of their task quantity (task-quantity imbalance). The second issue is that the tasks sampled from different language task sets T k (k = 1, · · · , K) actually have very different recognition difficulty due to the aforementioned language specificities (Waibel et al. 2000), leading to an imbalance in terms of task difficulty (task-difficulty imbalance). So sampling approaches become especially important in ASR. Note, uniform sampling or task-quantity-balanced sampling, namely sampling rate for each language positively relying on its task quantity in T k , usually ignore the task-difficulty (a) Our AMS framework  imbalance, and cannot achieve satisfactory performance as shown in our experiments.
To resolve above two imbalance issues, we propose a novel and effective adversarial meta sampling approach that adaptively determines the sampling probability for each language task set T k in the meta-training process to balance both task quantity and difficulty in different language domains. Specifically, we observe that the query losses of tasks can well measure both imbalances because that if one language task set T k are not well sampled in terms of its task quantity and task difficulty, then its tasks have relatively large query loss. Intuitively, at each iteration, one can sample several tasks from each language tasks T k and compute the average query loss Q T k for each T k . Then a simple way is to use Q T k / K i=1 Q T k as the sampling probability for each T k . However, this method ignores the long-term query loss information and only greedily assigns a large probability to some language tasks T k according to the current query loss, which could be too locally greedy and leads to performance degradation. Maintaining a query loss buffer which linearly or exponentially averages the historical query losses still cannot achieve satisfactory performance, since the importance of current query loss and historical query losses is not necessarily a simple (exponential) average relation. Moreover, it requires to tune several manual hyper-parameters (e.g., window size) for computing average query losses. All sampling methods mentioned above have been evaluated in Table 4.

Our Sampling Approach
To solve the above issues, we propose adversarial meta sampling method by designing a policy network which injects attention mechanism into LSTM (Hochreiter and Schmidhuber 1997). At each training iteration, it can adaptively predict the most befitting probability to sample from each language tasks T k by using the long-term information in LSTM and the current query losses. Moreover, the policy network can be jointly trained with MML-ASR model in an end-to-end way without manual tuning extra hyper-parameters.
Specifically, as shown in Fig. 2 (a), at each meta-training iteration, the policy network samples M kinds of language task set denoted by {T tj } M j=1 , t j ∈ {1, · · · , K} from the K kinds of language task T and then samples one task T i tj from each T tj to form training task set {T i tj } M j=1 for meta-training of MML-ASR model. So the meta-objective of MML-ASR model in Eqn. (1) can be reformulated as where π denotes the task sampling policy learnt by the policy network f φ parameterized by φ.
After meta-training of MML-ASR model, we can obtain the query losses {Q T i t j } M j=1 of each training task {T i tj } M j=1 . As mentioned in Sec. 4, the query losses of tasks drawn from each language task set T k can well measure the task imbalances in terms of both task quantity and task difficulty. This is because if one language task set T k are not well sampled in terms of its task quantity and task difficulty, then its tasks have relatively large query loss. This actually means that the language task set T k with large query loss Q T k requires more extra training. So at each iteration, our policy network attempts to increase the query loss of MML-ASR model through adversarial learning for sampling the proper language task set for training. Formally, the objective loss of our policy network is defined as: Then we focus on introducing the policy network f φ , which shown in Fig. 2 (b). First, we use K-dim vector Q s−1 T = (Q s−1 T1 , Q s−1 T2 , ..., Q s−1 T K ) to denote the query loss for each language task set T k at the (s − 1)-th meta-training iteration. Second, at the s-th iteration, the policy network will output a K-dim P s T = (P s T1 , P s T2 , ..., P s T K ) in which P s T k denotes the sampling probability for the k-th language task set T k . Third, as aforementioned, we select the top-M sampling probabilities and respectively sample one task from their corresponding language task set T k for meta-training. Then we use the M new query loss {Q s T i t j } M j=1 to update the corresponding query loss in Q s−1 T for obtaining Q s T . Next, at the (s + 1)-th iteration, we feed Q s T and P s T into the policy network and combine these two inputs (Q s T , P s T ) to calculate the feed forward attention, and then get the attention output c s+1 through a fully-connected layer. Then, a LSTM layer takes the hidden state of previous LSTM cell h s as well

Algorithm 1 Adversarial Meta Sampling
Require: α, β, γ: step size hyperparameters 1: Initialize θ, φ 2: Initialize QT k = 0, ∀k ∈ {1, 2, ..., K} 3: while not done do 4: Generate K-dim vector of sampling probabilities PT = (PT 1 , PT 2 , ..., PT K ) using f φ 5: Sample M language task set {Tt j } M j=1 , tj ∈ {1, · · ·, K} with top-M largest sampling probabilities 6: Sample one task T i t j from each Tt j to form {T i t j } M j=1 , tj ∈ {1, · · · , K} for meta-training 7: for all T i t j do 8: Generate support set D t j support and query set D 10: end for 11: 14: end while as attention output c s+1 as input, and generates the LSTM output y s+1 and current hidden state h s+1 . Finally, based on y s+1 , we use fully-connected layer with Softmax function to predict a probability vector P s+1 T = (P s+1 T1 , P s+1 T2 , ..., P s+1 T K ). In this way, same as the s-th iteration, we can select the top-M largest probabilities and sample tasks from their corresponding task sets for meta-training.
As the discrete sampling operations for obtaining M tasks is not differentiable, we apply REINFORCE algorithm (Williams 1992) to solve this issue and optimize the policy network via the following gradient, Through such an online and adversarial manner, the sampling policy is dynamically changed along with the training state of the MML-ASR. In this way, the language task set T k that is not well sampled in terms of its task quantity and task difficulty will be sampled more for more effective learning.
Moreover, our sampling method can be applied to not only MML-ASR methods but also multilingual transfer learning ASR (MTL-ASR) without any architecture modification. For MTL-ASR, we directly use the training loss L T k of a task sampled from each source language domain to construct the loss vector (L T1 , L T2 , ..., L T K ) and feed it into our policy network. Similarly, the policy network outputs the sampling probability for each language domain. Experimental results in Sec. 5 also verify the superiority of our policy network and show significant improvement when applying our sampling method into MTL-ASR.
Finally, our Adversarial Meta Sampling framework is summarised in Algorithm 1.

Experiments
Datasets. Common Voice (Ardila et al. 2020) is an opensource multilingual voice dataset and contains about 40 kinds of languages. For low-resource evaluation, we construct three different datasets: Diversity11, Indo12, and Indo9 which are described in Table 1. To construct the Diversity11 dataset, we randomly select 11 kinds of languages from different districts with varying diversities and quantities, and divide them into 9 source languages and 2 target languages. For Indo12, we randomly select 11 kinds of languages with little difference from the same Indo-European language family and 1 Afro-Asiatic language. To test our method on fewer source languages, we remove 3 languages (Russian, Swedish, and Welsh) from Indo12 to obtain Indo9. In addition, we also conducted experiments on the IARPA BABEL dataset (Gales et al. 2014) with 6 source languages (Bengali, Tagalog, Zulu, Turkish, Lithuanian, Guarani) and 3 target languages (Vietnamese, Swahili, Tamil). Implementation Details. We use the joint attention-CTC ASR model (Kim, Hori, and Watanabe 2017;Hori et al. 2017) as our ASR model. The encoder contains a 6-layered VGG (Simonyan and Zisserman 2015) extractor and 5 BLSTM (Graves, Jaitly, and rahman Mohamed 2013; Graves and Jaitly 2014) layers, each with 320-dimensional units per direction. Location-aware attention (Chorowski et al. 2015) with 300 dimensions is used in our attention layer and the decoder is a single LSTM (Hochreiter and Schmidhuber 1997) layer with 320 dimensions. We set λ ctc to 0.5. During inference, the greedy-search decoding is used to get the best hypothesis. Following ), we use 80-dimensional log Mel-scale filterbank coefficients with pitch features as the input features. Google's SentencePiece toolkit (Google. 2019)  is employed to process audio transcripts. All transcripts in the different multilingual datasets are used to train sub-word models separately based on the byte pair encoding (BPE) compression algorithm (Sennrich, Haddow, and Birch 2016;Kudo 2018). The transcripts are pre-tokenized to sequences of sub-word units (tokens) one-hot vectors using the subword models. The policy network contains a feed forward attention and a one-layer LSTM with the hidden size 100 and the input size 32. We use Adam with an initial learning rate γ = 0.035 and an entropy penalty weight 10 −5 to train the policy network. We set M as 3 after searching the range M ∈ {2, 3, 4, 5, 7, 9} and set w as 48, of which 24 examples are divided into support set and 24 examples into query set.

Results on Low-resource ASR
Results on Diversity11. Table 2 reports the results on Di-versity11 in terms of word error rate (WER). For all target languages, our AMS significantly outperforms all previous methods. First, the performance of monolingual is poor without the help of source languages. Second, meta-learning decrease WER over 6% thanks to its few-shot learning ability by learning better initialization parameters that enjoy fast adaptation ability. Moreover, by learning to sampling tasks for meta-learning, our AMS further improves the results over 7% on this dataset which has large task imbalance.
Results on Indo12 and Indo9. As shown in Table 2, our method consistently achieves the state-of-the-art performance on Indo12 which eliminates task-quantity imbalance and Indo9 which has much fewer source languages for training. This is because our AMS uses adversarial sampling to select better tasks for effective learning and well overcomes the task-difficulty imbalance issue. The results of Kabyle are much worse than that of Spanish and Dutch because Kabyle is an Afro-Asiatic language and all source languages are Indo-European language, which indicates that source languages from the same language family are more helpful for target languages.
Results on IARPA BABEL. In order to further verify the effectiveness of our AMS, we also conducted experiments to compare with previous works on the IARPA BABEL, which is another public multilingual dataset. Table 3 reports the results on BABEL. In addition to comparing with the baselines above, we also selected the results of recent papers for comparison. As can be observed, our AMS achieves the best results for all target languages, which can improve the performance of both MML-ASR and MTL-ASR with the proposed adversarial meta sampling method. It further demonstrates

Ablation Studies
Considering the superior performance of MML-ASR over MTL-ASR, all the following ablation experiments focus on AMS based on MML-ASR. Different meta-learning methods. Table 4 shows that our AMS can improve all meta-learning methods, including MAML Finn et al. 2017), FOMAML Finn et al. 2017), Reptile (Nichol et al. 2018). Among them, AMS-FOMAML achieves similar performance as AMS-MAML but has higher training efficiency. So in the other speech tasks, we focus on AMS-FOMAML. where the language is located to the black point represents the distance from the language to English and the circular area of the language represents the sample times of this language.
Different scale of training data. To test our method when data are only a few, we reduce the training data of target languages to 80%, 50% and 20%. From Table 4 and 2, one can observe that with 80% training data, our method still works better than most baselines with 100% training data, which testifies our method can effectively alleviate the need of heavy annotated training data.
Comparison among different sampling methods. We further compare the performances of different sampling methods. 1) Sampling tasks uniformly (Uniform). 2) Sampling tasks with the probability proportional to the task quantity of each language task set (PPQ) (Dou, Yu, and Anastasopoulos 2019). 3) Sampling tasks with the probability proportional to the query loss of each language task set (PPQL) (Sun et al. 2018a). 4) Sampling tasks with the probability proportional to the average query loss of each language task set with a window (PPAQL). 5) Sampling tasks with the probability proportional to the exponential average query loss of each language task set (PPEAQL). 6) Our AMS without attention layer (AMS w/o attention). From Table 4, one can find that (1) PPQ is slightly better than Uniform by considering the task-quantity imbalance; (2) the methods that use query loss to sample tasks have greatly improved the performance and PPAQL is the best; (3) our AMS significantly surpasses other methods, which indicates that our policy network can effectively exploit the long-term and instant information to sample the most benefiting tasks in the training process.

Generalization analyses
AMS on multilingual transfer learning. Our sampling method can be generalized to MTL-ASR as mentioned in Sec. 4. As shown in  (Sandler et al. 2018) as the feature extractor and  NetVLAD (Arandjelović et al. 2016) as the aggregation network. As shown in Table 5, our AMS outperforms most of the baselines with large improvement. AMS on speech translation. Here we consider translating other languages speech to English. We select 8 source languages of 10 hours and 3 target languages less than 10 hours from CoVoST ), a multilingual speech translation (ST) corpus. For simplicity, we use the same model architecture and data preprocessing procedure as ASR in Sec. 5, which can achieve the same performance as the model used in CoVoST. Table 6 shows the result of case-insensitive tokenized BLEU (Papineni et al. 2002) using sacreBLEU (Post 2018). By comparison, our method outperforms all baselines, including CoVoST, and achieves state-of-the-art performance in all the three target languages. Analysis of difficulty imbalance. We limit the quantity of each source language to 10 hours to analyze the task difficulty in our ST experiments. In Figure 3, we use the semantic similarity between English and other source languages (Senel et al. 2018) as well as the language learning difficulty for English speakers (FSI. 2007) as references to measure the distances, where the farther the language is, the more difficult is to translate it into English. It can be observed a trend that the sample times increase with the distances. For example, Chinese and Dutch are the farthest language with the highest sampling times and the closest language with the fewest sample times, respectively. This indicates that our method can automatically sample tasks according to the task difficulty to alleviate the imbalance from different language difficulties.

Conclusion
In this work, to tackle the task-imbalance problem caused by language tasks difficulties and quantities, we develop a novel Adversarial Meta Sampling framework to adaptively sample language tasks for learning a better model initialization for target low-resource languages. It can well handle the challenging multilingual low-resource ASR in real world. Extensive experimental results validate that our method effectively improves the few-shot learning ability of both meta-learning and transfer learning and also shows its great generalization capacity in other low-resource speech tasks.