{'fraction_non_alphanumeric': 0.03722619861166061, 'fraction_numerical': 0.01614933640254649, 'mean_word_length': 4.576413959085439, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Can Data Diversity Enhance Learning Generalization?

This paper introduces our Diversity Advanced Actor-Critic reinforcement learning (A2C) framework (DAAC) to improve the generalization and accuracy of Natural Language Processing (NLP). We show that the diversification of training samples alleviates overfitting and improves model generalization and accuracy. We quantify diversity on a set of samples using the max dispersion, convex hull volume, and graph entropy based on sentence embeddings in high-dimensional metric space. We also introduce A2C to select such a diversified training subset efficiently. Our experiments achieve up to +23.8 accuracy increase (38.0% relatively) in sentiment analysis, -44.7 perplexity decrease (37.9% relatively) in language modeling, and consistent improvements in named entity recognition over various domains. In particular, our method outperforms both domain adaptation and generalization baselines without using any target domain knowledge.


Introduction
We introduce the Diversity Advanced Actor-Critic reinforcement learning framework (DAAC) to improve the generalization and accuracy of Natural Language Processing (NLP). Training data plays a crucial role in Deep Learning (DL)-based NLP. Investigations (Hendrycks et al., 2020;Dodge et al., 2020;Ramponi and Plank, 2020) show that training data quality is imperative to the machine learning model's performance. Good data are effective and easy to generalize, while bad data bring noise and overfits on out-of-domain test sets.
Decades of work (van der Wees et al., 2017;Aharoni and Goldberg, 2020;Guo et al., 2020;Axelrod et al., 2011) have been devoted to finding the best data, with the most commonly used method we call domain adaptation (Qu et al., 2019; that uses target domain knowledge. Despite large improvements on specific domains, these adapted models lack robustness and are prone to rare events. For example, when models shift towards one target domain like Twitter, they may become erroneous for other domains like medical or travel, and real-world applications on news report data, for example, do not know which queries to receive before they launch, resulting in a critical performance gap between laboratory findings and reality. Therefore, we move away from the independent and identically distributed assumption (i.i.d.) and focus on reducing generalization errors by studying the inter-dependencies among samples.
In this paper, we ask three questions (RQs) and propose our answers. The first question (RQ 1) is: "does higher diversity among training samples lead to better generalized NLP model learning?". Semantic diversity is a desirable aspect in human annotation, where training data sets should be large and diverse enough to learn the many ways the objects differ (Shankar et al., 2017;Wu et al., 2018;Merler et al., 2019;Schumann et al., 2021), like intrinsic facial diversity in face recognition data, since every face is different. We show that diversity ensures that the training data can provide sufficiently discriminative information for the model (Gong et al., 2019), and thus give a more accurate prediction. Intuitively, for a fixed number of samples, the more semantic meaning they cover, the more information they contain, and thus the more effective they are for learning. We will use two examples to elaborate on our rationale.
Let us first take one explanatory example of V ={be, cheerful, happy, stay}, a four single-word dataset that forms two semantically close clusters: A ={be, stay} and B ={happy, cheerful}. Our hypothetical NLP task is to generate sentences from V . From 4 2 word combinations, four sentences are meaningful: "stay happy", "stay cheerful", "be happy", "be cheerful". Our dataset selection task is to find two single-word samples from V that generate the most sentences. If we select both single-word samples from A, which are semanti- cally close, we will generate no meaningful sentence (analogously that from B). However, we can generate one of the four meaningful sentences if we select one word from A and the other word from B, e.g., "stay" and "happy", which are semantically distant. In fact, since these four sentences can be treated as paraphrasing, generating any of them will roughly cover the semantic meaning for all. In this example, the diversified subset {stay, happy} is more effective for learning generalization.
Our second example is conducted on a real dataset. Figure 1 shows our experimental results on the Penn Treebank dataset. We equipartition 42K training sentences into five subsets and compute their diversity using the sum of pairwise Cosine distances of sentence embeddings (i.e., our first method, see next paragraph and Section 2.2.1). We sort the subsets according to their diversity scores. The first subset has the lowest diversity (3.18e 7 ), the second subset has the second-lowest diversity (3.26e 7 ), etc. We then train five language models on each subset 1 , and computes the perplexity (PPL) and out-of-vocabulary (OOV) on ten randomly sampled subsets of the test set. PPL and OOV are typical indicators of how training data is generalized (Rastogi et al., 2020;Müller et al., 2019). Our results show that the diversity of the subset strongly negatively correlates to both PPL and OOV rate, where the subset with the highest diversity is more likely to generalize on a random test and has fewer unseen words.
With the preliminary evidence showing that a training set with higher diversity leads to more gen-1 The language model is trained on Transformer with 2 heads, 2 hidden layers, 200 embedding size, and 200 hidden units on each subset for 10 epochs. Each test set subset contains 1000 sentences. eralized learning, we propose our second question (RQ 2): how to measure the diversity of a sample set? We introduce three methods to quantify training dataset diversity. The first method is based on the notion of dispersion, also known as Max-Sum Diversification (Cevallos et al., 2016). By viewing each training sample as a data point in the sentence embedding space, we maximize the dispersion of the training set, which is the sum of pairwise Cosine distances of sentence embeddings, as in Figure 2 (2). The larger the sum, the more informative the subset is.
The pairwise geometric distance method has a worst-case complexity of O(n 2 ), and to accelerate the computation, we introduce our second method (Figure 2 (3)) that measures diversity using the volume of a convex hull around the samples, namely the volume that encloses all vertices, taking each sentence embedding as a vertex. Its quickhull (Barber et al., 1996) implementation has an averagecase complexity of O(n log n) and a worst-case complexity of O(n 2 ).
The above two methods compute diversity based on distances in the metric space. We can obtain global information content by measuring the uncertainty of the semantic distances between samples based on frequencies. Our third method combines our view of geometric distance and entropy in NLP. As depicted in Figure 2 (4), we apply Graph entropy, H(G, P ), an information-theoretic function on graph G with a probability distribution P on its vertex set (Dehmer and Mowshowitz, 2011;Rezaei, 2013). Here, each sentence in the subset is a node in the graph, and the Cosine distance of the twosentence embeddings is the weighted edge between the two nodes. Empirically, all these three diversity measures consistently outperform the baselines. The Max Dispersion (MD) and Graph Entropy (GE) are more robust in various domains with the best performance overall because they account for every pair of distances in the training set.
After we have defined our diversity measure, we ask our third question (RQ 3): "how can we optimize the subset to maximize the information content?" Subset selection is NP-hard in general (Qian et al., 2016;Davis et al., 1997). We conjecture that this holds true for every objective function, including the notion of diversity introduced in this paper, and is the reason why we use actor-critic reinforcement learning (Konda and Tsitsiklis, 2000). We equipartition the training data into mini-batches  Figure 2: Diversity Advanced Actor-Critic Framework. and simultaneously learn a policy network (as an actor) to select data and a value network (as a critic) to estimate future returns using the diversity measure as an evaluation reward. Our actor-critic data selection method has the advantages of low variance updates as well as credit assignment and significantly outperforms domain adaptation  on various domains without requiring any target domain knowledge. The architecture is shown in Figure 2 and Section 2.
In summary, our work mainly contributes to the following: (1) Modeling sample dependency for effective data sampling using diversity; (2) Measure dataset diversity using max dispersion, convex hull volume, and graph entropy; (3) Introduction of the Advantage Actor-Critic reinforcement learning framework to select informative samples and enhance NLP accuracy and generalization.
The rest of the paper is organized as follows. In Section 2.1, we detail Diversity Advanced Actor-Critic reinforcement learning (DAAC). Then in Section 2.2, we introduce different diversity measures. Afterward, in Section 3, we empirically verify the generalization and accuracy improvement using DAAC. We discuss related work in Section 4 and conclude the paper in the last section.

Diversity Advanced Actor-Critic
We now present the details of our Diversity Advanced Actor-Critic (DAAC) algorithms. The details of data selection and fine-tuning process are depicted in Figure 2. Our task model F (any Deep Learning-based NLP model, such as Transformer, etc.,) is pre-trained on the full training data set , where x i is a sentence, n is training set size. Then, as in , we shuffle and randomly partition X into T disjoint data batches so that n|T is the integer division of n by T , and T ≤ t. If mod (n, T ) ̸ = 0, then the last batch has a variable size of mod (n, T ) and collects the remaining sentences. For each batch, we select and denotê as the selected data with size o. After obtainingB t , we useB t to fine-tune F. F and its encoder g is updated onB t for T times in an epoch, and each update is based on the previous checkpoint.
Our goal is to improve our task model F on any test domain whose distribution is different from the source training set X , by learning an effective subset of X to fine-tune F. Figure 2-(1) shows our new model. Our framework is based on Actor-Critic reinforcement learning. Our Actor-Critic method has the following properties, which are very desirable to achieve our goals: variance deduction, efficient sampling, and credit assignment (Konda and Tsitsiklis, 2000;Grondman et al., 2012). Our framework consists of policy and value networks jointly and dynamically learned together with the task model using the advantage error computed from the reward function. In the following context, we will first introduce our Actor-Critic algorithm (Section 2.1) and then our three different reward functions based on diversity (Section 2.2).

Markov Decision Process
We cast the data selection as a reinforcement learning problem with a Markov Decision Process (MDP). Our data selection policy π, a mapping from states to actions, serves as an agent to interact with an environment that constitutes an NLP model, we call task model F over T time steps. At each time step t, a state s t , an action a t and a reward r t are collected.
First, the encoder g inside the NLP model (e.g. an embedding layer in LSTM, or an encoder in transformer) transforms a batch of data B t into its embedding s t (s t = g(B t ) in Fig 2, e.g. vector representations of the encoder output of a sentence).
Secondly, the policy π outputs a probability distribution for the batch of state s t , so that each sentence is associated with a probability representing how likely it is going to be selected. The selected subset, denoted asB t , is then obtained by Bernoulli sampling each sentence in the state s t . For example, for the k-th sentence in the batch with probability p k , we generate a random number between 0 and 1. If p k is larger than the random number, then the kth sentence is selected; otherwise, it is not selected. The result of Bernoulli sampling is represented as a vector a t , where each value in it is either 0 or 1 representing each sentence in the batch not being or being selected.
Thirdly, task model F as well as encoder g are finetuned by the selected subsetB t . In the meantime, a scalar reward r t = R(g(B t )) is calculated by designed diversity reward functions R which we give definitions in Section 2.2.
Finally, the policy agent π updates its weights using the collected s t , a t and r t , where state s t is the encoded representation of a batch of data B t , action a t is a vector with each value of either 0 or 1 representing each sentence in the batch not being or being selected, and the scalar reward r t measures the diversity of selected batchB t . This optimization process is expanded in next section 2.1.2.

Training algorithm
We employ the Advantage Actor Critic algorithm that uses A(s t , a t ), the advantage of action a t in state s t to scale the policy gradient. Specifically, the advantage of action a t in state s t is defined in Mnih et al. (2016) as where γ ∈ (0, 1] is the discounting factor, and we set its value as 0.99. V is the value function (critic) implemented as a value network in Figure 2-(1).
The data selection policy π (actor) is implemented as a policy network, whose training objective is The parameters of the policy network θ are updated by: (3) where α is the learning rate, and Equation 2 estimates A(s t , a t ; θ, θ v ).
The objective of value network is: The parameters of value function θ v is updated by: The policy network π(a t |s t ; θ) is a two-layer network that has two nodes in the first layer and one hidden output produced by the tanh activation function and one softmax output. The value network is a two-layer network that has eight nodes Algorithm 1 DAAC Training Algorithm Input: Epoch L, learning rate α, discount factor γ, training set X , pre-trained task model F (including encoder g), reward function R (discussed in section 2.2) Output: selected data, fine-tuned F, policy π θ , data value estimator V θv 1: Initialize data selection policy π θ and value estimator V θv 2: for episode l = 1 to L do 3: Shuffle (uniformly at random) all training samples; 4: Initialize an empty list: episode history Υ 6: for all B t ∈ X (uniform transition probability) do 7: Obtain batch action a t by sampling based on π θ (s t ); Store (s t , a t , r t ) to episode history Υ; Clear episode history Υ; 20: end for 21: return F,π θ and V θv in the first layer and one hidden output produced by the ReLU activation function, and one linear output for the value network V(s t ; θ v ) following Mnih et al. (2016). The training algorithm is shown in Algorithm 1.

Diversity Measures
The reward function in Section 2.1 is the diversity of the selected batch subset. We do not need any target domain knowledge, and the diversity is measured with the following three methods.

Max Dispersion
The dispersion of a set is the sum of all pair-wise distances within the set (Cevallos et al., 2016). Intuitively, maximizing the dispersion (denote as MD) of a set can enlarge the semantic coverage of the set, and thus diversify the content of the set. Formally, x i and x j are any two training sentences in a training set G, and their sentence embeddings are g(x i ) and g(x j ), respectively, we define the dispersion as where The diversity of a set of training samples (sentences) is computed as the sum of each pairwise sample distance, which can be measured by any distance metrics, i.e., Cosine distance. We take the NLP task model encoder output as the sentence embedding to compute pairwise sample distance. The sentence embedding preserves pairwise sentence distance from the semantic space to the high-dimensional vector space. More details of the algorithm are shown in Algorithm 2, with a worst time complexity of O(n 2 ), where n is the number of samples in the set.

Convex Hull Volume
Instead of looking at each pair of sentences, we can simplify the computing process by looking at the boundary of a set of training samples in the high dimensional space and compute the volume of such space using Convex Hull Volume (denote as CV). Specifically, we treat each sentence as a point in the embedding space and approximate the volume of a convex body. In a Euclidean plane, given a finite set of points Q, it is sometimes interesting to determine its convex hull, namely the minimum convex polygon so that any point of Q is either inside this polygon or at its border. There are a number of algorithms to compute the convex hull. Since our embedding is in high dimension, i.e., 256, we consider N-dimensional quickhull, which was introduced in Barber et al. (1996) . Just like the quicksort algorithm, it has the expected time complexity of O(n log n) by divide and conquer approach, but may degenerate to O(n 2 ) in the worst case. Details shown in appendix Algorithm 3.

Graph Entropy
Applying entropy on the distances, we introduce Graph Entropy (GE), following the so-called local measure of distance-based substructure entropy, details in Dehmer and Mowshowitz (2011). As shown in Figure 2, graph entropy measures the uncertainty of the pairwise distance of sentence (or word) embeddings in the geometric space. To compute graph entropy of a set of training samples, we cast the sentence embeddings as vertices and distances between sentence embeddings as edges to form a clique. The distance entropy I of each sample x is defined as M is the number of sentences in the set, d(x, x ′ ) is the Cosine distance between the encoder output vector representation of x and is the relative distance of edge d(x, x ′ ) (instead of words). The graph entropy of entire set is then defined as the sum of all distance entropy I: For a set with n data samples, there are n 2 edges in the resulting graph. The time complexity of graph entropy is thus O(n 2 ) by looping through each edge in the graph. Details are shown in Algorithm 5.

Experiments
We describe our experimental details and demonstrate that our DAAC improves baselines in three NLP applications, including two recognition tasks: sentiment analysis, named entity recognition (NER), and one generation task of language modeling, without the knowledge of the target domain. We give hyperparameters of DAAC in appendix.
Finetuning vs. Data Selection We introduce our four models based on DAAC. The first three models finetune a pretrained NLP model using max dispersion MD, convex hull volume CV and graph entropy GE as reward function. As a comparison, we add the fourth model trained on selected data from scratch that does not rely on a pretrained model, denoted as data selection DS. Specifically, we train A2C using MD as reward function for 150 epochs first, and use the trained A2C to select data samples from all source data, then use only the selected data to train the NLP model from scratch. We select 3833 (63.8%) and 7630 (54.3%) samples from the source domain of sentiment analysis and NER respectively, and 21017 (49.9%) samples

Sentiment Analysis
Settings We use the Amazon product review dataset (Blitzer et al., 2007) for the sentiment analysis task. The sentiment analysis baseline is a CNN classifier (Kim, 2014). We pretrain the CNN for two epochs as  for a fair comparison.
Results   (Yoon et al., 2019;Fan et al., 2017). We also conduct a domain adaptation experiment which aims to maximize the test accuracy on one specific target domain and outperform all baselines with a significant test p = 0.00038, as shown in table 8 in appendix.

Named Entity Recognition
Settings We use the CoNLL2003 English NER dataset (Sang and Meulder, 2003) as source training set and the five domains from CrossNER dataset  as test sets, which has specialized entity categories for each domain. We finetune the pretrained BERT model (Devlin et al., 2018) on source training set by adding a linear layer on top of the hidden-states output and then evaluate the F1-scores on five test domains.

Results
The result is reported in Table 2. Graph Entropy achieves best performance among all the three diversity measures. In particular, Data Selection performs better than training with all data directly (All) and random selection (Rand), which means important samples are selected and noisy samples are filtered out.

Language Modeling
Settings Our baseline is a Transformer language model (Vaswani et al., 2017) with default hyperparameters. We experiment with two moderate size datasets WikiText-2 (Merity et al., 2016) and Penn Treebank. As for evaluation, we report perplexity scores on two translation datasets from different domains, IWSLT' 17 (TED talk english) and WMT Biomedical' 21 (english). The baseline models are trained using the fairseq toolkit (Ott et al., 2019) and stop training until the validation perplexity score does not improve for 5 epochs.

Results
The evaluation results are shown in Ta  21 has an improvement of 44.65 (37.9% relative improvement) while trained by our selected data. The comparison with randomly selected result (Rand) further proves the effectiveness of data selected by DAAC.

Ablation Analysis
MD vs. GE In our three experiments we observe Max Dispersion (MD) performs best in sentiment analysis and language modeling, and Graph Entropy measure (GE) achieves the best performance in named entity recognition. We infer the smaller improvement of convex hull volume might be due to error propagation of the quickhull algorithm.
Regarding the reason why GE performs best in named entity recognition, we first analyze the difference of operations on sentence embeddings for GE and MD. Since GE is calculated by the sum of distance entropy of each sentence (sample), there are more operations (eg. log and normalization) depending on the sentence embedding input than MD. Thus, performance of GE is more relying on the precision of sentence embeddings compared to that of MD. Then looking into the key difference between the task of NER and other tasks, the source of sentence embeddings are apparently different. In NER, sentence embeddings are trained by BERT, while in sentiment analysis they are trained by CNN. As a result, we conjecture the more precise sentence embeddings trained by BERT lead to the better performance of GE on NER task. To verify this assumption, we quantitatively evaluate the quality of sentence embeddings generated by finetuned CNN and finetuned BERT on semantic textual similarity (STS) benchmarks STS-B-Dev (Cer et al., 2017), STS14 and STS15 (Agirre et al., 2014(Agirre et al., , 2015. We report the Spearman's rank correlation between the cosine similarity of sentence embeddings using the SentEval toolkit (Conneau and Kiela, 2018). The results in table 4 show preliminary evidence that our conjecture holds. In short, MD can be consid-   Table 5: Out-of-vocabulary of diversified and nondiversified set policy network converge. In practice, training with all source data directly (All) in sentiment analysis takes 131 seconds while finetuning with MD takes 217 seconds, and selecting data out then training (DS) takes 4774 seconds on one Tesla V100 GPU.
Visualization Figure 4 visualizes data samples selected by our DAAC (diversified, red) and those randomly selected (nondiversified, blue), as well as the test data samples (unseen domains) in the embedding space. We observe that our DAAC selected samples are more widely spread, thus covering more semantic meaning for most test sets. Figure 4 (right) shows that DAAC selected samples has a larger convex hull volume than random selected samples after removing outliers. Furthermore, table 5 shows that our DAAC has a smaller size of out-of-vocabulary than the baseline on the test domains. Data samples are selected from the magazines domain of Amazon product reviews (Blitzer et al., 2007). We use max dispersion (MD) diversity measure, sentence-transformer toolkit (Reimers and Gurevych, 2019), and plot the embedding into two-dimension t-SNE.

Related work
Existing domain generalization methods related to data focus on data augmentation (Zhou et al., 2020;Qiao et al., 2020) and data generation . These methods increase the quantity of training set while our method utilizes existing data maximally by modeling the intrinsic sample dependency.
There have been several influential works (Moore and Lewis, 2010; Axelrod et al., 2011;Ruder and Plank, 2017a) on data selection that significantly contributed to today's NLP state-of-the-arts. Fan et al. (2017) proposes a data filter based on deep reinforcement learning on image and sentiment classification task. Feng et al. (2018) implemented an instance selector using reinforcement learning to filter noisy data to improve the accuracy on natural language inference (Qu et al., 2019), sentiment analysis, part-of-speech tagging and dependency parsing . These work select the training data close to a given target domain for domain adaptation. In contrast, we aim to enhance the model generalization and increase the accuracy on any arbitrary domain.
As for diversity measures, Ruder and Plank (2017b) examines the effects of Shannon entropy in the transfer learning setting, but they do not consider content semantic meaning. Shi et al. (2021) uses determinantal point processes to select diverse data to reduce the labor of annotating training examples for dependency parsing, but the proposed diversity measure cannot be generalizaed to other NLP tasks. Other works incorporate the notion of diversity into topics like language representation (Chubarian et al., 2021), and ensemble language modeling (Duan et al., 2021).

Conclusion
We introduce Actor-Critic reinforcement learning rewarded with diversity measures to select effective training data that significantly enhances the accuracy of sentiment analysis, named entity recognition, and language modeling tasks across various domains. Without any target domain knowledge, our method outperforms the CNN, Transformer and BERT baselines. Our experiments show that modeling sample dependency by increasing training set diversity enhances the learning generalization and prediction accuracy. action on batch B t at time step t a k action on single sample x k at time step t |B t | batch size π policy r t reward at time step t R t total future reward from t to T Q π (s t , a t ) action value V π (s t ) expected future return following π since time step t b(s t ) baseline function θ parameters of policy network ∇ θ ∂X ∂θ

A Appendix
objective function of policy L epoch number α learning rate γ discount factor F ′ pretrained task model (including encoder g) E episode record, including s t , a t , r t G a set of samples d cosine distance of embeddings M number of sentences in a set G J sentence length h(·; n) n-gram entropy

7:
end for 8: end for 9: Sort d in descending order 10: Add the two sentence sp and sq that has the max distance to convex hull set h. Line formed by sp and sq segment the space into left half S1 and right half S2. 11: Call subroutine FindHull(S1, sp, sq ) 12: Call

Algorithm 4 Subroutine: FindHull
Input: A batch of training samples S; point p and q.
1: From the given set of points in S, find farthest point f 2: Add point f to convex hull set h. Three points p, q, and f partition the remaining points of S into 3 subsets: S0, S1, and S2, where S0 are points inside triangle pqf , S1 are points on the right side of the line from p to f , and S2 are points on the right side of the line from f to q.