{'fraction_non_alphanumeric': 0.039431337269709335, 'fraction_numerical': 0.014343275639238743, 'mean_word_length': 4.55255382017729, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling

Neural text generation models are likely to suffer from the low-diversity problem. Various decoding strategies and training-based methods have been proposed to promote diversity only by exploiting contextual features, but rarely do they consider incorporating syntactic structure clues. In this work, we propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. In detail, we introduce POS Guided Softmax to explicitly model two posterior probabilities: (i) next-POS, and (ii) next-token from the vocabulary of the target POS. A POS Guided Sampling strategy is further proposed to address the low-diversity problem by enriching the diversity of POS. Extensive experiments and human evaluations show that, compared with existing state-of-the-art methods, our POS Guided Softmax and Sampling (POSG) can generate more diverse text while maintaining comparable quality.


Introduction
Maximum likelihood estimation (MLE) is a standard approach to training a neural text generation model, e.g. Transformer (Vaswani et al., 2017), to generate human-like text. However, existing generation systems often suffer from the low-diversity problem (Holtzman et al., 2020;Welleck et al., 2020), which leads to dull and repetitive generations. This problem unavoidably affects the overall generation quality.
We conclude that the low-diversity problem is mainly manifested in two aspects: form and content (Fu et al., 2021;Holtzman et al., 2020;Tevet and Berant, 2021). As shown Table 1, the low form diversity can be reflected in repeating some words, using similar lexicon and syntax, and more. The low content diversity can be expressed as a single and dull content with nothing different. 1 Our code is available at https: //github.com/FadedCosine/ POS-Guided-Neural-Text-Generation Several feasible fixes have been proposed, such as post-hoc sampling strategies including temperature (Caccia et al., 2020), top-k (Fan et al., 2018, and nucleus sampling (Holtzman et al., 2020). Recently, some works suggest that it is the maximizing likelihood itself that should account for the lowdiversity problem (Holtzman et al., 2020;Welleck et al., 2020). Holtzman et al. (2020) think that MLE can not adequately capture the rich diversity and expression in human language. Choi et al. (2020) argue that the imbalanced token distribution inherent in natural language even worsens the low-diversity problem. Based on these analysis, many trainingbased methods have been proposed. Welleck et al. (2020) propose the unlikelihood training to penalize repetition with auxiliary losses. Jiang et al. (2019) propose to utilize dynamically scaling losses conditioned on the token frequency in the training phase. Choi et al. (2020) factorize the probability distribution and design an elaborate token cluster algorithm for a balanced training.
Though those encouraging progress has been made, we argue that current training-based methods only take plain contextual features to promote diversity, rarely considering incorporating syntactic structure clues. For example, when humans are writing articles, it is natural to predetermine the part-of-speech (POS) before giving the next token. Existing studies have verified that incorporating POS can improve the translation quality in neural machine translation (NMT) (Sennrich and Haddow, 2016a;Yang et al., 2021). Intuitively, since the vocabularies of different POS vary a lot, the diversity of POS will certainly lead to the diversity of text. Unfortunately, we observe that existing methods with no consideration of the inner POS structure fail to learn the diversity of POS in human language (shown in Table 3).
All these factors motivate us to address the lowdiversity problem with the guidance of POS. Thus, in this work, we first present the POS Guided Soft- Table 1: Examples of low-diversity generated text, given context from the Wikitext-103 dataset (Merity et al., 2017). Text 1 has a poor form diversity due to many useless repeating words (highlighted in blue). Text 2 keeps talking about only one single content, with similar lexicon and syntax (highlighted in orange), indicating low diversity in both terms of form and content. Though Text 3 has various syntactical and lexical forms with no repetition, all the content of it is about the "attacks", which means low content diversity. Text 1 is sampled from MLE, Text 2 from max ( Figure 1), building upon a hybrid decoder that predicts two posterior probabilities: (i) next-POS, and (ii) next-token from the vocabulary of the target POS. Our work shows that, following the POS clue, our model can gain a deeper insight into text's syntactic structure. Thereafter, we propose a POS Guided Sampling to improve the diversity of generated text lexically and syntactically while maintaining comparable quality.
To sum up, the contributions of our work are three-fold. (i) We introduce a novel POS Guided Softmax, incorporating POS tags as the observed discrete decisions to improve text generation. (ii) Based on POS Guided Softmax, POS Guided Sampling is proposed to promote text diversity effectively without degrading quality. (iii) We conduct extensive experiments on language modeling and paraphrase generation. Experimental results and human evaluation show that our model can easily adapt to different downstream tasks and generate text with high diversity as well as quality.

Diversity-promoting Methods
Decoding-based Methods. Although greedy search and beam search are well known decoding strategies for neural text generation, Holtzman et al. (2020) have shown that these methods always generate generic, repetitive, and awkward words. Kulikov and Cho (2019) and Vijayakumar et al. (2018) have proposed several variants of beam search as alternatives. Recently, stochastic decoding methods have been widely used, and some studies propose to sample from a truncated and renormalized Softmax distribution. Top-k sampling (Fan et al., 2018) only samples from the top-k most probable tokens. Nucleus sampling (Holtzman et al., 2020) only samples from the smallest set whose cumulative probability is at least α. However, those decoding-based methods are lack of controllability. Combined with above methods, our proposed method can further promote diversity using POS as a more controllable clue. Training-based Methods. As a standard approach to training a neural text generation model, MLE has been proved to be defective. Choi et al. (2020) have shown that MLE may mislead the model because of the imbalanced token distribution. Thus, they design a greedy approach MefMax and factorize Softmax to ensure a balanced training according to the word frequency. FACE (Jiang et al., 2019) utilizes the target word frequency to modify the cross-entropy loss with a frequency-based weight factor. Welleck et al. (2020) introduce an unlikelihood loss to implicitly reduce the frequent tokens and potential repeats. Other approaches, such as negative training (He and Glass, 2020), reinforcement learning (Shirai et al., 2020), and imitation learning (Zhou and Lampouras, 2020), have recently been applied to promote the diversity during the training phase. All above training-based methods only learn from plain contextual features, while ignoring other linguistic features. Our focus is on leveraging POS features to guide both phases of training and decoding.

POS in Text Generation
Previous works, which leverage POS for text generation, can be summarized as follows: POS in Encoding. A branch of previous works (He et al., 2019;Sennrich and Haddow, 2016b;Wray et al., 2019) (Su et al., 2018;Bugliarello andElliott, 2021) or NAG plannings (Yang et al., 2021) in order to improve the generic quality of generated texts, while our proposed methods regard POS tags as observed sequential variables and directly model the POS distribution during both phases of training and decoding with the goal of improving text diversity.
To our best knowledge, we are the first to introduce an explicit POS-guided generation method as a generic way to promote text diversity while maintaining quality.

Language Modeling
The goal of language models is to assign a probability to text (i.e. word sequence) where each x t in the sequence is a token from a vocabulary V, i.e., x t ∈ V, and T ∈ N. We train the language models to learn a distribution p θ (x) with the goal to fit the ground-truth distribution p ⋆ (x) for all x. Specifically, when the language model is a neural network, θ is regarded as the model parameters of the neural network, and we can factorize p θ (x) as p θ (x) = Π T t=1 p θ (x t | x <t ). The conven-tional approach for learning the language model parameters θ is to maximize the log-likelihood by minimizing: where h t−1 is a hidden state of the context x <t , and w xt is the output embedding vector for x t ∈ V.

Methodology
In this section, we describe an overview of our proposed method, POS Guided Softmax and Sampling (POSG). POSG is designed to exploit syntactic structure, i.e., POS tags for text generation in both the training and decoding phases. Specifically, giving text sequence x = [x 1 , . . . , x T ], we first use off-the-shelf POS tagger (Manning et al., 2014) to annotate corresponding POS sequence ρ = [ρ 1 , . . . , ρ T ], where each ρ t is a POS tag from the POS vocabulary P, i.e., ρ t ∈ P, and T ∈ N. We define all the tokens whose POS is ρ as a vocabulary V ρ , where V ρ ⊂ V. Figure 1 illustrates the core idea of our POS Guided Softmax. Given a context, there exist various choices for the next POS, which can be modeled as the next POS distribution. For the context "no one knows", the next possible POS includes WHpronoun (WP), preposition (IN), etc. For example, if WP is predicted as the next POS, the model will decode the next token from the WP vocabulary (V WP ) with the token distribution of WP. Consequently, the complete sequence can be "no one knows what will happen". For another case, if IN is predicted as the next POS, the next token will be decoded from V IN with the corresponding token distribution. Then, the sequence may end up saying "no one knows until it finally happens". This example also shows that the different choices of POS at each time step can result in vastly different generated text, thus promoting text diversity.

POS Guided Softmax
Following the core idea, we assume that the decoding process can be divided into two stages: for each time t, a POS tag ρ t is predicted first, and then the model decodes next-token x t from V ρt . Therefore, the joint conditional probability of x t and its no one knows  Figure 1: Illustration of POS Guided Softmax. The decoding process is decomposed into two stages: first predicts the next-POS distribution, and then decodes the next-token distribution from the vocabulary of the previously predicted POS. Since there exist some tokens with more than one POS, the final next-token distribution is the sum of all the POS's token distributions.
corresponding POS tag ρ t is formulated as: where p θ 1 (ρ t | x <t ) is the next-POS probability and p θ 2 (x t | ρ t , x <t ) is the next-token probability conditioned on ρ t . These probabilities are defined empirically by applying a linear output embedding on h t−1 and then a Softmax function respectively: where o ρt and w xt are the output embeddings for ρ t ∈ P and x t ∈ V ρt , respectively. In this way, we regard POS tags as observed sequential variables, which also contributes to the model interpretability and controllability. Then, the final next-token distribution can be formulated as: . Note that some tokens may have more than one POS, and p θ (x t , ρ t | x <t ) = 0 for x t / ∈ V ρt . Since the number of POS in a specific language family is fixed, there is no problem of insufficient exploration in variables' space.
As mentioned before, we think of POS tags as observed sequential variables and extend the training text set with annotated POS sequences, so we define the POS guided training objective as follows:

POS Guided Sampling
We propose POS Guided Sampling based on POS Guided Softmax. Consistent with POS Guided Softmax, the key idea is to divide the whole sampling process into two stages: POS sampling and token sampling. In POS sampling, we first sample a POS, and then in token sampling, we use the sampled POS to control the sampling of tokens. Note that arbitrary sampling strategies can be adopted to both the POS sampling and token sampling. Here, we take top-k sampling for POS sampling, and nucleus sampling for token sampling as an example, and then we can formulate our POS Guided Sampling as follows: where P ′ ⊂ P is a POS set containing top-k most probable POS tags, and V ′ ρt ⊂ V ρt is the smallest token set such that x∈V ′ ρ t p θ 2 (x | ρ t , x <t ) ≥ α (token) . k (POS) and α (token) (0 < α (token) ≤ 1) are the hyper-parameters for the sampling of POS and token, respectively. For other sampling strategies used in POS sampling and token sampling, POS Guided Sampling can be similarly defined.

Experiments
We systematically evaluate our proposed methods on language modeling task (Section 5.2) and paraphrase generation task (Section 5.3). Note that decoding-based methods, including top-k and nucleus sampling, can be directly compared to POSG, when they are applied to MLE. The details will be described in the sections of Generation Details.

Language Modeling
Dataset We performed experiments on the Wikitext-103 2 dataset (Merity et al., 2017) for language modeling. In order to train our POS Guided Softmax, we need the corresponding POS tags. We 2 https://s3.amazonaws.com/research. metamind.io/wikitext/wikitext-103-v1.zip use the Stanford CoreNLP's POS tagger (Manning et al., 2014) to annotate words in Wikitext-103 with XPOS 3 tags (Hornby et al., 2017). In our implementation, there are 45 different POS tags in total. Generation Details We conduct the text completion task to evaluate models on the test set. Specifically, for each sample, we truncate 50 tokens as the prefix, and then guide model to decode following 100 tokens as the continuation from the given prefix. Finally, there are 1536 prefixes in the test set. We use stochastic decoding to generate text. Note that all the baselines have only one sampling stage, i.e., token sampling, while our POSG has an additional POS sampling. To reach a good tradeoff between quality and diversity, we adopt nucleus sampling with α (token) = 0.5 for token sampling (for all models including our POSG and baselines). For our POSG, we adopt top-k sampling in POS sampling, since the size of the POS vocabulary P is much smaller than the total token vocabulary. We then conduct a grid search to find the k (POS) whose generated continuations have the smallest reverse language model score (Semeniuta et al., 2018) on the validation set. k (POS) is finally set to 20. Some generated cases are shown in Appendix D.
Metrics Following Choi et al. (2020), we evaluate the generated text with two sets of metrics: (i) Diversity: We use Self-BLEU (Zhu et al., 2018) which is calculated by computing BLEU (Papineni et al., 2002) of each generated text with all other generations as references. We also compute the generated continuations' unique tokens (Uniq), distinct n-gram (Distinct-n). We also use repetition (Rep) (Holtzman et al., 2020), the percentage of continuations ending with a repetition loop, to evaluate text diversity. (ii) Quality: We measure the perplexity (PPL) (Mnih and Teh, 2012), KL-Divergence (KLD) (Kullback, 1997) on unigram distributions, and MS-Jaccard (Alihosseini et al., 2019) on n-gram. All the metrics are calculated between the generations as hypotheses and the ground truths as references. Automatic evaluation Table 2 shows the automatic evaluation results comparing different models on the language modeling task. In terms of Self-BLEU4, Rep, and Distinct-n, our POSG performs much better than all the baselines, indicating that our proposed model can generate diverse text effectively. The FACE also performs well, and it  Table 2: Automatic evaluation results for different models on the language modeling task. Numbers n ∈ {1, 2, 3} in the column heads under Distinct and MS-Jaccard refer to n-gram. (Bold: the best; Underline: the second best).

Models
Distinct  Table 3: Results of distinct n-gram and n-POS with corresponding Pearson product-moment correlation coefficient (PPMCC). n-P and n-G where n ∈ {1, 2, 3} are abbreviated notations for n-POS and n-gram.
achieves the best in Uniq. However, by checking the outputs (Table 15 in Appendix D), we find that FACE produces more incoherent text that is hard to understand. Since training-based methods including ours make a trade-off between the text diversity and the likelihood of ground truth, MLE gets the lowest PPL. However, the optimal or second best results of quality metrics confirm that POSG can still maintain comparable generation quality.
We further conduct a correlation test to verify that the text diversity is closely correlated with the POS diversity. We first randomly sample 500 generated continuations from each model, and annotate them with the POS tagger. We define a n-POS to be contiguous n POS tags from the annotated POS tag sequence. Then, we can describe the degree of POS diversity by calculating the proportion of the distinct n-POS. Table 3 presents results of distinct n-gram and n-POS with corresponding Pearson product-moment correlation coefficient. In terms of distinct n-POS, POSG also surpasses all the baselines. This demonstrates that our proposed model can substantially promote the POS diversity. Moreover, the Pearson correlations between dis- tinct n-POS and n-gram are extremely high, which indicates that the high POS diversity indeed leads to the high text diversity.
Human evaluation For the language modeling task, following Tevet and Berant (2021) we randomly sample 100 generated continuations from each model. Each of them is scored between 1 to 5 (5 is the best), by five workers to evaluate the overall Diversity (Div.) and Quality (Qua.). The results of the human evaluation on language modeling are shown in Table 4. It can be seen that our POSG significantly outperforms all other baselines in diversity, and performs relatively well in quality.

Paraphrase Generation
Dataset We use the the ParaNMT-50M 4 dataset (Wieting and Gimpel, 2018) for paraphrase generation. ParaNMT-50M consists of over 50 million paraphrases, generated by back-translation. For better training, we first remove the sentences that are less than 10 tokens. Moreover, ParaNMT-50M dataset has provided translation scores to measure the quality of back-translation, that a low translation score means semantically inconsistent, while a high translation score usually accompanies low  diversity. Therefore, we only keep the paraphrase pairs whose translation scores are between 0.7 to 0.8. Finally, we get a filtered dataset containing 1.6 million paraphrase pairs with both high quality and diversity. We also use Stanford CoreNLP to tokenize the text and get corresponding POS tags.  conduct a grid search, and k (POS) is finally set to 5. Some generated cases are shown in Appendix D.
Metrics We also evaluate the generated paraphrases with two sets of metrics, (i) Diversity: To assess how different the generated paraphrases are compared to the original sentences, we calculate BLEU and Word Error Rate (WER) (Goyal and Durrett, 2020) between generated paraphrases and input sentences. We denote them as Self-BLEU (see Appendix A.3 for the difference with the Self-BLEU in language modeling) and Self-WER, respectively. We also compute the generated paraphrases' distinct n-gram (Distinct-n) to evaluate text diversity. (ii) Quality: we calculate BLEU score on n-gram to evaluate the closeness of the generated paraphrases to references. Besides, we use the BERTScore (Zhang et al., 2020) to measure the semantic consistency between generated paraphrases and input sentences. We also compute ROUGE-1,2,L between the generated and the reference to evaluate the generation quality.
Automatic evaluation The experimental results on the paraphrase generation task are shown in Table 5. Our proposed model outperforms other baselines on all the diversity metrics. In terms of quality metrics, our POSG performs better than MLE,    Table 9: Results of controllability analysis on the paraphrase generation task. "×n" means that we manually multiply the probability of "Adjective" by n.
FACE, and UL, while the best model in quality, i.e., F 2 -Softmax performs badly in diversity. Moreover, compared with other syntax-guided models, i.e., SGCP, our model performs much better in diversity and has a comparable performance in quality. This further confirms that our model can effectively promote text diversity without the help of exemplars.
To make a more intuitive comparison, we further apply stochastic decoding for different models, and tune the sampling hyper-parameters to generate different sets of paraphrases. Then, we calculate BLEU4 and Self-BLEU4 scores for these sets, and draw the quality-diversity trade-off in Figure 2. Clearly, POSG surpasses all the baselines with a significant gap. These results confirm that our model can produce equally high-quality text that is more diverse, and vice versa. Human Evaluation We also conduct a human evaluation for the generated paraphrases. 100 examples are randomly sampled from each models' outputs, respectively. Each of them are evaluated by five workers from the following four aspects: Lexical Diversity (LeD.), and Syntactical Diversity (SyD.), Fluency (Flu.), Relevance (Rel.). All these aspects are scored between 1 to 5, the higher the better. As shown in Table 6, the results of the human evaluation are strongly consistent with the automatic evaluation. Compared with MLE, UL and SGCP, POSG substantially improves the generation quality, and it only has a tiny gap from the best model in fluency and relevance scores. Meanwhile, POSG has the best scores in diversity, which further verifies that our proposed methods can generate more lexically and syntactically diverse paraphrases. The detailed questionnaire, and other details are shown in Appendix E.

Ablation Study
We perform ablation studies to reveal the effect of POS Guided Softmax and POS Guided Sampling. As shown in Table 7 and Table 8, compared with MLE, POS Guided Softmax (without POS Guided Sampling) can improve text quality for both the tasks. It is worth to mention that, it is natural to find that the model without POSG-Sampling performs better than the model with POSG. That is because POSG-sampling is a stochastic decoding method like nucleus sampling, which will sacrifice the quality of the generated text to promote text diversity. Therefore, POS Guided Sampling can dramatically promote text diversity for both the tasks. These results confirm the effectiveness of both the components.

Interpretability
Compared with one-stage sampling such as top-k sampling, POSG will lead to the entropy increasing of a language model's distribution, and thus lead to more diverse outputs (see Appendix B for the proof, Appendix C.1 for experimental results).

Controllability
Our proposed POSG first samples a POS, and then samples a token from the vocabulary of the previously predicted POS. Therefore, we can control the POS sampling stage by forcing the probability of some specific POS to be higher or lower. For example, on the paraphrase generation task, we can multiply the probability of "Adjective" ("JJ") and renormalize by dividing by the sum, aiming at generating more descriptive style paraphrases.
The results are shown in Table 9. These results confirm that by leveraging POS as an observed and controllable clue, the generated text can be successfully modulated with negligible effect on quality and diversity (see Appendix C.2 for cases).

Conclusion
In this paper, we have introduced POS Guided Softmax and Sampling, simple but effective methods to address the low-diversity problem in text generation. POSG guides models to capture contextual and syntactical information by leveraging POS as an observed and controllable clue in both the training and decoding phases. Experimental results and human evaluation on language modeling and paraphrase generation have demonstrated the effectiveness of our methods.

A.1 Dataset
The dataset statistics of Wikitext-103 and ParaNMT-50M are reported in Table 10 and Table 11, respectively. Since ParaNMT-50M is generated by backtranslation, the dataset has provided translation scores to measure the quality of back-translation, that a low translation score means semantically inconsistent, while a high translation score usually accompanies low diversity. Therefore, we only keep the paraphrase pairs whose translation scores are between 0.7 and 0.8. Moreover, for better training, we remove the sentences that are less than 10 tokens. Finally, we get a filtered dataset containing 1.6 million paraphrase pairs with both high quality and diversity.
For language modeling, we use the original settings of Wikitext-103 dataset for training, validation, and test set splitting. For paraphrase generation, we use the filtered training, validation set of ParaNMT-50M, and the test set provided in the work of SGCP. It is worth to mention that Wikitext-103 is under the CC BY-SA 3.0 license, and ParaNMT-50M is under the CC-BY license.

A.2 Architectures and Hyperparamters
For the language modeling task, we use a 12-layer Transformer Decoder with 8 attention heads, embedding dimension 512, and projection dimension 2048. For the paraphrase generation task, we use a 6-layer Transformer Encoder and Decoder with the same other settings. All the algorithms are implemented in Pytorch and trained on a machine with 8 NVIDIA GTX 2080Ti GPUs for 10 epochs with the hyper-parameters reported in Table 12  We choose the architecture settings and batch sizes according to the GPU memory constraint. Note that we use FACE-OPR among the four variants of FACE, and we train it in the way of finetuning with corresponding finetuning LR and finetuning step. Additionally, we use 7 mixture components in MoS.

A.3 Metrics
Note that, the calculations of Self-BLEU are different for language modeling and paraphrase generation. This is because the typical definitions of Self-BLEU for these two different task are indeed different. For language modeling, Self-BLEU (Zhu et al., 2018) is a metric to evaluate the inner diversity of the generated data, while for paraphrase generation, Self-BLEU (Cao and Wan, 2020) is used to evaluate the degree to which the generated paraphrases are different from the original sentence.

B Proof
We prove that our POS Guided Softmax and Sampling can certainly generate more diverse text than the one-stage sampling, top-k sampling as an example.
In information theory, the entropy of a random variable is the average level of "information", "surprise" in the variable's possible outcomes. Therefore, we can use the entropy of a language model's distribution p(x) to measure its diversity. We denote the entropy of p(x) as H(p): H(p) = − x∈V p(x) log p(x). The increase of H(p) means the increase of diversity.
For example, compared with greedy search, diversity-promoting sampling methods, such as top-k sampling can increases H Z k , where x max is the token with the max probability, V k is the set of top-k most probable tokens, Z k = x∈V k p(x), and obviously x max ∈ V k . Now, we prove that our POSG with two sampling stages can lead to the entropy increasing, compared with one-stage top-k sampling as an example.
For one-stage top-k sampling, According to the Log sum inequality, it follows: Since p(x, ρ) = 0 for x / ∈ V ρ , it follows: where V k,ρ = {x ∈ V k | ρ ∈ POS(x)}, POS(x) is the set of all POS tags of token x. Thus, V k,ρ ⊆ V k .
For our POSG with two sampling stages, where p ′ (x, ρ) is defined in Equation 2, p ′ (ρ) and p ′ (x | ρ) are defined in Equation 5. Again, according to the Log sum inequality, it follows: For the sake of briefness and fairness, we assume that our POSG adopts pure sampling in the first sampling stage (POS Sampling), and adopts top-k sampling with the same k in the second sampling stage (Token Sampling). So, p ′ (ρ) = p(ρ) for ρ ∈ P , while Note that, in our paper, we denote all the tokens whose POS is ρ as a vocabulary V ρ , and here, V ρ,k is the set of top-k most probable tokens in V ρ . Thus, V ρ,k ⊆ V ρ . Then, it follows: Since V k,ρ ⊆ V ρ,k and we use the same setting of k, i.e., Z 2 ≈ Z k , we can finally conclude from Equation 8 and Equation 11 that the lower bound of H(p) (POS) is greater than or equal to the lower bound of H(p) (top-k) . When compared with other one-stage sampling strategies, this conclusion still holds, and can be proved in a similar way. Consequently, this will account for the effectiveness of our methods.
C Additional Analysis C.1 Compared with One-stage Sampling We further conduct an analysis to test whether the traditional one-stage sampling can achieve the  Table 13: Results of POSG and one-stage sampling (we use top-k sampling here) on the paraphrase generation task. Note that we tune the sampling hyper-parameters of both methods to reach the same level of diversity, and then compare the text quality. same level of diversity by increasing the randomness, e.g. using larger k in top-k sampling. On the paraphrase generation task, we tune the sampling hyper-parameters in top-k sampling and our POSG to reach the same level of diversity, and then compare the text quality. The results are shown in Table 13. In this experiment, POSG adopts top-k sampling with k (POS) = 5 in POS sampling, k (token) = 500 in token sampling, while MLE adopts top-k sampling with k = 1000. Obviously, our POSG significantly outperforms top-k sampling on MLE in terms of quality metrics, while performing equally well in diversity. Therefore, we can conclude that, by increasing the randomness, the traditional one-stage sampling on MLE can finally achieve the same level of diversity as our POSG, but the quality of the generated text will seriously deteriorate. This further confirms the advantage of our methods over prior works.

C.2 Controllability Analysis Example
An example of the controllability analysis is provided in Table 14. When we control the probability of adjective increasing during the POS sampling stage, the generated paraphrase will contain correspondingly more adjectives.

C.3 Tuning α (POS) and k (POS)
We have conducted an additional analysis on paraphrase generation by tuning α (POS) or k (POS) while fixing all other hyper-parameters. The results are shown in Figure 3 and Figure 4. It can be observed that: with other sampling hyper-parameters fixed, as α (POS) or k (POS) increases, the results of diversity metrics get better, but the results of quality metrics get worse.  Source: this is going to make good economic sense for the city .
Reference: that it would be good for the city in a certain economic sense . MLE: this will be an economic sense for the entire city . FACE: this will create a good economic point in the city . F 2 -Softmax: this will make sense of economic sense for the city . UL: this will be considerable economic considerations for the city 's going to be able to economic point of the city . SGCP: this will make economic sense for the city . POSG: it is what makes good economic sense to the city .   Table 16. We observe that almost all models can generate high-quality paraphrases with well-preserved semantic meanings, while our POSG exhibits more syntactic diversity than other baselines.

E Human Evaluation
We post the human evaluation questionnaire, as shown in Table 19 and Table 20, and then recruit five workers with sufficient high English skills. We pay each worker 60 US dollars, and let them complete the evaluation within a week. For both tasks, workers are given 100 randomly sampled inputs, and corresponding outputs from each model. Then, they need to score those outputs according to the description in the questionnaire. The term "diversity" in language modeling is typically regarded as a property of the collective outputs of a system, but it is really difficult for a human to remember such a large scale of outputs and give an overall score for a system. So we make a compromise that we asked the worker to rate the diversity of individual outputs, and intuitively the more diverse individual outputs are, the more diverse the system is.
We employ the Krippendorff's alpha for the interannotator agreement analysis. As shown in Table 17 and Table 18, all the results are fair agree-ment (0.2 ≤ κ ≤ 0.4) or moderate agreement (0.4 ≤ κ ≤ 0.6).

Models
Div.

F Impact of the POS tagger
In our work, we use an off-the-shelf POS tagger to annotate the POS tags, and build the POSG upon these annotated POS tags. Consequently, the better the quality of POS tagging, the better the performance of our method. Stanford CoreNLP's POS tagger (Manning et al., 2014), the POS tagger we use, is one of the state-of-the-art tagger, which is the most commonly used tool for NLP research. This ensures the high quality of tagging results.

G Impact Statement
Our work has developed generic generation methods to promote text diversity while maintaining comparable quality. Therefore, despite the contributes to better text generation, our proposed methods may be used to generate more humanlike fake text. But the impacts are more apparent when considering deployed applications, while our proposed methods as the methodologies can not have any direct negative societal impacts. Moreover, all the datasets we used in our work are open source datasets. Wikitext-103 was extracted from Wikipedia, and ParaNMT-50M was created by the back-translation. Therefore, the data we used would not contain personally identifiable information or offensive content.
The goal of this review is to evaluate the quality and diversity of generated texts. In this review, you will read an excerpt from Wikipedia with first 50 words as prefixes, and its corresponding 100-word continuations. You should rate the continuations between 1 -5 in two ways: (1) Diversity. The overall diversity of text can be evaluated from form (How to say it?) and content (What to say?). (1 = The continuation is always repeating some words, its sentences share the similar forms syntactically and lexically, and its content is dull; 5 = The continuation seldom repeats words, its sentences have various syntactical and lexical forms, and it contains different things related to the prefix) (2) Quality. The overall quality of text can be evaluated in many different aspects, such as fluency, readability, coherence, and so on. (1 = The continuation is incoherent, difficult to understand, not related to the prefix, and has many syntactically and semantically errors; 5 = The continuation is coherent, easy to understand, related to the prefix, and grammatically correct) You should score between 1 -5, where 5 is best and 1 is worst. You can consider and make a final decision by comparing different continuations of the same prefix. These prefixes and continuations have been preprocessed by separating punctuation, and splitting conjunctions. And because of length constraints, they may be truncated in the middle of the text. Please ignore these when evaluating and do not allow them to affect your judgments. For example, if you are given the following prefix and continuations: Prefix: Infantry fought a series of hard battles . Simultaneously it had to send combat patrols to its rear to clear infiltrating North Koreans from Changnyong and from its supply road . On the morning of September 1 the 1st and 2nd Regiments of the NK 9th Division , in their. . .

Continuation 1:
Score: Diversity 3, Quality 3 second action around 380 yards <unk> 290 m <unk> from .395 , were assigned to the 7th Division. Meanwhile , the 2nd Regiment of the 27th Infantry , which had been sent out of Murmansk in late October , had moved to iconoclasts Pass . It was the first major US unit to advance across the mountains to the west , but it was not successful . On the morning of September 2 , the Eighth Army's 2nd Battalion , 5th Cavalry Regiment was moved into a position about 10 miles <unk> 16 km <unk> from Highton .

Continuation 2:
Score: Diversity 1, Quality 5 reserve , were positioned to engage the North Korean troops . The 3rd Regiment of the NK 10th Division was to attack the North Korean forces . The 1st Regiment of the NK 12th Division was to attack the North Koreans from the north and east of the 2nd Division , and the 1st Regiment of the NK 15th Division was to attack the North Koreans from the south . The 1st Regiment of the NK 12th Division was to attack the North Koreans from the south . The 1st Regiment of the NK 15th Division was to attack the Continuation 3: Score: Diversity 1, Quality 1 position north of Wonju , were repeatedly pushed back by the ROK 3rd Division . At 08 : 00 am the units of the 1st Battalion attempted to attack . Kim of the 1st and 2nd Battalions attacked the 3rd and 3rd Battalions of the 2nd Battalion of the 3rd Battalion of the 3rd Battalions of the 1st Battalion of the 2nd Battalion of 2nd Battalion , 7th Marines on North , 7th Marines on Hill 60 . Task Force 51 and 9th Marines attacked Sangju 's 1st Battalion of the 3rd Battalion of the 2nd Battalion , 1st Platoons Continuation 4: Score: Diversity 4, Quality 3 " Series B " Company , carried out three assaults on the Pusan on 29 September against three resistance groups that included the blacksmiths , truck commanders , and air support . They then conducted three raids into a line south of psalmody by the 2nd Battalion , 3rd Field Artillery Regiment . At the same time , units from the 3rd Infantry Division and the 3rd Marine Division advanced on all four sides of the road , while infantry units of the 2nd Infantry Division advanced on the northern slope . The 5th Marine Corps , in particular Analysis: As for diversity, Continuation 1 gives various details about the "hard battles", and is of high diversity in the text form. But all the content of it is about the deployment of armies, which means low content diversity. Therefore, Continuation 1 gets 3 points in Diversity. Since there are some words difficult to understand (highlighted in red), Continuation 1 gets 3 points in Quality. Continuation 2 keeps talking about only one single content, that is "some Regiment attacks the North Koreans from somewhere" (highlighted in orange). Although it is fluent, relevant, and gets high scores in Quality, Continuation 2 will receive the lowest score in Diversity due to its dull content. Continuation 3 contains many useless repeating text (highlighted in blue), which makes the continuation incoherent and hard to understand, so it gets the lowest score in both Quality and Diversity. Continuation 4 also states from many different aspects of the "hard battles", but compared to continuation 1, it is not that diverse (That's why comparing different continuations can help to make a decision). Therefore, it gets 4 points in Diversity. In the meantime, high diversity of it also leads to some strange words in the text, and affects the overall quality. So, Continuation 4 can only get a mediocre score in Quality.