{'fraction_non_alphanumeric': 0.04113456109692163, 'fraction_numerical': 0.017240220459739213, 'mean_word_length': 4.47910145461241, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Coherent and Cohesive Long-form Text Generation

Generating coherent and cohesive long-form texts is a challenging task. Previous works relied on large amounts of human-generated texts to train neural language models. However, few attempted to explicitly improve neural language models from the perspectives of coherence and cohesion. In this work, we propose a new neural language model that is equipped with two neural discriminators which provide feedback signals at the levels of sentence (cohesion) and paragraph (coherence). Our model is trained using a simple yet efficient variant of policy gradient, called ‘negative-critical sequence training’, which is proposed to eliminate the need of training a separate critic for estimating ‘baseline’. Results demonstrate the effectiveness of our approach, showing improvements over the strong baseline – recurrent attention-based bidirectional MLE-trained neural language model.


Introduction
The terms coherence and cohesion in linguistics are commonly defined as follows (Williams and Colomb, 1995).
• Cohesion: sentence pairs fitting together the way two pieces of a jigsaw puzzle do.
• Coherence: what all the sentences in a piece of writing add up to, the way all the pieces in a puzzle add up to the picture on the box. In layman's terms, cohesion indicates that two consecutive sentences are locally well-connected, and coherence indicates that multiple sentences globally hold together.
Generating cohesive and coherent natural language texts that span multiple sentences is a challenging task for two principal reasons. First, there is no formal specification of cross-sentence linguistic properties, such as coherence and cohesion of a text. Secondly, there is no widely accepted model to measure the two properties.
Most state-of-the-art neural approaches to natural language generation rely on a large amount of human-generated text to train language models Graves, 2013;Sutskever et al., 2014). Although these models can generate sentences that, if judged individually, are similar to human-generated ones, they often fail to capture the local and global dependencies among sentences, resulting in a text that is neither coherent nor cohesive. For example, neural language models based on Recurrent Neural Networks (RNNs) are widely applied to response generation for dialogue (Vinyals and Le, 2015;Shang et al., 2015;Sordoni et al., 2015;Li et al., 2015). Although the responses by themselves look reasonable, they are detached from the whole dialogue session. See  for a comprehensive survey.
In this paper, we address the challenge in a principled manner, employing a pair of discriminators to score whether and to what extent a text is coherent or cohesive. The coherence discriminator measures the compatibility among all sentences in a paragraph. The cohesion discriminator measures the compatibility of each pair of consecutive sentences. These models, given a conditional input text and multiple candidate output texts, are learned to score the candidates with respect to the criterion. The scores are used as reward signals to train an RNN-based language model to generate (more) coherent and cohesive texts.
Contributions. Our main contributions are: (1) we propose two neural discriminators for modeling coherence and cohesion of a text for longform text generation; (2) we present a simple yet effective training mechanism to encode these linguistic properties; (3) we propose negative-critical sequence training, a policy gradient method that uses negative samples to estimate its reward baseline and therefore eliminates the need for a sepa-rate critic function; and (4) we develop a new neural language model that generates more coherent and cohesive long-form texts, and empirically validate its effectiveness using the TripAdvisor and Yelp English reviews datasets.

Related work
Coherence and cohesion. Coherence and cohesion have been extensively studied in the computational linguistics community, particularly in the 'pre-deep-learning' era. Lack of formal specifications for coherence and cohesion (Mani et al., 1998), resulted in many different formalisms, such as Rhetorical Structure Theory (Mann and Thompson, 1988), and other forms of coherence and cohesion relations and their quantification (Mani et al., 1998;Hobbs, 1985;Hovy, 1988;McKeown, 1985;Cohen and Levesque, 1985;Hovy, 1991;Cristea et al., 1998;Halliday and Hasan, 1996;Liddy, 1991;Van Dijk, 2013;Edmundson, 1969;Barzilay and Lapata, 2008). This list is not exhaustive. However, prior work jointly exploring coherence and cohesion using neural models in the context of long-form text generation has not come to our attention.
Reinforcement learning for text generation. The text generation task can be framed as a reinforcement learning (RL) problem (Daumé et al., 2009), in which the generator G is acting as a policy π, with parameters θ π , and each generated word at time t, w t , can be viewed as an action to be chosen by the policy from a large discrete space, or vocabulary, conditioned on state s t−1 = w ≤t−1 .
Let r t be the reward for a partially generated text sequence w ≤t . We define the long-term expected reward J (π) = E s 0 ∼q,π [ ∞ t=1 γ t−1 r t ], where q is the initial distribution of conditional input texts. Following Sutton et al. (1999), the gradient of J with respect to θ π is where ρ π is the stationary distribution and Q π (s, a) is the expected return from state s and taking action a, both following policy π. For brevity, we omit the derivation. In this work, we formulate text generation as an episodic RL problem with episode length L, rewards r L being available only at the end of episode and γ = 1.
There are many works on training neural language models using rewards, such as Ranzato et al. (2015) and Paulus et al. (2017). These works directly optimize for specific metrics, such as BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003), using REINFORCE (Williams, 1992). However, these metrics do not give a complete picture of the text generation quality. Only recently have there been efforts to provide more relevant objectives, such as consistency and repetition in a text (Li et al., , 2016aHoltzman et al., 2018). But these works use the objectives to re-rank candidate outputs, not to reward or penalize them. Li et al. (2016b) constructed a set of reward models for the dialogue task, such as information flow and semantic coherence, to tune the generator, yet they do not provide an ablation study on the relative contribution of these reward models individually. It is not clear that these reward models can be generalized to other tasks, in particular, long-form text generation tasks.
The most relevant to our work is , which promotes text generation in the correct order, and discourages in its reverse order using rewards. However, this may not be sufficient in capturing coherence since there are many negative orderings given a paragraph. From this pool, we assess the relative quality of generations. Furthermore, we model cohesion between consecutive sentence pairs using word-level features.
GANs for text generation. Another line of research involves the use of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to incorporate feedback signals for text generation Lin et al., 2017;Guo et al., 2017;Fedus et al., 2018;Zhang et al., 2018). The discriminators in these works are trained to distinguish real texts from generated ones, operating as a black-box than providing feedback on linguistic aspects. Yang et al. (2018) partially addressed this issue by using a trained language model as the discriminator. Although the discriminator provides a fine-grained feedback at the word level, it does not model linguistic properties, such as cohesion and coherence.
Many text generator models are inadequate for generating a cohesive and coherent long-form text that span multiple sentences. As a result, human readers can easily distinguish the generated texts from real ones. In this paper, we argue that the primary reason is the lack of an effective mechanism to measure and control for the local and global consistency in model-generated texts.

Coherence and Cohesion Models
We assume that global coherence of a text depends to a large degree upon how its individual sentences with different meanings are organized. Therefore, we focus our evaluation of coherence solely based on the sentence-level features. If the sentences are not organized properly, the intention of the paragraph as a whole is obscure, regardless of seamless local connectivity between consecutive sentences. This is not to say that local connections between any two neighboring sentences can be overlooked. One can easily distinguish a generated sentence from a real one by judging whether it is semantically cohesive with its neighboring sentences.
We strive to embody these two different yet important concepts by developing coherence and cohesion discriminators, operating on the sentence level and word level, respectively. Our design of these two discriminators is inspired by the Deep Structured Semantic Model (DSSM) which was originally developed to measure the semantic similarity between two texts (Huang et al., 2013;Gao et al., 2014;Palangi et al., 2016;Xu et al., 2017). In this study, we extend 'semantic similarity' to coherence and cohesion in a long-form text.

Coherence discriminator: D coherence
The coherence discriminator models the coherence score, which measures how likely two text chunks add up to a single coherent paragraph. Let S := [s 1 , s 2 , ..., s n ] be the source text chunk that consists of n sentences, T := [t 1 , t 2 , ..., t m ] be the real target text chunk that consists of m sentences, and T := t 1 , t 2 , ..., t m be the artificially constructed incoherent target text chunk that consists of m sentences. D coherence is designed to distinguish a positive (coherent) pair (S, T ) from a negative (incoherent) pair (S, T ) by assigning different scores, i.e., D coherence (S, T ) > D coherence (S, T ).
Model architecture. The model takes a form of dual encoder. Given source text chunk S and target text chunk T , the coherence discriminator D coherence computes the coherence score in three steps, as illustrated in Figure 1 (upper). First, each sentence is encoded by the bag-of-words (BOW) embedding, i.e., the average of its word vectors from a pre-trained word embedding (Pennington et al., 2014). Secondly, an encoder which can be implemented using a convolutional neural network Figure 1: Illustration of coherence and cohesion discriminators. D coherence takes in bag-of-words sentence embeddings as inputs, and D cohesion takes in the raw word embeddings of consecutive sentences as inputs.
The source encoder f (or u) is different from the target encoder g (or v).
(CNN) 1 or RNN 2 , denoted as f , takes as input the BOW vectors of the source text chunk S and encodes it into a single vector f (S). Similarly, g encodes the target text chunk T into g(T ). The two encoders f (·) and g(·) share the same architecture but do not share parameters, i.e., θ f = θ g , and thus D coherence (S, T ) is not symmetric. Thirdly, D coherence (S, T ) is computed as the cosine similarity of the two vectors f (S) and g(T ). The score is a real value between −1 and 1, where 1 indicates maximal coherence, and −1 minimal coherence.
Note that we use the simple BOW vectors to encode sentences in the coherence discriminator, which is different from the CNN sentence embedding scheme in the cohesion discriminator that we introduce in Section 3.2. Although the BOW vector ignores the word-order information in the sentence, it is empirically shown to be effective in preserving the high-level semantic information in the sentences and achieves success in sentence similarity and entailment tasks (Wieting et al., 2016;Arora et al., 2017). Because high-level semantic information of sentences is sufficient to determine whether a paragraph is coherent, we choose to use BOW vectors to encode sentences in D coherence .
The parameters of D coherence , θ f and θ g are optimized using a pairwise ranking loss. To this end, we need both positive and negative pairs. While the positive (coherent) pairs come from the train-ing data, negative (incoherent) pairs need to be artificially constructed. The next section describes the way these negative pairs are generated.
Constructing negative (incoherent) pairs. Given a training minibatch Wieting et al. (2016). For notation simplicity, we omit the minibatch index i in the rest of this section. For each positive pair (S, T ) in the minibatch: • We rotate T with S fixed, and thus obtain all • We shuffle the sentence order in T once, known as a derangement, to break its coherence. This yields one negative pair (S, T ). • We combine the previous two methods, that is, we rotate T in the minibatch and shuffle sentences within the target chunk, yielding another B − 1 negative pairs {(S, T j )} B−1 j=1 . These 2B−1 negative pairs and a single positive pair, in total, pose a challenge for the discriminator in learning to retrieve the correct pair.
Training using a pairwise ranking loss. The parameters of f (·) and g(·) are optimized in such a way that a positive pair scores higher than its negative pairs, i.e., D coherence (S, T ) > D coherence (S, T j ) for any j. To achieve this, we propose to minimize the following pairwise ranking loss (Gong et al., 2013) with margin δ: Notice that AVG λ is the mean operator when λ = 0 and approaches the max operator when λ → ∞. These two extreme cases correspond to ranking against the average of all negative pairs and ranking against the single most challenging negative pair, respectively. Empirically, training the models using the weighted average (0 < λ ∞), which assigns larger weights to more challenging negative pairs, stabilizes the training and expedites the convergence.

Cohesion discriminator: D cohesion
The cohesion discriminator models the cohesion score, which measures how likely two sentences form a cohesive pair of consecutive sentences. Let s k := s 1 k , s 2 k , ..., s n k be the k th sentence that consists of n words, s k+1 := s 1 k+1 , s 2 k+1 , ..., s m k+1 be the real next sentence that consists of m words, and s k+1 := s 1 k+1 , s 2 k+1 , ..., s m k+1 be the artificially constructed incohesive next sentence that consists of m words. D cohesion is designed to distinguish a positive (cohesive) pair (s k , s k+1 ) from a negative (incohesive) pair (s k , s k+1 ) by assigning them with different scores, i.e., D cohesion (s k , s k+1 ) > D cohesion (s k , s k+1 ).
Model architecture. Like the coherence discriminator, this model also takes a form of dual encoder. Given (s k , s k+1 ), D cohesion computes the cohesion score in three steps, as illustrated in Figure 1 (lower). The first step is to obtain two sequences of word embedding to represent the two sentences. Then, a pair of source network u(·) and target network v(·) are utilized to encode both s k and s k+1 into two low-dimensional continuous vectors. The two encoders u(·) and v(·) share the same architecture but do not share parameters, i.e., θ u = θ v , and thus the D cohesion (s k , s k+1 ) is not symmetric. Finally, D cohesion (s k , s k+1 ) is computed as the cosine similarity of the two vectors.
Note that we use CNNs or RNNs to embed sentences in D cohesion , which takes the word order in a sentence into consideration. This is different from the BOW embedding in the D coherence where the word order does not matter, because the word order indeed matters when determining the cohesion of two consecutive sentences. As an example from Table 1, for the source sentence "Once you get there you are greeted by the staff.", "They explain everything to you." is a cohesive follow-up while "You explain everything to them." is not.
The parameters of D cohesion , θ u and θ v are optimized using the same pairwise ranking loss. The positive pairs (a training minibatch) for D cohesion is obtained from (1) decomposing each paragraph into pairs of consecutive sentences and (2) randomly selecting B pairs as the positive (cohesive) pairs {(s k , s k+1 ) i } B i=1 . We construct negative (incohesive) pairs using the same methods as in the coherence discriminator.
• We combine the previous two methods and obtain additional pairs {(s k , s k+1,j )} B−1 j=1 . In total, we obtain 2B − 1 negative pairs for each positive pair in the minibatch.
Training using a pairwise ranking loss. The parameters of u(·) and v(·) are optimized such that D cohesion (s k , s k+1 ) > D cohesion (s k , s k+1,j ) for any j. To achieve this, we propose to minimize the following pairwise ranking loss with margin δ: We leave the training details and hyperparameter configurations to Section 5.2.

Negative-Critical Sequence Training
for Long-form Text Generation

Long-form text generator: G
The generator G is an attention-based bidirectional sequence-to-sequence model  and is pre-trained by maximizing the log likelihood on training data, which we denote as G MLE . However, long-form texts generated using G MLE often do not meet our high coherence and cohesion standards. We propose to use the two pre-trained discriminators, D coherence and D cohesion , to modify the text generation behavior of G MLE . The scores from the discriminators are used as reward (or penalty) signals to adjust the parameters of G MLE using a variant of policy gradient, called negative-critical sequence training, which we propose for our task and describe in details in the next subsection.

Negative-critical sequence training
For an arbitrary pair of S and T gen , where T gen is the generator's output conditioned on S, we compute the coherence and cohesion scores by calling D coherence and D cohesion . Since each generated text consists of multiple sentences, the overall cohesion score is computed as the mean of all the consecutive sentence pairs, (s k , s k+1 ) ⊂ [S −1 , T gen ], where S −1 is the last sentence from the source.
These scalar scores, however, are not interpretable since the discriminators are trained by op-timizing a pairwise ranking loss. Instead, the differences between positive pair scores and the maximal or average negative pair scores provide insights of how well the models distinguish between the positive and the negative pairs. This difference relates to reward with baseline in actor-critic methods (Barto et al., 1983;Witten, 1977;Williams, 1992;Sutton et al., 1999) that typically require a separate critic function as a baseline. In NLP, we have observed similar practices by Ranzato et al. (2015), Bahdanau et al. (2016), and Nguyen et al. (2017). Rennie et al. (2017) proposed a method that avoids learning a separate critic. Similarly, our method does not require learning a separate critic since this margin is a form of reward minus baseline. Specifically, we define the reward functions with baselines as: where |T gen | denotes the number of sentences in T gen , and E T ( and E s k+1 ) are computed by averaging over an ensemble of negative pairs. Notice that this reward resembles the ranking loss we use to train our discriminators, except that our baseline is the mean score (instead of the weighted mean) over negative pairs. The rationale for this difference is that: because the best artificially constructed negative sample may be a formidably good sample, the maximal or the weighted mean can in fact be noisy as a baseline and thus introduce noise in rewards. To alleviate such noise, we use the mean discriminator score of negative pairs as the baseline, and this turns out to be an empirically better alternative. Then we use policy gradient to maximize a weighted sum of the coherence and cohesion rewards.

Experiments
In this section, we detail the training and evaluation of D coherence , D cohesion , the baseline generator G MLE , and the RL-tuned generators G MLE+RL(cohesion) , G MLE+RL(coherence) , and source cohesion coherence this hotel was unbelievably overpriced . 0.0002 we were looking for something cheaper but thought we would at least be staying in a decent hotel having paid that much when booking . 0.0411 it wasn t clear when booking that we would have to share a bathroom . 0.0084 there was one shower for the whole floor which was tiny and unclean . 0.0054 the room was old and lacking in facilities .
target the beds were very uncomfortable and the linen was very old . 0.0768 breakfast was ok , but the staff were incompetent . 0.0591 on our last day they were too lazy to clean our table and never bothered taking our order .
-0.0097 we had to leave having had no breakfast , as we ran out of time .
0.0457 they saw us get up and leave and didn t even apologise for the appalling lack of service .

+0.3735
negative target the staff recommended great restaurants with very reasonable prices within walking distance . 0.0514 the paris hop on bus stops nearby . 0.0798 the gare l est is within 3 blocks .
-0.0156 we paid 75 euro per nite excluding breakfast but paid for breakfast one day and found it very good and reasonably priced . 0.0082 the rooms are clean and bathrooms ensuite .  Table 1: Coherence and cohesion rewards on test data. The cohesion reward at the end of each line is computed with its next sentence. This is an example of contradiction and inconsistent sentiment, suggestive of incoherence. We append more examples with extreme cohesion rewards.  Table 2: Retrieval ratios for coherence and cohesion discriminators from a collection of 100 negative candidates from the test data. The reported numbers are the averages over 20 evaluations. Notations: Conv 512 2,3,4,5 is a convolutional input encoder with filter sizes 2, 3, 4, and 5, and there are 512 filters for each filter size. GRU 1024 1-layer, bi-dir. is a 1-layered bi-directional GRU input encoder with hidden size 1024. We experimented different configurations for both encoder types, and selected the best performing models for the negative-critical sequence training step.
G MLE+RL(coherence, cohesion) . We show that, by using feedback from the discriminators, the quality of the generated texts is significantly improved. See Table 3 for a sample comparison.

Dataset
We use the TripAdvisor hotel English reviews dataset collected by Wang et al. (2010) and the Yelp English reviews dataset 3 . We use only the source sentences the hotel inglaterra delivered as promised . the staff was welcoming and spoke good english . the cleaning staff did a very good job every day . the rooms were spotless and very modern . the bathroom was large and had a very nice shower , and there were two generously sized bath towels that were twice the size of normal towels .
G MLE the breakfast in the morning was delicious and very good . it was the only hotel where i slept very well . the staff was very helpful in late afternoon or late times . the breakfast was adequate , with a decent range of cereals , fruit , and fruits . there is also free use of the coffee in the reception area .
G MLE+RL(coherence, cohesion) the breakfast was plentiful including fresh breads and cooked to order . the location was fantastic . it is in the north of the marina and in a very short distance . the marina has a small swimming pool with sitting area and a small gym . they are very popular and guests have an evening reception which is very nice . Table 3: Sample generations from our MLE-trained baseline model, G MLE , and our discriminator-guided model G MLE+RL(coherence, cohesion) . The red texts highlight a common problem in G MLE -it exhibits a repetition, and an inconsistent opinion as a review. In contrast, our discriminator-guided model is able to generate a more interesting, and sentiment-consistent continuation.  Table 4: An ablation study with automated evaluation metric scores: NLL, PPL, BLEU-n, intra/inter-unique-n, along with the length ratio with the length of corresponding true target sentences as 1. Significant numbers are highlighted in bold before rounding.
subsets of the two datasets that satisfy the following two conditions: (1) a review must have at least 10 sentences, and (2) each sentence has from 5 to 30 words. This yields roughly 60,000 TripAdvisor reviews and 220,000 Yelp reviews, split into [0.8, 0.1, 0.1] ratio for train/dev/test sets. We merge the source and target vocabularies, and limit it to the top 50,000 frequent words, excluding special tokens. For each review, we use the first five sentences as the input S to G, and the next five sentences as the target output T from G.

Implementation details
Baseline G MLE . G MLE takes individual words as inputs and embeds into a pre-trained GloVe 300dimensional word vectors. This embedding layer is fixed throughout training. G MLE uses a twolayered GRU and hidden size of 1024 for both encoder and decoder. During optimization using Adam (Kingma and Ba, 2014), we set the learning rate to 2e-4 and clip the gradient's L2-norm to 1.0. We initially train G MLE for 60 epochs on the TripAdvisor data and 30 epochs on the Yelp data.
Discriminators. For the CNN-based encoder, the convolutional layer consists of filters of sizes 2, 3, 4, and 5 for D coherence (3, 4, 5, and 6 for D cohesion ), each with 512 filters. Each convolution filter is followed by a tanh activation. Then, we max-pool in time and append a fully connected layer to generate a feature vector of dimension 512, followed by a batch normalization layer and a tanh activation. For the RNN-based encoder, we use a 1-layered bi-directional GRU, concatenate the final hidden states at both ends, and append the same remaining layers.
Both discriminators use the pre-trained GloVe word embedding vectors 4 , which are fixed during the training. We use an Adam optimizer with a learning rate of 1e-5. We fix λ = 2 and δ = 0.2 in equations (1) and (2). 5 We train both discriminators for 50 epochs and choose the models with the best R@1 scores on the validation dataset.
Model G MLE+RL . In the fine-tuning stage, we use the negative-critical sequence training method,  as described in Section 4, up to 5 epochs, with a learning rate of 1e-5. We equally weight the coherence and cohesion rewards, 1 2 R coherence (S, T gen )+ 1 2 R cohesion ([S −1 , T gen ]). We also continue the supervised learning of G to constrain the policy search within a space that represents the sentences that are likely to be grammatically plausible, similar to Paulus et al. (2017); Wu et al. (2016); Lewis et al. (2017). For all the generations from G MLE and G MLE+RL , we use the simple greedy decoding method because we do not observe any significant difference when switching to beam search.

Results
Evaluating D coherence and D cohesion . Since the discriminators are implemented as pairwise rankers, we employ the metrics commonly used in information retrieval for evaluation, i.e., recall at K (R@K), which is defined as the fraction of correctly identifying an item in the TOP-K retrieved list (Baeza-Yates and Ribeiro-Neto, 1999). We present the retrieval results in Table 2.
To help readers understand the roles of D coherence and D cohesion , we present examples of positive and negative pairs and their rewards in Table 1.
Automatic evaluation of G. It is widely known that there is no perfect automated metric to evaluate text generators. Nevertheless, we report the scores of widely used metrics, including negative log-likelihood (NLL), perplexity (PPL), BLEU and the proportion of unique n-grams within a single generation (intra-unique-n), and across generations (inter-unique-n), as in Gu et al. (2018). Results in Table 4 show that our discriminators significantly improve BLEU scores, NLL and PPL, with marginal difference in diversity.
Human evaluation of G. Coherence and cohesion of a text cannot be easily measured using standard automated metrics. Thus, we perform crowd-sourced human evaluation. We ran-domly selected 200 samples from the TripAdvisor dataset, including corresponding generated output from the baseline G MLE and our model G MLE+RL . For comparison, we pair systems as (Human ↔ G MLE+RL ) and (G MLE+RL ↔ G MLE ).
The outputs of these system pairs are presented in random order and each is ranked in terms of coherence and cohesion using a five-point Likert scale by human judges. Initially, we hired 7 judges to judge each pair. We identified a group of poor judges (probable spammers) who choose G MLE+RL over the Human more than 40% of the time, and eliminated them from the judge pool. Table 5 reports the final scores in terms of percentages of the total remaining judgments.

Conclusion
This paper proposes a neural approach to explicitly modeling cross-sentence linguistic properties, coherence and cohesion, for long-form text generation. The coherence discriminator D coherence provides a macro-level view on structuring a paragraph. The cohesion discriminator D cohesion provides a micro-level view on local connectivity between neighboring sentences. The pre-trained discriminators are used to score the generated texts and artificially constructed negative pair scores are used to form baselines for the policy gradient, which we call negative-critical sequence training, to train neural language models.
On two long-form text generation tasks, human evaluation results are consistent with automatic evaluation results, which together demonstrate that our proposed method generates more locally and globally consistent texts with the help of the discriminators.
Despite the encouraging initial results, we only scratched the surface of the problem. The proposed method is yet to be significantly improved to meet the ultimate goal of generating meaningful and logical long-form texts.