{'fraction_non_alphanumeric': 0.03401077007719111, 'fraction_numerical': 0.010603357729947816, 'mean_word_length': 4.48431928316723, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Global-aware Beam Search for Neural Abstractive Summarization

This study develops a calibrated beam-based algorithm with awareness of the global attention distribution for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring mechanism is then developed to regulate beam search to generate summaries in a near-global optimal fashion. This novel design enjoys a distinctive property, i.e., the global attention distribution could be predicted before inference, enabling step-wise improvements on the beam search through the global scoring mechanism. Extensive experiments on nine datasets show that the global (attention)-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters. The algorithm is also proven robust as it remains to generate meaningful texts with corrupted attention distributions. The codes and a comprehensive set of examples are available.


Introduction
As the barriers exist from the auto-regressive design of neural probabilistic text generators to predicting the global optimum directly [30], the heuristic algorithm beam search that factorizes global optimization to multiple local optimizations, has been popularly used for text decoding [24]. In the original beam search setting, the global optimum is a hypothesis g of the highest probability among all possible sentences, and consists of words in vocabulary V. Given the global optimum at step t denoted as g ≤t , the local optimums l ≤t refer to K candidate sequences with the highest probabilities at each step. While it is necessary to compromise on the beam size K V to ensure text quality [3, 25,24] and search efficiency, beam search suffers from a major limitation due to its local property. Concretely, assuming that the global optimal hypothesis is within the K local optimal hypotheses of the highest probabilities, i.e. p(g ≤t ) ≥ p(l ≤t ), for all t until the termination T , it operates solely with the local information available at each step. In practice, such assumption may however fail in the case that the probability of the global optimum at step τ < T is less than those of the local optimums, i.e. p(g ≤τ ) < p(l ≤τ ), but is adjusted to a higher level in the later steps, p(g >τ |g ≤τ ) > p(l >τ |l ≤τ ) and p(g ≤τ )p(g >τ |g ≤τ ) > p(l ≤τ )p(l >τ |l ≤τ ). This often leads beam search to get stuck in the local optimum from step τ onward in generating texts. To cope with this limitation, this study proposes a calibrated beam-based algorithm with global awareness at all searching steps. Generally, our novel algorithm is implemented in two phases. Before the beam search (Phase I), the global attention distribution is predicted in order to be included as a protocol to calibrate beam search at each step, encouraging the generated hypotheses to attend to the source in a more near-global optimal way. Specifically, the global attention distribution describes how all reference tokens should assign the attention to each source token (illustrated in Figure 1.a), which could be predicted from the source by training an attention-prediction model. The training is fairly straightforward and resembles a sequence tagging task [12], except that the predicted attention distribution from the source is a regression result. There are several advantages of using the attention distribution as the global protocol. 1) Attention distributions are sensitive to the decoder input, suggesting that any input to the decoder leads to a unique attention distribution with fixed model parameters; 2) attention distributions are accessible for almost all text generation tasks thanks to the recent advances in attention models [27,4,31]; 3) relying on the source only, the global attention distribution can be predicted before beam search, thus offering a rigorous mechanism to calibrate a global-aware beam search.
During beam search (Phase II), we develop a novel global scoring mechanism composed of attention scores and length rewards to guide beam search based on the predicted global attention distribution. As one main theoretical result, we show that the attention score can be considered as the probability that generated texts attend to sources in line with the predicted global attention distribution. Specifically, the generated tokens in each step update the local attention distribution to source tokens dynamically, where the attention values grow monotonically as the generation proceeds (see Figure 1.b). Since the desired situation is that the local distribution reaches exactly the global distribution at the terminal step, we regulate the inference by discouraging local attention from exceeding their corresponding predicted global attention at all steps.
With regards to the core target to investigate the possible paradigm that improves beam search with global awareness during decoding, contributions of this study are summarized as follows: • We argue that the limitation of beam search roots from its defect in finding the global optimal hypothesis. We improve the algorithm by proposing a global protocol to regulate beam search step-by-step. This paper is the first to predict and deploy the global attention distribution to calibrate the inference in a rigorous way, thus returning a hypothesis that attends to source tokens in a more near-global optimal manner. In contrast, previous works [37,7,20,29,21] try to use attention distributions to improve beam search, but ignore that the global attention distribution is predictable.
• A novel global scoring mechanism is designed to evaluate the generated sequences at each step based on the desired situation described in Figure 1.c. As theoretically justified, its major component can be elegantly integrated into beam search in the form of a probability so that merely O(K) of the time complexity is increased in each step (see Section § 3.1 for more details). • The proposed algorithm with global awareness manifests a robust and plug-and-play property in enhancing beam search for neural abstractive summarization. Without requiring any model or parameter modification, the global-aware inference shows excellent performance in generating meaningful texts, even if the attention distribution is corrupted or not of its own. Further, it is identified that summaries generated by global-aware inference are both higher-quality and different from beam search hypotheses (see Global-aware in Table 1). More interestingly, we find that the generation style of a dataset could be transferred by the designated global attention distribution. For instance, summaries of higher abstractness for CNN/DM could be generated by only replacing its global attention distribution with a highly abstractive distribution during inference, as presented in Global-aware † of Table 1. • On the empirical side, we show that the proposed global-aware inference can stably and significantly boost two state-of-the-art summarization models BART [17] and PEGASUS [39] to produce higher quality summaries on nine datasets, even if only the empirical hyper-parameters are used.

Preliminary
The proposed decoding strategy is applied in BART [17] and PEGASUS [39] to perform summarization. BART is a pre-trained seq-to-seq model whose structure essentially follows a vanilla Transformer encoder-decoder [31]. PEGASUS has a similar structure but is pre-trained on a larger dataset differently. Notably, the fine-tuned parameters of both models are downloaded from Hugging-Face Models 3 and are fixed in all subsequent operations, where "fine-tuned" means the pre-trained model has been fine-tuned on a specific dataset.

Beam Search
Since the decoder of the seq-to-seq model is an auto-regressive model, the probability of a target sequence y = (y 0 , · · · , y t , · · · , y T ) can be factorized to the probabilities conditional on the source x = (x 1 , · · · , x i , · · · , x n ) and y <t = (y 0 , · · · , y t−1 ), i.e., When T = 0, y = (y 0 ) and p(y 0 |x) = 1 because y 0 is a fixed start token. Beam search [8] is a decoding strategy to predict a target sequence by maximizing this factorization. Given a vocabulary set V, at each inference step t, beam search selects a candidate beam set is a candidate sequence) from an all-possible beam set B t of size K × |V|, namely, where argtopk(·) outputs K beams with the highest conditional probability, and • is the concatenation operation. Besides, is an accumulated value. Its calculation can be simplified as: where the value of log p(b k t−1 |x) is computed from the previous step. Therefore, at each step, we only need calculate the condition probability of each token in the vocabulary set.
A beam is terminated after it generates an ending token, and the beam set of K terminated beams is defined as Y. The final hypothesis y * is chosen from Y based on the beam probability normalized by length a where a is a hyper-parameter of length [37]: where y k = (y k 0 , · · · , y k T ). |y k | − 1 is used since the start token is not considered in calculating the length.

Attention Distribution
Attention distribution is a continuous vector whose element indicates the degree of attention paid to a source token. The element is formed by the accumulation of cross attention, i.e., t α t,i , where α t,i refers to the cross attention that the t th token in the target sequence gives to the i th source token. 4 Specially, cross attention is a scaled dot-product [31] of hidden states of the source x and the target sequence y. Notably, since Transformer-decoder is an auto-regressive model, the cross attention assigned by t th target token is actually calculated by y <t = (y 0 , · · · , y t−1 ).
Global Attention Distribution. The global attention distribution g = [g 1 , · · · , g i , · · · , g n ] ∈ R n is the attention distribution given by the reference, where global attention g i refers to the total attention that the reference attends to the i th source token, and n is the source length.
Optimal Length. The summation of g, namely n i=1 g i , is equal to n i=1 T t=1 α t,i = T due to n i=1 α t,i = 1 in softmax operation, where T is the reference length, or equivalently the optimal length Z.
Local Attention Distribution. The local attention distribution l k t = [l k t,1 , · · · , l k t,i , · · · , l k t,n ] ∈ R n is the attention distribution of the k th generated sequence and updated at each decoding step t. Thereinto, the local attention l k t,i denotes the total attention paid to the i th source token by the k th beam sequence (b k 1 , · · · , b k t ) and is dependent on the sequence generated before t, i.e., b k t−1 = (b k 0 , · · · , b k t−1 ).
3 Proposed Global-aware Inference

Global Scoring Mechanism
The global scoring mechanism consists of an attention scoring function and a length reward function. Given the global attention distribution g, the attention scoring function A(·) at the step t depends on b k t−1 , where ζ k t indicates the total attention that the generated sequence (b k 1 , · · · , b k t ) gives to the source, and ζ k t = |b k t−1 | = t because the assignable attention for each generated token is 1. Notably, Eq. 6 attains the maximum score provided that each l t,i ≤ g i . As mentioned in Section § 1, the reason for 4 Mean pooling is used for multi-layers and multi-heads this design is that we desire the local attention l T,i at the termination is exactly g i , since the final hypothesis is expected to attend to source tokens in the global-optimal manner. Meanwhile, we have l T,i > l t,i for t < T because l t,i monotonically increases on t with α l m,i > 0. Therefore, at any step, the local attention l t,i should not surpass g i . Otherwise, the attention score will decline, and the penalty depends on the total amount by which these l t,i exceed g i (see Theorem 1). Further, the attention score could be considered as the proportion of correctly assigned attention to the total attention given by the generated sequence, where correct assignment indicates that all parts of l t,i do not exceed g i . Also, it could be interpreted as the correct allocation probability of local attention against the global attention distribution (see Corollary 1.1). In this case, the total attention score can be expressed as the same multiplicative form as Eq. 1 to be elegantly integrated into beam search.
is between 0 and 1.
Proof. See App. A.
In addition to the constraint for l t,i , we still desire l T,i = g i for each token. An ideal hypothesis should have two characteristics simultaneously. Namely, its attention score at the termination is the maximum 1, and its length equals the optimal length. Therefore, we introduce a length reward function to cooperate with the attention score to penalize the situation l T,i = g i , which will be discussed at the end of this subsection.
As mentioned before, the total attention score at the decoding step t is defined as: Thus, the joint scoring function J(·) is modified from Eq. 4: , where β is a hyper-parameter to trade-off between the probability and attention score. Similar to log p(b t |x) in Eq. 4, J(b t , x) is also an accumulative score. Consequently, at each step t, we only need compute p(v|x, b k t−1 ) and A(b k t−1 ). Compared with Eq. 4, the time complexity of each step is only increased by O(K) as there are K attention scores. Replacing log p(b t |x) in Eq. 3 by J(b t , x), we can select the top K beams of each decoding step according to not only the probability distribution conditional on local information b k t−1 but also the score conditional on global information g.
Considering the length reward function, the final hypothesis is thus defined as: where R(·) is the length reward function dependent on the optimal length Z and the candidate hypothesis length ζ k T . Exactly, ζ k T is the total attention that a candidate hypothesis (y k 1 , · · · , y k T ) pays to the source and equals |y k | − 1. Besides, the attention score and length reward are weighted by a hyper-parameter γ, and the role of β is to ensure that the two are at the same level relative to the probability. We remove a in Eq. 5 as it only adjusts the length preference without really controlling the length.
The design of R(·) could be straightforward -one only need ensure that it increases as ζ k T approaches Z, and reaches the maximum only at ζ k T = Z. In this paper, we design a step-wise length reward function R(ζ k t , Z) to better balance the relationship between the attention score and the length reward and make the whole searching process as succinct as beam search. We put the design details of the step-wise length reward in App. B, and we regard Eq. 9 as the general scoring formulation of global-aware inference.

Predict the Global Attention Distribution
Since the reference is unknown practically, the global attention distribution could only be predicted from the source. We construct an attention-prediction model to learn the relationship between the source tokens and the global attention distribution.
The input of the attention-prediction model is the fixed encoder output E ∈ R n×d of BART or PEGASUS plus learnable positional encodings P ∈ R n×d , where d is the dimension of hidden states. The input is fed to a learnable Transformer-encoder to obtain E ∈ R n×d that is encoded with additional context information, followed by a linear transformation with an exponential function: where g ∈ R n refers to the prediction of g, W g ∈ R d×1 and b g ∈ R n are the learnable weights and biases. The exponential function is imposed to ensure g > 0. We choose the exponential function for this operation because it is shown stable in the training and testing stage. Given the objective of minimizing the distance between g and g, the loss is defined as their Euclidean distance: The predicted optimal length Z is the sum of elements in g. Note that the length reward function is not affected no matter whether Z is an integer or not.  [39]. Thereinto, XSUM [5] is a highly abstractive dataset whose summaries are all expressed in a short sentence.
Implementation Details. We adopt a randomly initialized 2-layer transformer-encoder in the attention-prediction model wherethe structure of each layer is the same as the BART-encoder layer. The optimizer is the Adabelief-optimizer [41] with eps 1e − 16, betas (0.9, 0.999), weight decay 1e − 4 and learning rate 2e − 5. The attention-prediction model is trained on the training set for about 50, 000 steps, and checkpoints are saved per 10, 000 steps to select the best checkpoints on the development set. Since the attention prediction is slightly different from common sequence tagging tasks, we have summarized two notable points after several attempts -the dropout rate should be 0, and a small learning rate is preferred. All experiments are conducted on 3 RTX 6000. We include the global-aware inference in the generation code of HuggingFace transformers [36]. At the time of evaluation, ROUGE-1, ROUGE-2 & ROUGE-L (R-1, R-2 & R-L) scores [22] are computed from the ROUGE code 5 used by BART [17].

Results
Comparison with Beam Search. Beam search is a hard-to-beat baseline which has stood the test of time and proven its superiority in practice for long [24]. In Table 2, we compare our global-aware   inference to beam search with length regularizations (i.e., α in Eq. 5, accompanied with two hard constraints, namely minimum length and maximum length). We strictly follow the hyper-parameter setups of PEGASUS [39] in terms of beam search, while we only adopt empirical hyper-parameters for our method. Even so, significant improvements can be observed on all the data sets, especially when the summary is of normal or longer length.   Comparison with Other Attention Approaches. In this part, we focus on comparing other approaches which also exploit certain attention distributions to improve beam search. The first is the coverage penalty [37,21]. To enhance its performance in summarization, we replace its preset attention distribution with our predicted global attention distribution. Note that the coverage function can only evaluate the generated sentences at the terminal step. Instead of comparing the global-aware inference to the methods [29,20,7] that aim to reduce repetition using the dynamic attention distribution, we compare our algorithm with the CTRL repetition penalty [13] which has similar motivation but is more systematical and independent of training. Table 3 lists the comparison results against different algorithms. It can be observed that our global-aware approach can improve the performance of beam search stably and fundamentally. We also observe that the attention head masking [1] appears to outperform the global-aware approach on CNN/DM, but it fails to gain any improvement on XSUM. To further show the advantage of the proposed approach, we will take a closer examination on the attention head masking [1] and our proposed approach in the next part.
Further Comparison with Attention Head Masking. First, one should bear in mind that the attention head masking [1] acts on the model instead of beam search, which is opposite to us. Specifically, it selects contents during decoding by disabling partial attention heads for unimportant   tokens to decrease the attention to these tokens. According to the reported results presented in Table  3, we can see that although attention masking achieves amazing results on CNN/DM, it does fail completely on XSUM. Since hiding unimportant tokens from some heads results in the loss of context information of salient tokens, this would lead to its instability. Thus, it could be ineffective for tasks that require contextual information of the whole source such as XSUM. Taking a further comparison, we deploy the attention head combinations selected for CNN/DM and XSUM to examine its effect on PEGASUS [39]. These comparison results are shown in Table 5. Evidently, our method enjoys a robust feature that is able to boost summary inference on various datasets and models even with the same set of hyper-parameters. In contrast, attention masking [1] behaves much sensitive to the changes of models and datasets. Besides, attention masking has to construct its training saliency labels based on the longest common subsequences between a reference summary and the source. This may be hardly achieved in some text generation tasks (e.g. translation) where no common subsequence exists at all. Such drawback presents one main limitation for attention masking.
ORACLE Results and Ablation Study. ORACLE refers to the global-aware inference combined with (true) global attention distribution instead of predicted global attention distribution. The related results have been presented in Table 4, and the significant boosting certifies that the proposed method could improve beam search with the global attention distribution. On the other hand, we conduct ablation experiments on ORACLE and global-aware. Both results indicate that length reward plays an important role in generating normal-length text but causes adverse effect on generating text of very short length. Besides, the performance declines significantly when only length reward is applied, due to the fact that sole length reward cannot calibrate beam search step-wise.
Robustness. We intend to examine the robustness of our proposed global-aware algorithm in this and next part. To do so, we substitute the parameters in CNN/DM's attention prediction model to XSUM's to create more abstractive summaries for CNN/DM. For comparison, we set the minimum length of beam search as 0 to allow it to generate shorter summaries. Table 6 shows the F 1 score, recall and precision of the shorter beam search and global-aware †. It is surprising that even by using the attention distribution from a different dataset with distinct properties, the proposed global-aware mechanism still manages to generate meaningful summaries with competitive F 1 scores, proving the robustness of this algorithm. Moreover, the higher Precision and lower Recall of the global-aware suggest that although information is partially lost, the algorithm still summarizes core information in a concise format, compared to the standard beam search. On the other hand, we exploit a BART model fine-tuned on CNN/DM to generate summaries of NewsRoom directly, and the ROUGE scores of beam search are shown in Table 7. Next, we randomly select 10K and 100K samples from the training set and use them to fine-tune the attention-prediction model, where the global-aware improves beam search substantially. The experiment once again validates the robustness of the proposed inference algorithm, as it maintains a reasonably good performance even from learning a corrupted attention distribution from a BART without fine-tuning.
Sensitive Analysis. We further examine the performance robustness with respect to the hyperparameters. From Figure 2, we could see the global-aware inference is always better than beam search in CNN/DM, no matter how its hyper-parameters change. Besides, the performance is less sensitive to the hyper-parameters when β ≥ 10 or γ ≥ 1. While in XSUM, the global-aware could improve beam search stably with γ = 0, but there is a significant decline when applying length reward. In fact, the attention score favors shorter hypotheses, and the length reward could alleviate the bias. However, if the references of a dataset are already very short such as XSUM, the length reward may lead to a counterproductive effect. Since the setup of CNN/DM is applicable to most datasets, we argue that the global-aware inference is robust to both hyper-parameters in most cases.

Distribution Law
The distribution law of global attention in BART [17] is shown in Figure 3. It is observed that most attention is assigned to the start token and full stops which are not semantically salient, and most of the remaining attention is unevenly allocated to (semantically) overlapped tokens between the source and the reference (i.e., important words). It is worth mentioning that the importance here is no longer a simple binary classification like in [7, 1], but a continuous numerical value decided by the model knowledge learned from data. In general, one should not simply equate the global attention with the word importance, but should be clear that it essentially reflects knowledge learned by the attention model such as the syntactic and semantic structure of sentences. Meanwhile, the distribution law indicates that attention distributions in pre-trained models may not be relevant with the uniform distribution at all. That is to say, it is not reasonable to still use an uniform attention threshold (like the threshold 1 preset in [37,7]) to regulate the decoding of pre-trained models. Last but not least, our general motivation is to alleviate locality bias by aligning the attention distribution of reference and hypothesis, which does not really care how the global attention is distributed only if it is predictable. However, the proposed penalty mechanism is indeed insensitive to some distributions, and we will provide more thinking about this in App. G by applying the global-aware inference to translation.

Why It can be Predicted from Source?
Since the ORACLE experiments indicated that the global attention is helpful for decoding, the concern remains if it is predictable using only source. In App. E.1 we will show the predictability empirically, while here we just provide an interesting explanation. In our opinion, the global attention distribution is an interpretable representation of reference which not only has the characteristics of hidden representation but also can be explained by the source tokens. First of all, given the source, the global attention is calculated by the input reference and trained neural network parameters; this is similar to achieving hidden representation. Moreover, like hidden representation, the global attention distribution could also capture the semantic similarity, e.g., replacing a reference word with a semantically similar word leads to slighter attention changes than that with a random word (see Figure 4.a). Besides, it is observed from Figure 4.b and Figure 4.c that global attention is able to represent the sentence order and factual knowledge. On the other hand, the global attention distribution enjoys a distinct characteristic that each of its features can be explained as the degree of attention paid to a source token, which means changes of such representation are regular to some extent. For example, in Figure 4.c, we distort a numerical number in the reference to violate the fact stated in the source, and then find that the attention assigned to the actual number originally is most transferred to the special tokens and punctuation. Overall, similar sources should have similar global attention distributions, since similar sources often have similar references and global attention distribution is a representation of reference. Moreover, the global attention and source tokens are in an one-to-one correspondence. Thereby, we argue that it is convenient to use the source to predict the global attention distribution.

Related Work
In the field of text generation, efforts have been made to boost beam search with regards to the attention distribution. For instance, some studies engage the attention distribution to penalize the situation where sources fail to be fully covered in translation tasks [37,21], while others [7, 29,20] incorporate dynamic attention distributions to evade tokens that have been highly regarded to reduce repetition. However, none of the aforementioned studies attempts to apply the global attention distribution to acquire the knowledge that the level of attention should be allocated to each token from the reference. Further, the existing score functions used by those studies are rather different from the proposed global scoring mechanism. More details can be seen in App. D.
In addition to the attention distribution, other techniques are developed to improve beam search in terms of length bias [38], diversity-less [32], vapidity [11] and degradation [25,3]. These methods are not included for comparison because they are not suitable for summarization [38,11], or do not aim to enhance beam search as the main purpose [32,25,3]. Besides, we argue that these patches to beam search are supposed to be hard for improving the performance stably and fundamentally (as shown in Table 3 given by our global-aware method) because they fail to specify what a final hypothesis should look like and are easy to trap into local optimums.

Conclusion
Beam search tends to fall into local optimums due to the lack of global information during inference.
To calibrate beam search on the premise of global awareness, this study proposes a predictable protocol to stipulate how a global optimal hypothesis should attend to source tokens. By training a simple prediction model of the global attention distribution, a novel global scoring mechanism is then developed to regulate the inference on a step-wise basis. Our experiment shows that the proposed global-aware beam search generates higher-quality summaries, relative to beam search and other counterparts of the similar nature. Besides, the proposed algorithm is proven robust in generating meaningful texts even with corrupted attention distributions, implying its potential to cooperate with user-defined global attention distributions. We plan to focus our future study on generalizing the global-aware inference to a broader range of text generation tasks, including not only text2text but also image caption [33], multimodal summarization [19], graph2text [14] and data2text [26]. A Proof Theorem 1. Let M = s : l k t,s > g s and ∆ k t = δ | ∀s ∈ M, δ = l k t,s − g s . Given ∆ = δ∈∆ k t δ where ∆ ≥ 0, then A(b k t−1 ) decreases as ∆ increases.
Proof. By the formula derivation, Eq. 6 can be converted to: Thus Eq. 12 is equal to: Then it is easy to prove that A(b k t−1 ) goes down as ∆ goes up.
Corollary 1.1. The bound of A(b k t−1 ) is between 0 and 1.

Proof. Since we have achieved
where A(b k t−1 ) decreases monotonically on ∆ and ∆ ≥ 0, A(b k t−1 ) reaches the maximum 1 when ∆ = 0. Next, we assume a beam can be generated indefinitely. In this case, there must be a situation that each l k t,i > g i . Therefore we have Therefore, the bound of A(b k t−1 ) is proven to be between 0 and 1.

B Length Reward
In the first subsection, we will introduce our step-wise length reward function and how it works. In the second part, we will discuss in details how it is designed.

B.1 Our Step-wise Length Reward
Our length reward function R(ζ k t , Z) is to give a length score to a beam at each decoding step, so that its cumulative score R(ζ k T , Z) for a terminated beam could approach the maximum as its length ζ k T gets closer to the optimal length Z. Our designed length reward function is as follows: Eq. 8 is thus expanded to: In this case, the final score of y k is re-defined as The original R(ζ k T , Z) in Eq. 9 has been integrated to J(y k , x), and exactly equals which gets the maximum at ζ k T = Z. Related theorem and proof are presented in the next subsection.

B.2 How It Is Designed
Intuitively, if we want to give a score to the beam length at each step, the score should reach the maximum when the beam length gets the optimality. Accordingly it can be defined as: Since ζ k t = t, it is represented as: Then, we test whether its cumulative value at the terminated step R(ζ k T , Z) reaches the maximum when ζ k T = Z. The cumulative length reward is defined as If we replace ζ k T by j, it can be represented as: Unfortunately, when we plot the curve of R j as the orange dummy line in Figure 5, we find that it reaches the maximum at a length longer than the optimal length. This prompts us to translate R t from the green dummy line to the green full line, and the adjusted length reward is designed as: which is the same as Eq. 16. After the adjustment, the normalized cumulative length reward R j (the orange solid line in Figure 5) peaks right at the optimal length. Step of t  Proof. Since t ≥ 1 and t ∈ Z + , R t can be regarded as two arithmetic sequences with difference of 1 Z and − 1 Z , so that it is easy to calculate j t=1 R t . Then R j could be computed by: Hence the derivative of R a j is: In addition, when j = Z, letting ∂Rj ∂j = 0, we have: If we solve this equation, we can get: Omitting the constant item in the radical due to 8Z 2 4 under normal conditions, we can get: If we further take D into R t , we can get Eq. 23.  Training: whether the score function is a part of loss function of the seq-to-seq model. Inference: whether the score function is a part of the hypothesis score. Desired Situation: the score function reaches the optimum in this situation. †: the function described in the paper [7] is only activated at the termination, but their code added a step-wise modification on it.
Article Score Function Training Inference Step-wise Desired Situation We summarize the characteristics of each score function in Table 8, where α t,i refers to the cross attention that the t th generated token pays to the i th source token. Previous works [37,21,29,20,7] design the score function based on some assumptions which may be invalid. Specifically, [37], [21] and [7] use the same constant (1 or β) to constrain all source tokens, based on the assumption that the minimum/maximum attention allocated to each token is the same. [29] and [20] assume the attention assigned before t should be lower than the attention of t. Obviously, both assumptions may often not hold in practice. By contrast, our global scoring mechanism does not depend on any assumption but follows the nature of generation, i.e., the growing local attention should not surpass the global attention during inference and exactly reach it at the termination.

E Discussions E.1 Predictability of Global Attention
We plot the decline trend of training loss and validation loss on CNN/DM and XSUM to Figure 6. It is observed that training loss has been falling significantly, while validation loss has an upward trend at later epochs.
On the other hand, we randomly select 1, 000 test samples from each dataset and plot their average R 2 scores of the predicted optimal attention distributions in Figure 7. For each sample, its R 2 score 1]. A higher R 2 signifies better model fitting, while R 2 < 0 means the model is entirely invalid.

E.2 Degradation of Beam Search
It is widely known that text quality generally degrades with increased beam size [3, 25,24]. The degradation originates from the gap between training and inference, where teacher forcing training only need ensure the local optimality, but beam search pursues the global optimality [35,28,40]. The proposed algorithm reduces the gap by informing beam search how the global optimal hypothesis attends to the source so that beam search can be calibrated to pursue a more reasonable global optimum instead of that obtained by teacher forcing training. We enlarge the beam size and find that the global attention distribution can guide beam search to overcome degradation, though the practical improvement might not be significant due to the prediction bias. According to Figure 8, the average ROUGE F 1 score of beam search summaries declines sharply as beam size increases. Even if there is a prediction bias, the global-aware inference manages to resist degradation effectively with increased beam sizes. Moreover, a significant boosting in ROUGE scores is observed as beam size increases in the ORACLE global-aware inference, where the true potential of the algorithm is reflected.

E.3 Length Comparison
As shown in Table 9, even with empirical hyper-parameters, our method often leads to more concise summary with its length closer to the reference length on many datasets. By contrast, beam search,  which conducts searching for length penalty and length constraints, generates many lengthy summaries on some datasets. For example, In CNN/DM, Reddit, and NewsRoom datasets, beam search must extend summaries to obtain higher ROUGE scores, making the generated text more bloated. Overall, in most cases, the global-aware inference could produce summaries of higher quality in a more concise way.

E.4 Are the Outputs Different from Beam Search from the Beginning?
One may question whether these hypotheses are just the subsets or extensions of beam search hypotheses. The answer lies in the selection of β. Specifically, the global-aware with a greater β (such as β = 12 for CNN/DM) generates quite different summaries. We plot the count distribution of discrepancy positions between beam search and global-aware inference in Figure 9. It is interesting to find that about one-third of CNN/DM test samples (total count: 11, 490) start to differ at the first word, and most start to differ at the first 5 words. In other words, the global-aware inference generates texts in a way different from beam search that leads to higher scores. Table 10 shows that summaries generated by our method appear more creative. Generally, a greater proportion of novel words implies that the generated text resembles more human-writing, at the same time increases the risk of deviating from reference.

E.6 Inference Speed
As presented in Table 11, the speed of the global-aware inference is comparable with beam search.

E.7 Our Advantages over Predicting Future Rewards
A previous study [18] guides inference by predicting the future automatic matrix score at each decoding step. There are two main disadvantages of training such predictor, (a) there are too many training samples especially when the target sequence is long, making it hard to train; (b) this method forces us to predict every decoding step, greatly slowing the inference speed. By contrast, one appealing benefit of our approach is its high efficiency, since our method only need predict once but can calibrate beam search at each step, which improves beam search from scratch with very little burden of training and inference.

E.8 Can Our Method Work Together with Attention Head Masking?
There is no doubt that the answer is yes because the attention head masking [1] still need beam search to generate summaries, and our approach is an alternative to beam search. However, we do not recommend the joint usage of the two due to the following three reasons. First, the roles of the two overlap; second, the loss of context information due to masking will negatively affect the global-aware inference; third, adding attention head masking will impose additional burden on training and hyper-parameter searching.

E.9 Can Predicted Attention Distribution still Lead to the Same Theoretical Result?
Through explaining that the global attention and the predicted attention have the same value range, we will simply prove that the predicted attention distribution is also applicable to the theory in the paper. Since each attention value in the global attention distribution is a cumulative value of the attention probability ∈ [0, 1] given by the reference tokens, the lower bound tends to be 0 and the upper bound is 1 * the number of reference tokens. When the reference is infinite, the upper bound is ∞. Therefore, we only need ensure that the predicted attention value is larger than 0 which is constrained by exp.

F Case Study
Examples are presented in the following pages. We sample some good cases and some bad cases of global-aware inference by the ROUGE-1 F 1 score.

Reference
The ramp agent fell asleep in the plane's cargo hold . He can no longer work on Alaska Airlines flights .
Beam search (R-1: 25.05) "Iḿ inside a plane and I feel like itś up moving in the air," the caller says . "There could be a person in there so weŕe going to come back around," the pilot tells air traffic control . The ramp agent is an employee of Menzies Aviation, a contractor for Alaska Airlines . The airline says the man has been permanently banned from working on planes .
Global-aware (R-1: 43.29) A ramp agent fell asleep in the cargo hold of an Alaska Airlines flight . The crew and passengers reported unusual banging from the belly of the Boeing 737 . The pilot radioed air traffic control and said he would make an emergency landing .
Global-aware † (R-1: 50.81) A ramp agent falls asleep in the cargo hold of an Alaska Airlines plane . The plane makes an emergency landing in Seattle . The man has been banned from working on Alaska Airlines planes .

Reference
Three people killed; five wounded in attack on attorney general's office in Balkh province . Staff and civilians have been rescued as gunmen engaged Afghan security forces .
Beam search (R-1: 38.78) A group of armed assailants stormed into the attorney general's office in Balkh province, northern Afghanistan . Two police officers and a security guard of the provincial attorney general's office were among the dead . Most staff members and civilians have been rescued, but an exchange of fire between Afghan security forces and the assailants is ongoing .
Global-aware (R-1: 54.00) Three people were killed and five others were wounded in an attack on the attorney general's office . An exchange of fire between security forces and the assailants is ongoing ." Global-aware † (R-1: 44.00) Gunmen storm attorney general's office in northern Afghanistan . Three people killed, five others wounded .

Reference
Six young Minnesotans conspired to sneak into Syria and join ISIS "by any means necessary," prosecutors say . The men, ages 19 to 21, were arrested Sunday . They plotted for 10 months, the U.S. attorney for the District of Minnesota says .
Beam search (R-1: 47.68) The group of friends, ages 19 to 21, were arrested Sunday . They plotted for 10 months, U .S . Attorney Andrew M . Luger says . Recruiting for the ISIS terrorist network is a problem in Minnesota's Somali community . Four of the men who were arrested appeared in federal court on Monday but did not enter a plea .
Global-aware (R-1: 61.97) Six men, ages 19 to 21, were arrested Sunday . They plotted for 10 months, U .S . attorney says . Recruiting for the ISIS terrorist network is a problem in Minnesota's Somali community .
Global-aware † (R-1: 38.92) The six men were arrested Sunday . They are accused of plotting to join ISIS in Syria . Recruiting for ISIS is a problem in Minnesota's Somali community .

Reference
Raul Reyes: In seeking Latino vote, Marco Rubio his own worst enemy on two key issues: immigration reform, Cuba relations . He says on health care, climate change and other issues, he breaks from Latinos' positions. Polls show they don't favor him .
Beam search (R-1: 35.71) Ruben Navarrette: Marco Rubio has been his own worst enemy on immigration and Cuba . Navarrette: Rubio has embraced a typical conservative approach to immigration . He says Rubio's political philosophy will be a tough sell to Hispanics . Navarrette: Unless he can offer new ideas, his climb to the Republinomination will be steep .
Global-aware (R-1: 36.50) Ruben Navarrette: Marco Rubio is running for the GOP presidential nomination . Navarrette: Rubio has been his own worst enemy on immigration reform and Cuba relations . He says Rubio has embraced a typical conservative approach to immigration . Navarrette: Rubio's political philosophy will be a tough sell to Hispanics .
Global-aware † (R-1: 40.11) Ruben Navarrette: Marco Rubio is running for the GOP presidential nomination . Navarrette: Rubio has been his own worst enemy on immigration reform and Cuba relations . He says Rubio has embraced a typical conservative approach to immigration .

Reference
Police: Yuhei Takashima, 64, says he had sex with girls as young as 14 in Philippines . Officers seize nearly 150,000 photos that the former principal kept of his activities .
Beam search (R-1: 52.48) Yuhei Takashima, 64, says he paid for sex with more than 12,000 women in the Philippines . Police seized 147,600 photos that Takashima took of his activities over the years . The ages of the prostitutes he hired ranged from 14 to over 70, he says .
Global-aware (R-1: 37.33) Police say Yuhei Takashima, 64, paid for sex with more than 12,000 women . He took 147,600 photos of his activities over more than a quarter of a century, police say .
Global-aware † (R-1: 49.28) Police: Yuhei Takashima, 64, paid for sex with more than 12,000 women in the Philippines . He took 147,600 photos of his activities, police say .

Reference
Kabul faces uncertain future as NATO presence -and the money that came with it -fades away . Interpreters are out of work, NATO trucks sit idle on roads, restaurants are empty .
Beam search (R-1: 21.95) Kabul is a city swollen in size but shrunken in scope, anxiously awaiting what comes next . Vast supply chains once kept 120,000 troops fed and watered . Now the bases are gone, and the trucks that once supplied millions sit still . One road, forever pot-holed in the past decade, is now being covered over by Afghans .    Table 14: Generated summaries on arXiv sampled by ROUGE-1 F 1 score.
Reference in this paper , we develop a structure -preserving discretization of the lagrangian framework for electromagnetism , combining techniques from variational integrators and discrete differential forms . this leads to a general family of variational , multisymplectic numerical methods for solving maxwell s equations that automatically preserve key symmetries and invariants . in doing so , we demonstrate several new results , which apply both to some wellestablished numerical methods and to new methods introduced here . first , we show that yee s finite -difference time -domain ( fdtd ) scheme , along with a number of related methods , are multisymplectic and derive from a discrete lagrangian variational principle . second , we generalize the yee scheme to unstructured meshes , not just in space but in @xmath0-dimensional spacetime . this relaxes the need to take uniform time steps , or even to have a preferred time coordinate at all . finally , as an example of the type of methods that can be developed within this general framework , we introduce a new asynchronous variational integrator ( avi ) for solving maxwell s equations . these results are illustrated with some prototype simulations that show excellent energy and conservation behavior and lack of spurious modes , even for an irregular mesh with asynchronous time stepping .
Beam search (R-1: 37.82) the yee scheme is one of the most successful numerical methods used in the field of computational electromagnetics , particularly in the area of microwave problems . although it is not a high -order " method , it is still preferred for many applications because it preserves important structural features of maxwell s equations that other methods fail to capture . in this paper , we show that these desirable properties are direct consequences of the variational and discrete differential structure of the yee scheme , which mirrors the geometry of maxwell s equations . moreover , we show how to construct other variational methods that , as a result , share these same numerical properties , while at the same time applying to more general domains .
Global-aware (R-1: 45.96) the yee ( finite -difference time -domain ) scheme is one of the most successful numerical methods used in the field of computational electromagnetics , particularly in the area of microwave problems . in this paper , we show that these desirable properties are direct consequences of the variational and discrete differential structure of the yee scheme , which mirrors the geometry of maxwell s equations . moreover , we show how to construct other variational methods that , as a result , share these same numerical properties , while at the same time applying to more general domains . the key symmetry of maxwell s equations is not rotational or translational symmetry , as in mechanics , but a differential gauge symmetry . without taking additional care to preserve this gauge structure , even variational integrators can not be expected to capture the geometry of maxwell s equations . this differential / gauge structure also turns out to be important for the numerical performance of the method , and is one of the hallmarks of the yee scheme . this differential / gauge structure also turns out to be important for the numerical performance of the method , and is one of the hallmarks of the ye . . .

Reference
we present a new general mechanism to generate curvature perturbations after the end of the slow -roll phase of inflation . our model is based on the simple assumption that the potential driving inflation is characterized by an underlying global symmetry which is slightly broken .
Beam search (R-1: 36.88) most inflationary models assume that there is only one scalar field involved in the dynamics of inflation . in this work we point out a qualitatively new effect that might arise if one relaxes the assumption of a single dynamical field . in a multi -field scenario in which the inflationary potential is characterized by a broken symmetry , the quantum fluctuations generated during the inflationary stage represent fluctuations in the initial conditions for the dynamics of the inflaton in the subsequent stage , thus implying that the background dynamics after the slow -roll phase has ended will differ in different regions of the universe . since the background fields are coupled to the other fields into which they decay , the fluctuations generated during the slow -roll phase will affect the subsequent decay process . the present work , assuming that the inflaton decay into other fields through the non -perturbative process of preheating , is then aimed to understand whether isocurvature inflaton fluctuations , generated during the slow -roll stage , can lead to perturbations of the background metric through variations of the preheating efficiency . while the generation of curvature perturbations during the stages following the slow -roll phase has already been considered in some works , the . . .
Global-aware (R-1: 53.46) we show that in a multi -field scenario in which the inflationary potential is characterized by a broken symmetry , quantum fluctuations generated during the slow -roll stage of inflation may affect the efficiency of the subsequent preheating process . we obtain a general formula for the curvature perturbations generated from an inhomogeneous preheating efficiency related to the quantum fluctuations produced during inflation . the present work is then applied to the case of a broken @xmath0 symmetry .
Reference in this work we investigate an influence of decoherence effects on quantum states generated as a result of the crosskerr nonlinear interaction between two modes . for markovian losses ( both photon loss and dephasing ) , a region of parameters when losses still do not lead to destruction of non -classicality is identified . we emphasize the difference in impact of losses in the process of state generation as opposed to those occurring in propagation channel . we show moreover , that correlated losses in modern realizations of schemes of large cross -kerr nonlinearity might lead to enhancement of non -classicality .
Beam search (R-1: 47.48) decoherence due to losses is a main obstacle to implementations of schemes using kerr and cross -kerr nonlinearities . in our work we discuss an aspect of the decoherence which has been seldom discussed when considering an influence of losses on states generated via kerr nonlinearity . namely , we address losses arising in the process of generation and not due to propagation of the generated state via lossy channels . we concentrate our attention on a feature that might be quite significantly pronounced in modern schemes of generating large kerr nonlinearity : the modal loss can be strongly correlated . indeed , the modes occupy the same volume and interact with the same physical systems which form the reservoirs . also , if the kerr -nonlinearity scheme implies a sufficiently strong dispersive coupling of light modes to emitters , then coupling of these emitters to dissipative reservoirs might also appear to be quite strong . as a result , this would mean strongly correlated modal losses . on a number of examples we show how the correlated loss enhances and creates intermodal correlations and even entanglement , and might lead to generation of entangled states quite . . .
Global-aware (R-1: 41.50) decoherence of quantum states generated via kerr and cross -kerr nonlinearities is discussed . it is shown that modal loss can be strongly correlated , i .e . , it can be decoupled from uncorrelated reservoirs . this strongly correlated loss enhances and creates intermodal correlations and even entanglement , and might lead to generation of entangled states quite different from those generated in the same scheme without loss . the strongly correlated loss can result in the significantly enhanced robustness of the generation scheme . Reference -Joy Milne noticed a small difference in her husband just before he turned 40 . "His smell changed," the Scottish woman tells the BBC . "It wasnt all of a sudden . It was very subtle-a musky smell ." He was diagnosed with Parkinsonś disease six years later and died in June at age 65 . Milne found herself supporting the Parkinsonś UK charity, where she noticed others had the same distinct smell . She happened to mention her observation to scientists at a talk, and they later put her to the test, presenting her with T-shirts worn by six people with Parkinsonś and six without . "She got the six Parkinsonś, but then she was adamant one of thećontrolśubjects had Parkinsonś," says a scientist . Researchers were impressed and dubbed her a "super-smeller," per AFP . Then eight months later, that one control subject returned to tell scientists he had just been diagnosed with Parkinsonś . Because of Milneś sniffer, scientists now theorize . . .

Beam search
(R-1: 48.93) -Joy Milneś husband, Les, was diagnosed with Parkinsonś disease at age 45-six years before he died in June . "His smell changed and it seemed difficult to describe . It wasnt all of a sudden . It was very subtle-a musky smell," the Australian woman, who lives in Perth, tells the BBC . "I got an occasional smell ." After joining the charity Parkinsonś UK, she met people with the same odor, and researchers at Edinburgh University decided to test Milneś ability to detect the disease using t-shirts worn by six people with Parkinsonś and six without . "Her job was to tell us who had Parkinsonś and who didnt," says Dr . Tilo Kunath . "Her accuracy was 11 out of 12 . We were quite impressed ." Six of the Parkinsonś patients wore the shirts for a day, while the other six wore them for a week . Milne correctly identified six of the Parkinsonś patients and six of the non-Parkinsonś patients, but she was "adamant" that one of the non-Parkinsonś subjects had the disease, Kunath says . Eight months later, he told Kunath hed been diagnosed with Parkinsonś . . .
Global-aware (R-1: 55.18) -When Joy Milneś husband was diagnosed with Parkinsonś disease six years before his death in June, her sense of smell told her something was amiss . "His smell changed and it seemed difficult to describe . It wasnt all of a sudden . It was very subtle-a musky smell," she tells the BBC . After joining the charity Parkinsonś UK, she met people with the same odor . She mentioned it to scientists at the University of Edinburgh, and they decided to test her-and were "quite impressed," says one of the scientists . Milne correctly identified six people with Parkinsonś and six without it using T-shirts that had been worn by six people with Parkinsonś and six without it . Her accuracy was 11 out of 12 . "She got the . . .

G Global-aware Inference in Neural Machine Translation
Though the focus of this paper is Neural Abstractive Summarization, we will share how to enable global-aware inference to improve beam search significantly in NMT using a simple variation.
The gap between NMT and NAS is that NAS is a more-to-less generation task where the global attention values of many words are very low, which makes it relatively easy to trigger punishment at the beginning. As one can see in App. E.4 and the examples provided, many global-aware summaries are different from that of beam search from scratch, implying that the global scoring mechanism is activated in the very early stage. Since translation is a nearly one-to-one task, it is more likely to ignore some locality biases during the beginning stage when the local attention is generally low. Notably, although there is no very low global attention in translation tasks, their global attention distributions are still irrelevant with uniform distributions (here we only consider pre-trained translation models). To alleviate the gap, a straightforward-but-effective approach is to transform the one-to-one generation task to several more-to-less tasks. We divide the reference into several blocks of equal length and predict the global attention distribution of each block. Taking a reference of 25 tokens as an example, if we set the block length as 10, then there will be three blocks, namely the block of 1 − 10 tokens, 11 − 20 tokens, and the last 5 tokens. We use the special token b 1 to denote the first block, b 2 to denote the segment composed of the first and second block, and b 0 to denote the whole reference. When we train the attention-prediction model, these special tokens are concatenated before the source document to tell the predictor which segment it should predict. During inference, we first predict the global attention distribution of b 0 and calculate the optimal length, e.g. 32, then we can figure out there are three more global attention distributions that need be predicted, i.e., b 1 , b 2 and b 3 . We apply the predicted attention distribution of b 1 to guide the inference at the beginning, and the attention distribution is transformed to that of b 2 when the length of generated sequence is larger than 10 and so on. When the decoding step reaches 30, the attention distribution of b 0 is deployed until all candidate sequences are terminated. Although this blocking operation would increase the difficulty of training and inference, these negative effects are controllable by adjusting the block length.
We use mBART [23] as the neural translation model to examine the blocking operation on WMT16 en-ro sentence-level translation dataset, and related results are presented in Table 16. Compared with beam search, our proposed global-aware elevates the BELU score from 37.6 to 37.8, and ORACLE global-aware improves the result to 39.2. The blocking operation further improves global-aware to 38.1, and ORACLE global-aware to 41.6. We also experiment this operation on summarization tasks where minor improvement is observed.
Finally, since the blocking operation would inevitably increase the difficulty of training and inference and is not conducive to the robustness, it may not be the most reasonable direction for improving the global-aware inference in one-to-one tasks. We think there are three following issues that might be worth exploring in the future. 1) Whether the proposed global-aware inference has already been able to improve document-level translation significantly; 2) intuitively, we averaged the global attention distributions of all layers, but it is possible that the global attention in a single layer or a combination of several layers can better guide decoding; 3) whether a more comprehensive global protocol can be found to regulate beam search in a more detailed way.