{'fraction_non_alphanumeric': 0.03507082932196397, 'fraction_numerical': 0.016137165910237016, 'mean_word_length': 4.673211963589076, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Multi-Document Coverage Reward for RELAXed Multi-Document Summarization

Multi-document summarization (MDS) has made significant progress in recent years, in part facilitated by the availability of new, dedicated datasets and capacious language models. However, a standing limitation of these models is that they are trained against limited references and with plain maximum-likelihood objectives. As for many other generative tasks, reinforcement learning (RL) offers the potential to improve the training of MDS models; yet, it requires a carefully-designed reward that can ensure appropriate leverage of both the reference summaries and the input documents. For this reason, in this paper we propose fine-tuning an MDS baseline with a reward that balances a reference-based metric such as ROUGE with coverage of the input documents. To implement the approach, we utilize RELAX (Grathwohl et al., 2018), a contemporary gradient estimator which is both low-variance and unbiased, and we fine-tune the baseline in a few-shot style for both stability and computational efficiency. Experimental results over the Multi-News and WCEP MDS datasets show significant improvements of up to +0.95 pp average ROUGE score and +3.17 pp METEOR score over the baseline, and competitive results with the literature. In addition, they show that the coverage of the input documents is increased, and evenly across all documents.


Introduction
Multi-document summarization (MDS) aims to consolidate salient points of information across a set of documents into a concise summary. The main requirement for the summary is that it adequately represent the document set, with low redundancy and high coverage across all documents, while at the same time being readable and fluent. Combined with this, is the need to develop techniques that can handle the significant memory complexity required to tackle MDS. Recently, the re-lease of dedicated datasets (Fabbri et al., 2019;Gholipour Ghalandari et al., 2020), and intelligently designed Transformer models (Liu et al., 2018;Liu and Lapata, 2019;Beltagy et al., 2020), have helped drive advancements in multi-document summarization, generally improving the accuracy and fluency of the predicted summaries. However, aspects such as the requirement to cover as much salient information from the input documents as possible, whilst still maintaining low repetition and low redundancy, have certainly been less explored to date (Nayeem et al., 2018;Mao et al., 2020).
Within the sphere of contemporary neural MDS models, two main lines of investigation can be identified: graph-based approaches (Li et al., 2020;Pasunuru et al., 2021), and concatenation approaches (Liu et al., 2018;Zhang et al., 2020a). The former are approaches that rely on the construction of graphs to capture the inter-and intra-document relations. While powerful, they need to elicit the relations explicitly. The latter instead assume that all the input documents within a document set can be simply concatenated, possibly with document separators and tags, such that the relations can be "discovered" by the model. Like ordinary summarization, also MDS comes in two remarkably different styles: extractive, where the generated summaries consist of verbatim sentences from the original input documents (Nallapati et al., 2017), and abstractive, where the model is instead encouraged to generate a paraphrased understanding of the input documents. The intrinsic appeal of abstractive summaries and the advent of sequenceto-sequence models have increasingly shifted the trend toward abstractive summarization (See et al., 2017;Paulus et al., 2018;Fabbri et al., 2019;Lewis et al., 2020;Zhang et al., 2020a). As for what models are concerned, abstractive MDS has made increasing use of transformers, both "conventional" (Lewis et al., 2020;Zhang et al., 2020a) and modified to accommodate the characteristic input length of multi-document sets (Beltagy et al., 2020;Zaheer et al., 2020).
Similarly to general summarization, the majority of MDS models are trained using the negative log-likelihood (NLL) as training objective, which aims to maximize the conditional log-likelihood of the tokens of a given reference summary. Despite its speed and efficacy, the NLL exhibits both the wrong-objective problem (Ding and Soricut, 2017), where the model is trained on a convenient objective rather than a desirable one, and the wellknown exposure bias problem (Bengio et al., 2015;Ranzato et al., 2016). To alleviate these issues, reinforcement learning has been adopted in summarization, as in other language generation tasks, to train the model with a more appropriate objective (Li et al., 2019;Parnell et al., 2021). However, its effective use for MDS requires a reward function that can appropriately balance the reference summary and the multiple input documents in the document set. For this reason, in this paper we propose exploring a reward that combines a referencebased metric such as ROUGE with a coverage term over the input documents. To implement the reinforcement learning approach, we employ a contemporary gradient estimator of the policy gradient, RELAX (Grathwohl et al., 2018), which is both low-variance and unbiased. In addition, to limit the computation and the risk of parameter drift, we apply the objective to fine-tune an NLL-pretrained model in a few-shot manner. In light of the above, this paper makes the following contributions: 1. a reward for reinforcement learning that combines a ROUGE score and a multi-document coverage score, to simultaneously adhere to both the reference summaries and the input documents; 2. a reinforcement learning implementation that leverages a low-variance and unbiased gradient estimator of the policy gradient, RELAX; 3. experimental results and a comprehensive analysis over two MDS datasets (Multi-News and WCEP), showing the empirical effectiveness of the proposed approach.
The rest of this paper is organized as follows: first the related work is reviewed in Section 2, and then the proposed approach is introduced in Section 3. Section 4 describes the experimental set-up and main results, while Section 5 presents a more detailed analysis of the main components of the proposed approach. Eventually, Section 6 summarizes our findings and concludes the paper.

Related Work
Early work in multi-document summarization (MDS) that pre-dates the neural era (Mani and Bloedorn, 1997;Erkan and Radev, 2004;Christensen et al., 2013) was shaped around the notion of MDS as a collection of graph structures. As approaches in language generation naturally evolved into neural-based (Rush et al., 2015;Ranzato et al., 2016), later improved with the emergence of large, pre-trained language models (Devlin et al., 2019;Lewis et al., 2020;Zhang et al., 2020a), the effort shifted to integrating these graph structures into the models, often building on top of strong single-document summarization (SDS) baselines (Lebanoff et al., 2018;Zhang et al., 2018).
Concurrently, the growing interest in multidocument summarization has led to the development of dedicated, multi-document datasets such as WikiSum (Liu et al., 2018), Multi-News (Fabbri et al., 2019), Wikipedia Current Events Portal (WCEP) (Gholipour Ghalandari et al., 2020) and others. The typical amount of input data that comes with these datasets has increased the pressure on the models to be able to handle larger inputs. For instance, WCEP has up to 100 documents in each document set, and 63.7 on average. As such, the standard transformers used to develop successful SDS models such as BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020a) have proved inadequate for MDS due to their limited maximum input length (in the order of 10 3 tokens) and quadratic memory complexity (Beltagy et al., 2020). In turn, this has prompted the development of long transformer models such as Longformer (Beltagy et al., 2020) (built upon BART) and BigBird (Zaheer et al., 2020) (built upon PEGASUS) which, thanks to their smart attention layers that scale linearly with the input length, have opened up the possibility of presenting the input documents "at once", allowing these re-designed attention mechanisms to discover both inter-and intra-document relations.
Document summarization, as have other language generation tasks, has often been criticized for using maximum-likelihood training objectives that may prove limitative for the eventual performance of the models (Ding and Soricut, 2017). For this reason, reinforcement learning has been employed as an alternative, to directly optimize the models over evaluation metrics and explicitly reward the quality of the model's predictions. Reinforcement learning approaches have used metrics such as ROUGE-1, ROUGE-2 and ROUGE-L F1 (Paulus et al., 2018), and also more contemporary scoring functions such as BERTScore (Zhang et al., 2020b) as rewards, often mixed with maximum-likelihood objectives. When applying reinforcement learning to MDS, we contend that the reward should not simply be a ROUGE score against the reference summary, since this would dismiss key characteristics of the task such as inter-document information transfer. For instance, Mao et al. (2020) have leveraged maximal marginal relevance (Carbonell and Goldstein, 1998) to mollify higher-order information redundancy between the input documents. Several other performance measures could potentially be included in the reward, such as extractive fragment coverage and density (Grusky et al., 2018) and MINT (Dreyer et al., 2021), but to the best of our knowledge they have never been utilized as, or for, training objectives.
To address this gap, in this paper we propose leveraging a modified coverage reward to improve information coverage across all the documents in the input set, jointly with a principled policy gradient estimator (RELAX) and a performing long transformer model (the BART Longformer Encoder-Decoder, or BART-LED), in the hope of benefiting from the synergy between these components.

Proposed Approach
In this section, we present the details of the proposed approach, including the reinforcement learning framework (Section 3.1), the multi-document coverage reward (Section 3.2), and the overall training objective (Section 3.3).

Reinforcement Learning Gradient Estimators
Given a set of documents in input, simply noted as x, and a summary with T tokens, y = {y 1 , . . . , y T }, the predictive distribution, also known as policy in reinforcement learning, can be noted as p(y t |y 1 , . . . , y t−1 , x). The policy gradient theorem (Sutton et al., 1999) states that an estimator for the gradient of the reinforcement learning risk can be expressed as: where y s 1 , . . . , y s T is a sequence sampled from the policy, r is a function that rewards its quality, and θ collectively denotes all the policy's parameters. This estimator is the well-known REINFORCE (Williams, 1992) and is a baseline of reinforcement learning. At its turn, the gradient can be easily turned into a loss function to be used with automatic differentiation: The sampled sequence in (2), y s = {y s 1 , . . . , y s T }, can be obtained with any usual sampling approach such as teacher-forcing, student-forcing, or scheduled sampling (Bengio et al., 2015). While the samples can be drawn from a standard categorical distribution, in our experiments we utilize the Gumbel-Softmax re-parameterization (Jang et al., 2017) to obtain the categorical samples from transformed samples of a Gumbel distribution. The reason for the re-parameterization is that the Gumbel-Softmax samples are needed for the RELAX estimator that we introduce in the following. For a generic sample, y s t , the re-parameterization can be concisely expressed as: where z t is a Gumbel-Softmax sample of size equal to that of the vocabulary that acts as a "soft" prediction, p t is the probability vector over the vocabulary at slot t, τ is a temperature parameter controlling the sparsity of z t , and argmax(z t ) returns the index of z t 's largest value. This reparameterization is provenly equivalent to directly sampling y s t from Cat(p t ) (the reader can refer to Jang et al. (2017) for details).
REINFORCE is an unbiased estimator of the theoretical gradient, but it typically suffers from a high variance which can affect the convergence and effectiveness of training. To curb its high variance, techniques based on control variates and the subtraction of simple baselines have been proposed and even applied to summarization (Rennie et al., 2017;Paulus et al., 2018). However, our early experiments showed that these approaches were not promising for the given task. In addition, some of these estimators introduce a "bias", i.e. a mean difference with respect to the theoretical gradient. More recently, the RELAX gradient estimator has been shown to empirically outperform REIN-FORCE, thanks to its ability to reduce the variance while remaining unbiased (Grathwohl et al., 2018). The corresponding RELAX loss can be expressed as: (4) In (4), c φ (z) is a control variate of parameters φ which is expected to correlate tightly with the reward to reduce the variance, and term c φ (z)−c φ (z) ensures that the overall gradient remains an unbiased estimator of the theoretical gradient. Variable z = {z 1 , . . . , z T } denotes the sequence of the Gumbel-Softmax samples, while variablez denotes the sequence of samples from a Gumbel-Softmax distribution conditioned on the observed values of y s . Operationally, z t is sampled first, unconditionally, then y s t is derived with the argmax, and finallyz t is sampled from a suitably conditioned Gumbel-Softmax distribution; details can be found in Grathwohl et al. (2018), Appendix B -Categorical. Overall, the RELAX estimator is both unbiased and low-variance.
The control variate in our experiments is a simple two-layer feed-forward network that is constructed to correlate with the ROUGE scoring function. We obtain this by feeding the concatenation of the soft predictions, z (or, in turn,z), and the reference summary, y, as input to the control variate. This allows the model to learn to score the soft predictions and their targets in a way that mimics the ROUGE prediction-reference score. In detail, the architecture consists of two fully-connected linear layers, each followed by a ReLU linear activation function, and a final sigmoid activation function that normalizes the output of the last layer. Eventually, the output of the sigmoid is averaged to produce the control variate. 1

Multi-Document Coverage Reward
The design of an effective reward is another key aspect of a reinforcement learning objective. In our work, we have aimed to design an overall reward that could simultaneously remain faithful to: a) the reference summary, to ensure adequate generation performance, and b) the input documents, to cover as many important details as possible, and hopefully, support generalization. Relying solely on the reference summaries, given the large input size, does not seem to promise sufficient guidance, and our experiments have confirmed that. To implement the reward, we have chosen to use ROUGE-L F1 for the references and a multi-document coverage score for the input documents that we describe hereafter.
Several quantitative measures of coverage exist in the literature, and have found ample use in describing the properties of summarization datasets and the performance of models. For our work, we have adopted the extractive fragment coverage (EFC) of Grusky et al. (2018). The EFC measures the percentage of words in a summary that are part of "extractive fragments" within an input document, which are simply multi-word phrases shared between the input document and the summary. It is a simple precision-type measurement that looks at how much of the prediction is in the input document. Noting an individual document as D, a summary as y and an extractive fragment as f , the EFC can be expressed as: where the | · | operator is used to denote length. To promote an even improvement in coverage across the input documents, we propose a multi-document extension of the EFC that reaches its highest value when the coverage across the input documents is evenly distributed. Let us note the input document set here as D, and the EFC coverage vector over the document set as cov(y, D). We also note the sample mean of a vector x as µ(x), the sample standard deviation as σ(x), and their ratio (the inverse coefficient of variation) as c −1 v (x). This allows us to compute a "normalized" coverage score for a summary, c −1 v (cov(y, D)), which takes larger values the more the scores are uniform across the document set. In addition, inspired by Kryściński et al. (2018), we define a reward that pits the normalized coverage score of the prediction, y s , against that of the reference, y: Eventually, to ensure that short summaries are not unfairly rewarded with high coverage scores, we normalize the reward by the length ratio of the prediction and the reference: Overall, ther cov reward regards a prediction as "good" if it enjoys high average coverage of the input documents, the coverage is evenly distributed, and the prediction is of sufficient length. The reference summary acts as a baseline, making the reward additive if the prediction outperforms the reference, and subtractive if otherwise.
Since ROUGE-L F1 and the coverage reward are not necessarily up to scale, to obtain the final reward, r, we perform a convex combination with a scaling coefficient, β:

Overall Training Objective
As training strategy, we first train the model with the negative log-likelihood and choose the best model with a criterion based on the validation performance. After that, the model is fine-tuned with the reinforcement learning objective. In many past works, the reinforcement learning objective has been used mixed with the NLL for stability (Paulus et al., 2018;Li et al., 2019;Parnell et al., 2021). However, we assume that the model has already "warmed up" to the training data during its NLL pretraining stage, and only use either L REIN F ORCE (2) or L RELAX (4) for fine-tuning. To prevent excessive drifting from the NLL pre-trained model, we limit the fine-tuning to a few (≈ 1, 000) shots and a relatively low learning rate (3 × 10 −6 ).

Datasets
We For WCEP, we specifically use the WCEP-100 version, which exclusively limits the number of articles within a document set to 100. We have chosen these datasets as they cover an ample spread of summary lengths and numbers of input documents, with Multi-News having longer reference summaries on average. Appendix A.2 reports the datasets' main statistics as presented in the original papers (Fabbri et al., 2019; Gholipour Ghalandari et al., 2020) 2 .

Evaluation Metrics
Like most previous works, we use the F1 variants of the ROUGE-N scores 3 (Lin, 2004) for performance evaluation. In our use of ROUGE, we choose not to stem the predictions and the references during scoring. Since we use the ROUGE-L F1 score in our reward, to avoid circularity we also include METEOR 4 (Lavie and Agarwal, 2007) in the performance evaluation. Differently from our ROUGE implementation, METEOR uses stemming, synonyms, and other paraphrastic matching in the n-gram matching stage. In a recent study, both ROUGE and METEOR have displayed high correlation with a number of desirable summarization properties such as coherence, consistency, fluency, and relevance (Fabbri et al., 2021).

Main Settings
We have implemented our approach on top of BART-LED (Beltagy et al., 2020). We utilize the generous maximum encoding length (16384 tokens) of this long-input transformer, by concatenating all the documents in a document set to form a single input to the model. The individual documents are separated by an [END] token, and the input is truncated to the maximum length. For every experiment, we report the average of three independently-initialized training runs. For each result, we have also run a nonparametric bootstrap test for statistical significance, and highlighted the results that are significantly different from the baseline. In the reward, the β hyperparameter has been set to 1.0 with a validation described in Appendix A.3. All other hyperparameters are described in Appendix A.1.

Results
Multi-News. Table 1 compares the results over the Multi-News test set for the baseline, our proposed approaches and previous work from the literature. We first note that our BART-LED model has performed as a strong baseline, with its results being comparable to those of BART-Long (Pasunuru et al., 2021), which is based on the same BART Longformer architecture. In detail, BART-Long has reported a higher ROUGE-1 score, our baseline has reported a higher ROUGE-L score, and both have reported similar ROUGE-2 scores. Therefore, we regard our performance as comparable on the whole, with the differences most likely due to different hyperparameters. Amongst our results, the models fine-tuned with REINFORCE have achieved worse results than the baseline. This is evidence that a vanilla implementation of the policy gradient is not necessarily better than a standard NLL objective. Conversely, the models fine-tuned with RELAX have surpassed both the NLL baseline and virtually all the previous work. The best results have been achieved with the inclusion of the coverage term, with an improvement of +0.36 ROUGE-2 pp over the NLL baseline and a marked improvement of +0.92 ME-TEOR pp. In addition, both results have reported a p-value < 0.01. These results give evidence to both the improved performance provided by the RELAX gradient estimator and the usefulness of the coverage term. In Appendix B, we also provide a qualitative example which shows that the increase in METEOR score is most likely given by the positive impact of the coverage term, which has allowed the model to retrieve relevant phrases from the input documents.
WCEP. Table 2 shows the results over the WCEP test set. The trend is similar to that over Multi-News, but the improvements with the proposed models have been even more pronounced. In the first place, the NLL baseline has set a very strong performance compared to the previous work, showing the full potential of a long-input model such as the Longformer for MDS. As for Multi-News, the best results have been achieved with the RELAX gradient estimator, with improvements of up to +1.32 ROUGE-1 pp and +3.17 METEOR pp over the NLL baseline. The inclusion of the coverage term with RELAX has not been able to increase the ROUGE scores, but has increased ME-TEOR by +1.64 pp. Again, we attribute this to the model's improved coverage of the input documents, which leads to an increased number of matches under METEOR's more relaxed matching scheme. A qualitative example is discussed in Appendix B.

Analysis
In this section, we present a more detailed analysis of the impact of the coverage term, the few-shot fine-tuning, and the RELAX gradient estimator using the Multi-News validation set as reference. For a further insight into the coverage reward, we also include an analysis of its trajectory during training. All the selected hyperparameters are listed in Appendix A.1.

Impact of the Coverage Term
Our rationale for including a coverage term in the reward is to ensure coverage of the input documents beyond what can be driven by the reference summaries alone. We note that this may or may not translate into an improvement of the evaluation metrics, but it seems to add intrinsic value to the summaries nevertheless. For this reason, we further analyze the impact of the coverage term hereafter. Figure 1 shows the average EFC coverage (5) for the documents in the input sets, indexed by the document position in the set (first, second etc). The figure shows that the inclusion of the coverage term with RELAX has led to a marked increase of the coverage, and almost evenly distributed across all the documents in the input set. In particular, the document in the last position has achieved the largest coverage improvement.
In turn, Figure 2 shows the average ROUGE score for the documents in the input sets, obtained by averaging the ROUGE-1, ROUGE-2, and ROUGE-L scores computed between the predicted summary and the document (NB: not the reference summary). The figure shows that the improvements in ROUGE score across the document set are similar to those in EFC coverage, rather evenly distributed, and with an improvement of over +4 pp for the document in the last position. This is further evidence that the normalized coverage reward (7) is able to drive the model towards predictions that cover the input set more uniformly.

Few-Shot Fine-Tuning
To explore the behavior of the few-shot fine-tuning, we compare the validation-set performance on Multi-News with varying number of training examples, from 10 to 2000. The model's configuration is the best, with RELAX and the coverage term in the reward. Table 3 shows that the performance is the highest with 1000 examples, and starts to drop beyond this number. This is an important observation, as it shows that the reinforcement learning objective may lead to undesirable parameterizations beyond a point, and that the number of fine-tuning samples has to be treated as a hyperparameter.

Configuring RELAX
The RELAX gradient estimator introduces two new hyperparameters: the temperature parameter, τ , and the control variate, c φ . Hereafter, we discuss their impact and design.
Temperature parameter. The RELAX gradient estimator uses a temperature parameter, τ , in the Gumbel-Softmax sampling (3). This parameter is maintained in log scale for convenience and is learnable alongside all other parameters; yet, its initial value can have a significant impact on the final model. To explore its behavior, Figure 3 shows the trajectory of parameter log τ over 1000 Multi-News training steps for different initializations (0.25, 0.5 and 1.0). The trajectories show that, irrespective of its initial value, log τ converges to a stable value within approximately 400 training steps. For the initializations at 0.25 and 1.0, within the first 200-300 training steps log τ drifts significantly (≈ ±0.25 units) from its initial value. Conversely, with the intermediate initialization at 0.5, the value remains substantially stable over the whole trajectory. Since limiting drift during finetuning is generally desirable, we have initialized log τ to 0.5 in all experiments.
Control variate size. Many different architectures could be used for the control variate, but given our choice described in Section 3.1, the main parameter is the feed-forward layers' hidden size. To explore its impact, Table 4 shows the average values of the ROUGE score and the coverage score over the Multi-News validation set with different hidden sizes (128, 256, and 512). The ROUGE score is computed between the prediction and the reference and is the average of ROUGE-1/2/L, while the coverage score is the average EFC of all the input documents. The values in Table 4 show that, the larger the control variate, the more the model is able to increase the coverage score. However, the average ROUGE score drops beyond a size of 256. We speculate that this behavior is due to the larger scale of the coverage reward, as by providing more capacity to the network, we allow the control variate to increasingly correlate with the multi-document coverage reward rather than the ROUGE reward. To strike a satisfactory trade-off, we have therefore chosen 256 as the hidden size for all experiments with Multi-News, and carried out an equivalent selection for WCEP.

Coverage Reward Trajectory
In a reinforcement learning framework, it could be useful to monitor the value of the reward over the training steps. Typically, the reward should exhibit an upward trajectory, since the reward should tend to increase as the model learns to make better predictions. In our case, we look to explore the impact of our coverage reward on the coverage distribution over the input documents. In particular, we want to verify whether the coverage reward is able to promote predictions that cover the input documents more evenly, which should translate into a decreased standard deviation. To this aim, Figure 4 shows a plot of the standard deviation of the cov-erage scores (EFC) across the input document set against the training step. The trajectories show that both REINFORCE and RELAX have been able to decrease the standard deviation of the predictions to approximately 0.05 units from initial values of 0.08 − 0.09. The drop in standard deviation occurs quite quickly during training, coinciding with the improvement in the reward value of the predictions.
Comparing REINFORCE with RELAX also shows that RELAX has been able to achieve lower standard deviation values throughout the training, with the exception of the very start.

Conclusion
In this paper, we have proposed fine-tuning a multidocument summarization model with a reward that balances the use of the reference summaries with the coverage of the input documents within a reinforcement learning framework. The rationale for the proposed reward is that the reference summaries alone may not be sufficient for an effective fine-tuning of the model in the presence of very large inputs such as those typical of MDS datasets.
Another key component of the proposed approach is the use of a modern gradient estimator of the policy gradient, RELAX. The experimental results over two news-based MDS datasets, Multi-News and WCEP, have shown that the proposed approach has been able to achieve a marked improvement of ROUGE and METEOR scores compared to its NLL-pretrained baseline, and prove competitive against most existing approaches. In addition, the proposed approach has been able to increase the coverage of the input documents, and evenly across the entire document set. As future work, we aim to explore ways to prevent or mollify the model's drift with larger number of training steps, and explore alternative architectures and configurations for the control variate of the RELAX estimator.

A.1 Model Hyperparameters
Our baseline model is the BART Longformer encoder-decoder (BART-LED) of Beltagy et al. (2020). In all experiments, it has been first pretrained over the training set using the negative loglikelihood until convergence, which has typically set within 5 epochs, in line with what reported in Pasunuru et al. (2021). The best models on the validation set (average ROUGE) have then been finetuned with the reinforcement learning objectives in (2) and (4). BART-LED has 459M parameters, and the addition of the control variate for the RE-LAX experiments adds approximately 12M more parameters. We have used the Adam optimizer for training both the main BART-LED model and the parameters of the control variate of the RELAX gradient estimator. The learning rate of the main optimizer for the pre-training of the baseline has been set to the default of 3 × 10 −5 (Beltagy et al., 2020), before being reduced to 3 × 10 −6 for finetuning with the reinforcement learning approaches. For training the control variate, we have set an initial learning rate of 1 × 10 −2 and initialized the learnable log τ parameter with a value of 0.5. The optimal size of the hidden layers of the control variate appears to empirically correlate with the maximum length of the reference summaries in the dataset. For Multi-News, we have set the size to be 256, and for WCEP to 40 5 . For the multi-document coverage reward, we have used a β value of 1.0. The entire model has been implemented on top of PyTorch Lightning 6 . Please refer to Table 5 for a full list of the hyperparameters. For all experiments, we have used an NVIDIA Quadro RTX 6000 with 24 GB of memory.

Hyperparameter
Multi-News WCEP Learning Rate (Train) 3 × 10 −5 3 × 10 −5 Learning Rate (Tune) 3 × 10 −6 3 × 10 −6 Learning Rate (RELAX) 1 × 10 −2 1 × 10 −2 log(τ ) (RELAX) 0.5 0.  WCEP. Accessible from the following repository: https://github.com/ complementizer/wcep-mds-dataset. Table 6 shows the datasets' main statistics, including the number of samples within each split and the average length in tokens of the reference summaries. For our model, we have used the average length of the reference summaries as a guidance for choosing the maximum output length and the hidden size of the control variate.

DS
Train Test Dev Avg. Tokens M-N 44.9K 5.6K 5.6K 263.7 WCEP 8.1K 1.0K 1.0K 33.3 Table 6: Main statistics of the datasets used in the experiments. Multi-News has up to 10 individual articles in each document set, while WCEP has up to 100. The document split sizes have been rounded.

A.3 Scale of the Coverage Reward
The multi-document reward is used in (8) in convex combination with the ROUGE-L F1 score, and an appropriate value of the mixing coefficient, β, needs to be explored. To this aim, Table 7 shows the values of the average ROUGE and coverage scores over the Multi-News validation set. The coverage has not increased monotonically with the increase of the β coefficient. In turn, the ROUGE score has reached a maximum for β = 1.0. As a trade-off, we have set β = 1.0 in all experiments.

B Qualitative Analysis
Tables 8 through 11 present two qualitative examples, one per dataset, where we specifically compare our RELAX implementation with and without the use of the coverage term in the reward. Key points are highlighted in various colors, comments are addressed in the captions, and the ROUGE and METEOR scores are reported for each prediction. Document sets longer than a full page have been truncated to fit.

Source Document
Plenty of churches contain relics of saints, but not many of those relics were found in excavations from sixth-century churches.
Archaeologists at a medieval fortress site in Burgas, Bulgaria, found a lead vessel, which contains some of the ashes from the alleged grave of John the Apostle, in a reliquary that dates to the sixth century C.E. The reliquary, which was once part of an early Christian basilica, is named for Saint John the Theologian, who is considered one of Jesus' apostles. The vessel, which is less than an inch long, is decorated with crosses. Milen Nikolov, director of the Burgas Regional Museum of History, said that early Christians would have believed the relic had healing properties. John the Apostle's grave in Turkey was also a pilgrimage site for early Christians seeking healing, Ancient Origins reports. Nikolov said the reliquary was "one of the most important discoveries" in the museum's history. In addition to the relic, the archaeologists also uncovered a 10th century Bulgarian royal seal at the fortress site. Meghan DeMaria [END] Ashes from the grave of John the Apostle, one of the Twelve Apostles of Jesus Christ, have been discovered in a lead tube reliquary by Bulgarian archaeologists during excavations of the ancient and medieval port of Burgos (also known as Poros) on Cape Foros in today's Black Sea city of Burgas. The discovery of the lead tube containing ashes from the grave of John the Apostle, who is known as St. John the Theologian in Bulgarian (Eastern) Orthodox Christianity, located in the ancient city of Ephesus in Anatolia, today's Turkey, has been made during the 2014 excavations of the fortress of Burgos (or Poros) on Cape Foros in Burgas but was announced only on Wednesday, March 25, 2015, by Milen Nikolov, Director of the Burgas Regional Museum of History, at a special press conference. He has also announced other intriguing finds such as the discovery of a Late Antiquity latrine, also found at Burgos (Poros), and the discovery of a 10th century Bulgarian royal seal from the Rusocastro Fortress. The structures at the ancient and medieval fortress and port of Burgos (Poros) which were excavated in 2014 include an Early Christian basilica from the 6th century AD, a building complex from the 5th-6th century AD, and a Roman villa from the 3rd century AD. The John the Apostle reliquary was found in the 6th century basilica. "Probably a pilgrim from the Foros Peninsula (Cape) went on a pilgrimage to Ephesus, and came back here with this relic which was then donated to the basilica on Foros," Nikolov has explained, as cited by local news site Gramofona. Nikolov has described the finding of the reliquary as "one of the most important discoveries in the history of the [Burgas Regional History] Museum", and the lead tube as a "holy possession that preserved a holy substance" having to do with the beliefs that every year on May 8, the date of John the Apostle's death, there is manna, a holy curing powder, on the site of his grave. The lead tube reliquary itself containing the ashes from the grave of John the Apostle (St. John the Theologian) is really tiny: it is only 2.2 cm (less than an inch) long, and its diameter measures 1.7 cm. The reliquary is dated to the 6th century AD when pilgrimage to the Holy Lands was very common among Christians, Nikolov explains. On one of its sides there is an image of a cross with equal arms inside a medallion, and on the opposite side there is an image of two overlapping crosses with equal arms.  The comparison of summaries is in the following Table 9. The input has been truncated to fit the page.

Reference
Archaeologists in Bulgaria think they've found ashes belonging to one of the 12 apostles. The team found a small lead container with ashes while excavating a basilica that dates back to the sixth century AD, reports Archaeology in Bulgaria. The markings on the inch-long vessel match those at the grave of St. John the Apostle in what is now Turkey, leading researchers to surmise that a long-ago Christian from Bulgaria traveled to the grave site and returned home with the sample of ashes. Such journeys were common around that time, reports Novinite. "Probably a pilgrim from the Foros Peninsula went on a pilgrimage to Ephesus, and came back here with this relic, which was then donated to the basilica on Foros," says the director of the Burgas Regional Museum of History in Bulgaria. (The ancient basilica is located in what is now the Black Sea city of Burgas.) Archeologists also found a Bulgarian royal seal from the 10th century at the dig site, which includes a fortress, notes the Week. (Another archaeological discovery: an ancient underground city in Turkey.) NLL (47.05/16.12/24.59/28.10) Archaeologists have made a pretty significant find in Bulgaria: ashes from the grave of one of Jesus Christ's Twelve Apostles. The lead tube containing the ashes of St. John the Theologian, who is believed to have lived in what is now Turkey, was found in a medieval fortress and is named for the early Christian basilica that once stood there, Ancient Origins reports. The lead container, which is less than an inch long, is decorated with crosses and has only one handle, reports the International Business Times. The lead container is believed to have been carried by a pilgrim who made the trip to the ancient city of Ephesus, where the Apostle was once a favorite of Jesus. The grave of John the Theologian was a popular pilgrimage destination for early Christians seeking healing, and the lead container "must be connected with the cult for St. John the Theologian," says the head of the Burgas Regional Museum of History. The find was announced Wednesday, but it wasn't made public until Thursday.
(Click to read about how a lead container found in a 14th-century tomb could help heal ancient wounds.) ROUGE-L + RELAX (56.76/25.06/30.76/37.10) Archaeologists have made "one of the most important discoveries" in the history of the Burgas Regional Museum of History in Bulgaria, says the museum's director: a lead reliquary that contains ashes from the alleged grave of one of Jesus' Twelve Apostles. The tiny lead container, which is less than an inch long, was found during last year's excavations of the fortress of Burgos on Cape Foros, reports Ancient Origins. The reliquary was once part of an early Christian basilica, and "probably a pilgrim from the Foros Peninsula went on a pilgrimage to Ephesus, and came back here with this relic which was then donated to the basilica," says Milen Nikolov. The find was announced Wednesday, but the news was made public on Wednesday.
The find was made during the same excavations that turned up a 10th-century royal seal, reports the Independent. The find is significant because early Christians believed the relic had healing properties, and John the Apostle's grave in Turkey was a popular pilgrimage site for those seeking healing, reports Ancient Origins. (Click to read about how a lead-tipped glass bottle was found in a 14th-century Roman city.) ROUGE-L + Coverage + RELAX (57.59/30.00/31.93/38.64) Archaeologists have made "one of the most important discoveries" in the history of the Burgas Regional Museum of History in Bulgaria, says the museum's director: a lead reliquary that contains ashes from the alleged grave of one of Jesus Christ's apostles. The tiny lead container, which is less than an inch long, was found in a 6th-century basilica in what is now the Black Sea city of Burgas, reports Ancient Origins. "Probably a pilgrim from the Foros Peninsula went on a pilgrimage to Ephesus, and came back here with this relic which was then donated to the basilica," says Milen Nikolov, per the Independent. The reliquary is named for St. John the Theologian, who is believed to have been one of the Twelve Apostles. The find was made during 2014 excavations at the fortress of Burgos on Cape Foros, but the announcement was made only this week. The head of the museum says the reliquary is connected to the belief that there is manna, a holy curing powder, on the site of John the Apostle's grave every year. (Click to read about how a lead box found in a cave has been analyzed for ancient DNA.) Table 9: Multi-News example. Comparison of reference, NLL baseline, and RELAX-generated summaries for the document in Table 8. We compare specifically the addition of the coverage term in the reward, to qualitatively show its importance. The R1/R2/RL/METEOR scores are shown in the headers. Highlighted in blue are examples of key information that allow for the summary to remain faithful to the reference. Highlighted in green are examples where the coverage term has managed to improve the quality of the summary. Highlighted in red are examples where the model has conveyed incorrect statements with respect to the input documents, and where the subsequent use of the coverage has seemingly improved it. We note that these results are also in line with the average scores presented in Table 1.  Table 11. The input has been truncated to fit the page.  Table 10. We compare specifically the addition of the coverage term in the reward, to qualitatively show its importance. The R1/R2/RL/METEOR scores are shown in the headers. Highlighted in blue are examples of key information that allow for the summary to remain faithful to the reference. Highlighted in green are examples where the coverage term has managed to improve the quality of the summary. As mentioned in Section 4.4, shorter summaries are involved in this dataset, and are more likely to result in higher ROUGE scores. In this example, both RELAX objectives have drastically improved the accuracy. We can also see that the model has been able to use the coverage term to improve the summary quality by adding relevant fragments, and lead to a higher METEOR score. We note that these results are in line with the average scores presented in Table 2.