{'fraction_non_alphanumeric': 0.030515696472210037, 'fraction_numerical': 0.022557804373707625, 'mean_word_length': 5.034026465028355, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PoinT-5: Pointer Network and T-5 based Financial Narrative Summarisation

Companies provide annual reports to their shareholders at the end of the financial year that de-scribes their operations and financial conditions. The average length of these reports is 80, andit may extend up to 250 pages long. In this paper, we propose our methodology PoinT-5 (thecombination of Pointer Network and T-5 (Test-to-text transfer Transformer) algorithms) that weused in the Financial Narrative Summarisation (FNS) 2020 task. The proposed method usesPointer networks to extract important narrative sentences from the report, and then T-5 is used toparaphrase extracted sentences into a concise yet informative sentence. We evaluate our methodusing Rouge-N (1,2), L, and SU4. The proposed method achieves the highest precision scores inall the metrics and highest F1 scores in three out of four evaluation metrics that are Rouge 1, 2,and LCS and only solution to cross MUSE solution baseline in Rouge-LCS metrics.


Introduction
Annual Reports may extend up to 250 pages long as stated above, which contains different sections General Corporate Information, financial and operating cost, CEOs message, Narrative texts, accounting policies, Financial statement including balance sheet and summary of financial data documents. In the Financial narrative summarisation task, only the narrative section is summarised, which is not explicitly marked in the dataset, making it challenging and interesting. In recent years, previous manual smallscale research in the Accounting and Finance literature has been scaled up with the aid of NLP and ML methods, for example, to examine approaches to retrieving structured content from financial reports, and to study the causes and consequences of corporate disclosure and financial reporting outcomes (El-Haj et al., 2018).
Companies produce glossy brochures of annual reports with a much looser structure, and this makes automatic summarisation of narratives in UK annual reports a challenging task (El-Haj et al., 2020). Hence we summarize the narrative section of annual reports, particular narrative sentences that are spread loosely across the document need to be first identified and summarise those sentences. The summarisation limit is set to 1000 words, where the actual length of the report may go up to 250 pages long. Hence to summarize these long annual reports using a combination of extractive and abstractive summarisation.
The text summary method can be classified into two paradigms: extractive and abstractive. The extractive summarisation method extracts the meaningful sentences or a section of text from the original text and combines them (ranked or unranked) to form a summary (Cheng and Lapata, 2016;Narayan et al., 2018;Yasunaga et al., 2017;See et al., 2017). Whereas abstractive summarisation generates words and sentences that are similar in meaning to the given text to form a summary that may not be in actual text (Nallapati et al., 2016;Rush et al., 2015;Paulus et al., 2017;Li et al., 2018). When summarizing long documents such as in our case up to 250 pages long, extractive summarisation may not produce a coherent and readable summary, and abstractive summarisation cannot cover complete information using encoder-decoder architecture. One problem is that typical seq2seq frameworks often generate unnatural summaries consisting of repeated words or phrases (Li et al., 2018). Hence, we come up with a combination of extractive and abstractive summarisation to first select important narrative sentences and concisely convey them.
Pointer Networks (Vinyals et al., 2015) is used in various combinatorial optimization problems, such as Travelling Salesman Problem (TSP), Convex hull optimization. We used pointer networks in our task of financial narrative summarization to extract relevant narrative sentences in a particular order to have a logical flow in summary. These extracted sentences are paraphrased to summarise these sentences in an abstractive way using the T-5 sequence-to-sequence model. We train the complete model by optimizing the ROUGE-LCS evaluation metric through a reinforcement learning objective.

Related Works
In this section and Appendix B, we discuss related works in the fields of abstractive summarisation, extractive summarisation, combinations of these two methods, reinforcement learning applications, and summarisation of Financial Narratives and their methodology. Studies of human summarizers show that it is common to apply various operations while condensing, such as paraphrasing, generalization, sentence-level summarisation and reordering (Jing, 2002). We continue related work in Appendix B.

Data Description
The financial narrative summarisation dataset contains 3,863 annual reports for firms listed on LSE covering the period between 2002 and 2017 (El-Haj et al., 2019;El-Haj et al., 2014). Dataset is randomly split into training (75%), testing and validation (25%) Table:2. We use NLTK sentence tokenizer 1 to tokenize sentence in the annual report and summary processing for all our experiments. Data is further described and anayzed in Appendix A.

Model Description
Our model is composed of three parts that are first trained or executed individually and then finally brought together using policy gradient algorithm. As stated earlier the model is combination of extractive as well as abtractive methods. Initially dataset is provided in the form {x i , y j i }, 1 <= i <= N, 1 <= j <= 7. Here i represents number of annual reports, j represents number of summaries for each annual reports and x, y represent report and summary respectively.
In extraction process we assume that for every summary sentence there is matching sentence in the annual report. To train exaction model we need these corresponding sentences in the reports. Since, annual reports are not marked explicitly with sentences we followed ROUGE scores to extract these sentences as done in (Chen and Bansal, 2018;Nallapati et al., 2016).For every summary sentence we calculate ROUGE with every sentence in the report and then choose the sentence with maximum value.
In equation 1 j t represent sentence with maximum ROUGE score for summary sentence s t for every sentence in report d i . For every report in training set there are multiple summaries. We calculate ROU GE − Lsummary for extracted sentences as mentioned in (Lin, 2004).
In equation 2 R lcs represents ROUGE-L recall, r i , c i are report and summary sentences respectively and m is total number of words in extracted report sentences. Once summary level ROUGE-L recall is calculated we choose summary with maximum value for further processing. Once proxy sentences and a summary is selected applying above methods extraction model is trained. In extraction model sentence level representation of report sentences is calculated using hierarchical word to sentence level Bidirectional Long Short Term Memory (Bi-LSTM) (Hochreiter and Schmidhuber, 1997 applied to word sequence of the sentence to get sentence level semantic information. Then Bi-LSTM is applied to sentences representations to get document level information in each sentence representation. We train attention mechanism (Bahdanau et al., 2014) based Pointer Networks (Vinyals et al., 2015) different from copy mechanism used in (See et al., 2017). Given these proxy sentences which we treat as ground truth and sentences extracted using pointer network, we train it to minimize cross-entropy loss. Once sentences are extracted using above methods we fine-tune T-5 based sequence-to-sequence model for abstraction. T-5 architecture is pretrained on C-4 dataset 2 using denoising method similar to that of Bert (Devlin et al., 2018) produce better result in language modelling. T-5 treats every task classification, summarisation, question answering, and translation as text-to-text format. These extracted sentences are taken as input and ground truth summary sentences are taken as output which is trained using cross-entropy loss. Input and output are prepared using T-5 tokenizer which outputs input ids and attention masks for input and target. These are then fed to model for training.
Once these individual components are trained individually, final complete model is trained using policy gradient algorithm with similar process as in (Chen and Bansal, 2018). At every extraction step agent samples an action to extract document sentence an receive reward r(t + 1) which is ROUGE-L F 1 between output from T-5 after abstraction and ground truth summary sentence.
The model is trained using advantage actor-critic model to mitigate bias incurred in REINFORCE (Williams, 1992). Overall idea of our method is that first proxy sentences are extracted using ROUGE score maximisation, then extraction model is trained to extract unique narrative sentences from the report then these sentences are paraphrased using T-5 algorithm for abstraction to give concise yet informative sentence. Reinforcement learning helps to maximise ROUGE score by rewarding good sentences that are extracted and penalising bad sentences.

Parameter Tuning
During abstraction we limit maximum number of sentences to 80, since there is limit on word limit of 1000 words and most of the reports' narrative can be summarised in less than 80 sentences as seen in section:3. Word2vec (Mikolov et al., 2013) embedding is used in representation of word in extractor model.. Vocab size is limited to 20000, embedding size is 300, maximum number of words in a sentence 60. Model is trained using Adam optimizer with learning rate of 0.001, decay rate of 0.5. Gradient norm is clipped at 1.0. Beam size if fixed to 2 in T-5 network, repetition penalty of 2.0. Rouge-LCS is used to optimize RL training. Abtractor and extractor networks are trained using cross-entropy loss. Traiing is done on Tesla-K80 12Gb colab GPUs with batch size of 16 and check point frequency of 16 batches.

Results and Analysis
In this section we present results from our experiments and compare with different baselines MUSE (Litvak et al., 2010), Text-rank (Mihalcea and Tarau, 2004), Lex-Rank (Erkan and Radev, 2004), and Polynomial Summarisation (Litvak and Vanetik, 2013 1 bold is marked for highest value amongst all the solutions for the task including baselines. Pre-trained Bert for summarisation is not fine-tuned for this specific task hence its performance is not as good as Pointer Network and PoinT-5 (Pointer Network + T-5). PoinT-5 network gives highest results for precision on all the evaluation metrics ROUGE -1,2,L,SU4. From this it can be inferred that generated summmaries are highly precise in extracting narrative sentences and matches with ground truth summaries. Recall is comparatively much lower than precision in most of the evaluated metrics which means that generated summaries do not cover all the information in ground-truth summaries. This large difference in precision and recall shows that generated summaries do not cover all the sentences possibly due to restriction imposed on the number of sentences during training to follow word limit in summaries. Less recall and high precision value shows that generated summaries provide highly relevant information but does not cover complete information in ground truth summaries. From results it is evident that there is not much difference in performance between pointer net performance and PoinT-5 which extraction played key role in the architecture. PoinT-5 and Pointer Networks are the only system to cross MUSE baseline in ROUGE-L by atleat 5 points. These models give highest F1 results in ROUGE-L and ROUGE 1 metrics and highest precision in ROUGE-L,1,2,SU4.

Conclusion and Future work
In this work we present our solution on Financial Narrative Summarisation(FNS2020) dataset using PoinT-5 method explained in 4. It is combination of both extractive and abstractive methods using Pointer Network and T-5. With these methods we are able to achieve highest precision score in every evaluation metric and achieve highest F-1 scores in ROUGE-LCS and ROUGE-1.
In our future work we would like to address several limitation of our method such as factual correctness in summaries which is very important in financial domain as done in (Zhang et al., 2019) in summarizing radiology reports. To improve precision of our generated summaries under 1000 words we would formulate a penalty if system generates more than 1000 words during training of RL algorithm rather than restricting algorithm to fixed number of sentences.  Table 2 presented total summaries and annual reports in train, validation, and test set. During our analysis, we found that most of the annual reports contain 100-200 sentences. There are 279 summaries with more than 500 sentences, which is large. Whereas in summaries, the average number of sentences is 50. Therefore summaries are one forth on average of annual reports and up to one-tenth in many cases. We present these analysis in Figure 2.

Data Type
Training Validation Testing Report full text 3,000 363 500 Gold Summaries 9,873 1,250 1673  B Extended Related Work (Li et al., 2018) proposes a training framework based on the actor-critic model. They apply the attentionbased sequence-to-sequence model as the actor to conduct summary generation. For the critic, they combine the maximum likelihood estimator with a well designed global summary quality estimator. (Nallapati et al., 2016) propose RNN based encoder-decoder model for abstractive summarisation. They apply bi-directional GRU-RNN at the encoder side and uni-directional GRU in decoder with attention. In their approach, each mini-batch's decoder-vocabulary is restricted to words in the source documents of that batch. (Rush et al., 2015) also propose attention based sequence to sequence model for abstractive summarisation. (Paulus et al., 2017) states that attentional, RNN-based encoder-decoder models for abstractive summarisation have achieved good performance on short input and output sequences. For longer documents and summaries, however, these models often include repetitive and incoherent phrases. Hence they combine standard word prediction with the global sequence prediction training of RL which makes resulting summaries become more readable. (Narayan et al., 2018) propose a reinforcement learning-based sentence ranking approach in extractive summarisation. (Cheng and Lapata, 2016) use attention architecture for words and sentence level extraction in extractive summarisation method. (Yasunaga et al., 2017) proposes a multi-document summarization system that exploits the representational power of deep neural networks and the sentence relation information encoded in graph representations of document clusters. Specifically, they apply Graph Convolutional Networks on sentence relation graphs. (See et al., 2017) propose a novel "Pointer Generator" networks for abstractive summarisation. In this model a word is chosen with probability P gen from overall vocabulary and 1−P gen from current sentence using attention weights. They apply coverage mechanism by (Tu et al., 2016) to avoid word repetitions in the summary which is common in large documents. (Chen and Bansal, 2018) apply combination abstractive and extractive summarisation by join training using reinforcement learning. They apply pointer network for extraction (different from Pointer Generator) and RNN based encoder-decoder for abstraction. (Hsu et al., 2018) propose unified extractive and abstractive with a hierarchical sentence and word level attention model using novel inconsistency loss. (Wang et al., 2019) used T-5 (Raffel et al., 2019) based sentence representation for combined extractive and abstractive training.
Financial Narrative Summarisation has been explored in the past by (Cardinaels et al., 2018). They provide multiple evidence that algorithm-based summaries are less positively biased than management summaries.