{'fraction_non_alphanumeric': 0.0349756179586346, 'fraction_numerical': 0.020850849167647553, 'mean_word_length': 4.777076250607091, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
UPB at FinCausal-2020, Tasks 1 & 2: Causality Analysis in Financial Documents using Pretrained Language Models

Financial causality detection is centered on identifying connections between different assets from financial news in order to improve trading strategies. FinCausal 2020 - Causality Identification in Financial Documents – is a competition targeting to boost results in financial causality by obtaining an explanation of how different individual events or chain of events interact and generate subsequent events in a financial environment. The competition is divided into two tasks: (a) a binary classification task for determining whether sentences are causal or not, and (b) a sequence labeling task aimed at identifying elements related to cause and effect. Various Transformer-based language models were fine-tuned for the first task and we obtained the second place in the competition with an F1-score of 97.55% using an ensemble of five such language models. Subsequently, a BERT model was fine-tuned for the second task and a Conditional Random Field model was used on top of the generated language features; the system managed to identify the cause and effect relationships with an F1-score of 73.10%. We open-sourced the code and made it available at: https://github.com/avramandrei/FinCausal2020.


Introduction
Financial news contain various descriptions of causes and effects between financial objects that influence sales, employment, earnings, or stock prices. Causal analysis can be used to detect correlation between news and prices of different assets (Qu and Kazakov, 2019) by identifying different consequence, that in return can lead to major events as a financial crisis (Stavroglou et al., 2017;Tiffin, 2019) or possible arbitrage opportunities (Stavroglou et al., 2017). For example, given the follow-up statement regarding Brexit "The Leave win led to an 11 percent drop in GBP/USD overnight.", we can infer that "The Leave win" is the cause for "an 11 percent drop in GBP/USD overnight", which represents the effect and indicates a change in price of the GBP/USD currency pair (Mariko et al., 2020).
Nowadays, the high volume of published news and financial documents hinders and renders unfeasible the manual analysis of all articles in order to extract implications on the economy. Thus, a need for developing new tools for causality extraction emerged to facilitate this process. Methods based on pattern detection (Khoo et al., 2000), on combinations of patterns and rules (Sorgente et al., 2013), and on attention mechanisms with different embeddings (Li et al., 2019) were introduced in the field of causal detection. A shared task was proposed at the Financial Narrative Processing Workshops, namely FinCausal-2020 (Mariko et al., 2020), to evaluate the performance of each participant system on two tasks defined on a financial corpus proposed by YseopLab. The first task consisted in building a classifier to identify whether a sentence contains a financial causality or not, whereas the second task extracted causes and also their corresponding effects from a given text.
The rest of the paper is organized as follows. The second section introduces our proposed methods for tackling the two sub-tasks, while the third section presents our results, together with experimental setup and an error analysis. The paper ends with conclusions and proposals of future work. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/.

Dataset
The organizers made available 3 datasets for both tasks: trial, practical, and evaluation. The trial dataset for the first task contained 8580 samples, out of which only 569 (i.e., 6,69% from total samples) included causal events. The practice dataset has approximately the same distribution as trial, with 13478 samples out of which 1010 were labeled as causal (i.e., 7,49% from total samples). The evaluation dataset was kept blind, but it has the same distribution as trial and practice.
The dataset for the second taks largely corresponds to samples from Task 1 that are labelled as causal, with the specification of causal and effect sub-strings. The trial dataset contains 641 samples, while the practical dataset contains 1109 samples. The evaluation dataset contained 638 samples with removed annotations, i.e., the text is revealed, but there are no causal or effect representations.

Proposed Solutions
Five Transformer-based language models (Vaswani et al., 2017) were considered for the the first task and were fine-tuned on the FinCausal-2020 dataset, namely: BERT (base and large) (Devlin et al., 2018), AL-BERT (base and large) (Lan et al., 2019), RoBERTA (base and large) (Liu et al., 2019), SciBERT (base) (Beltagy et al., 2019), and FinBERT (base) (Araci, 2019). The fine-tuning mechanism was the one proposed by Devlin et al. (2018) who take the embedding of the first token and project it into a scalar that represents a probability. Then, we minimize the Kullback-Leibler divergence between the distribution of the dataset and the distribution predicted by the model by using the Adam optimizer (Kingma and Ba, 2014). We also experimented with various classifier ensembles taking into account different combinations of the fine-tuned language models as these ensembles surpass in general the performance of individual models; majority voting was considered for the final labeling of each sentence.
The second task was operationalised as a token labeling, similar to part-of-speech tagging (Bohnet et al., 2018) or named entity recognition (Dumitrescu and Avram, 2019), by marking the corresponding tokens of the causes and effects sequences with CAUSE and EFFECT labels, whereas the rest of the tokens were marked as O. Inside-Outside-Beginning (IOB) format was used for the labels to support the identification of the sequences at inference time. BERT-base models were used to obtain contextualized embeddings for each token, and then a Conditional Random Field (CRF) (Lafferty et al., 2001) was trained to predict the most probable sequence of labels for a given input text.
Although the IOB format and the CRF aided in modeling the problem as a token labeling task, the extraction of causes and effects is harder at inference because the model can make an incorrect prediction in the middle of a cause/effect sequence, or predict a cause/effect token in the middle of O tags. Several heuristics were introduced to mitigate this issue when extracting the causes and effects, namely: • If a cause or effect sequence has a length lower than 4, it is ignored.
• In case that the model does not predict a cause or an effect in a sequence, all the remaining O tags become the missing cause or effect.
• If a different predicted sequence (e.g. "EFFECT EFFECT CAUSE CAUSE EFFECT EFFECT") is found inside the initial sequence, the inside predicted sequence has a length lower than 4, and the other part of the initial sequence does not have the beginning tag 1 , then the inside predicted sequence is ignored and the two initial sequences are combined.
• After the tokens are labeled as either CAUSE or EFFECT, the original text is retrieved by identifying the boundary words using the Levenshtein distance (Levenshtein, 1966); a word is considered a boundary word if it has a Levenshtein distance equal to 0 and a length higher than 3. Afterwards, all words between boundary words are labeled as either CAUSE or EFFECT.

Experimental Setup
We fine-tune five pretrained models provided by Hugging Face 2 : BERT, RoBERTa, ALBERT, SciBERT, and FinBERT. The models are trained using a Tesla P100-PCIE-16GB GPU. Grid search is performed to determine best hyper-parameters, learning rate, and maximal length of sequence. We use a maximal length of the sequence of 256 for base models and 384 for large models. Learning rate is set to 2 × 10 −6 , with the exception of ALBERT-Base where a learning rate of 2 × 10 −5 is preferred. Tests before the evaluation submission were made on a dataset that contained data from trial and practice, separated in 80% train and 20 % validation partitions, while keeping the same distribution as previous datasets.

Results
Table 1 introduces the results for the first task obtained using the best ensemble of language models, as well as each individual language model, on both the validation and the evaluation sets. The best ensemble contained the following models: ALBERT-Base, SciBERT-Base, BERT-Large, FinBERT-Base and RoBERTA-Large, and it outperformed RoBERTa-Large with 0.23% on the evaluation set, obtaining an 97.55% F1-score, while being marginally behind by 0.19% from the winning model. While considering individual models, BERT-Large obtained the highest score on the validation set with an 95.77% F1-score, surpassing the baseline of the organizers by 0.54%; RoBERTA-Large obtained the highest score on the evaluation set with an 97.32% F1-score. The results on the second task using BERT-Base and CRF are depicted in Table 2. The model outperformed the baseline by 17.79% in terms of F1-score, but it was surpassed by 21.61% on the evaluation set.

Error Analysis
Several problems were identified when inspecting the trial and practice datasets, and the errors performed by our models. We observed that false positive examples on the first task exhibit statements containing actions or facts, but these actions or facts do not affect other entities in that sentence. For example, a penalty in the first entry from Table 3 may have effects on individuals with debts, but this is not explicitly mentioned in the sentence; thus, the context is not complete to consider it a cause-effect situation. In  the second example, shares purchased by insiders usually have impact on price, but these details are not present. Thus, our models make incorrect predictions because part of these structures are more common in causal statements. In addition, we identified issues in the case of false negative when the cause and the effect are placed in different environments or fields. For example, the cause in the third example is related to attractions located in Manhattan, but the effect is the decrease on an online ticket reselling platform. This is a long dependency and need more specialized knowledge is required to infer these effects. The same situation is encountered in the fourth example, where a company's revenue in this quarter is associate with a 3.9% decrease from same period of last year. Errors on the second task consist mostly in the incorrect detection of complete sequences of cause or effect. As presented in Table 4, the beginning of our sequences is well identified in most cases, but too few words are considered. This situation may occur from incorrect predictions made by our model or a bad translation from token to text given the post-processing.

Conclusion and Future Work
As the financial volume of data grows, it becomes harder and harder to analyze chain of events, with corresponding cause and effects. Our solution for the first task at FinCausal-2020 was an ensemble of five Transformer-based language models that placed second on the evaluation leader-board with an 97.55% F1-score. The second task considered a BERT-base model to extract contextualized embeddings for each token, that were further used to train a CRF; an F1-score of 73.10% was achieved on the evaluation set. Possible directions of research include using more powerful language models like XLNet (Yang et al., 2019) or exploring if an additional bidirectional long-short term memory (BiLSTM) on top of the language model to improve the results on either task. In addition, we envision the consideration of specific discourse markers, as well as transfer learning using similar tasks -for example, a model trained on the SNLI corpus from Stanford (Bowman et al., 2015).