{'fraction_non_alphanumeric': 0.03637647347740668, 'fraction_numerical': 0.016545923379174852, 'mean_word_length': 4.845505113942221, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Causal News Corpus: Annotating Causal Relations in Event Sentences from News

Despite the importance of understanding causality, corpora addressing causal relations are limited. There is a discrepancy between existing annotation guidelines of event causality and conventional causality corpora that focus more on linguistics. Many guidelines restrict themselves to include only explicit relations or clause-based arguments. Therefore, we propose an annotation schema for event causality that addresses these concerns. We annotated 3,559 event sentences from protest event news with labels on whether it contains causal relations or not. Our corpus is known as the Causal News Corpus (CNC). A neural network built upon a state-of-the-art pre-trained language model performed well with 81.20% F1 score on test set, and 83.46% in 5-folds cross-validation. CNC is transferable across two external corpora: CausalTimeBank (CTB) and Penn Discourse Treebank (PDTB). Leveraging each of these external datasets for training, we achieved up to approximately 64% F1 on the CNC test set without additional fine-tuning. CNC also served as an effective training and pre-training dataset for the two external corpora. Lastly, we demonstrate the difficulty of our task to the layman in a crowd-sourced annotation exercise. Our annotated corpus is publicly available, providing a valuable resource for causal text mining researchers.


Introduction
Causality is a core cognitive concept and appears in many natural language processing (NLP) works that aim to tackle inference and understanding (Jo et al., 2021;Dunietz et al., 2020;Feder et al., 2021). Generally, a causal relation is a semantic relationship between two arguments known as cause and effect, in which the occurrence of one (cause argument) causes the occurrence of the other (effect argument) (Barik et al., 2016). Figure 1 depicts examples of sentences expressing causality, and ones that do not. Notice that causality can be expressed in various ways: The first causal example is signaled by the explicit causal marker "due to" while the second example has causality indicated by alternative lexicalizations such as "created". In the last example, an implicit causal relation exists between the verbless clause and the matrix clause, conveying that the dissatisfaction with the package leads to the workers' sit-in. For sentences without causality, they must be missing either a cause or effect argument, or both. Causal Question Answering and Generation applications (Dalal et al., 2021;Hassanzadeh et al., 2019;Stasaski et al., 2021) are some immediate downstream Figure 1: Annotated examples from Causal News Corpus. Causes are in pink, Effects in green and Signals in yellow. Note that both Cause and Effect spans must be present within one and the same sentence for us to mark it as Causal.
Natural Language Understanding (NLU) applications popular in NLP today. Despite the importance of identifying causality in text, datasets are limited (Asghar, 2016;Xu et al., 2020;Tan et al., 2021;Yang et al., 2021) and oftentimes, different researchers craft their datasets with different rules, leaving users with no proper way to compare models across datasets. Our work is directed at annotating parts of the multilingual protest news detection dataset (Hürriyetoglu et al., 2021a;Hürriyetoglu et al., 2021b;Hürriyetoglu, 2021) for event causality. Our contributions 1 are as follows: • We created the Causal News Corpus (CNC), which is a corpus of event sentences annotated with binary labels indicating whether each sentence contains a causal relationship or not 2 .
• We showed that a neural network built on a stateof-the-art pre-trained language model could predict the causality labels with 81.20% F1, 77.81% Accuracy, and 54.52% Matthews Correlation Coefficient on test set.
• We crafted annotation guidelines that align with existing event causality and linguistic schemes. Our experiments show that our dataset is indeed compatible with other causality corpora.
• We observed that the layman struggles to replicate the annotations by our experts in a crowd-sourced annotation exercise, demonstrating the difficulty of our task.
The rest of the paper is organized as follows: Section 2 reviews the literature on event causality. Section 3 describes the compilation and annotation of the corpus. Section 4 describes the baselines we created for causal event sentence classification and the experiments that were conducted on CNC and two other datasets. Section 5 outlines experiences from a crowd-sourcing annotation exercise. Finally, Section 6 concludes the paper and discusses potential avenues for future work.

Related Work
The extraction of causality from text remains a significant challenge because semantic understanding of the context and world knowledge is oftentimes needed. In computational linguistics, a lot of effort has been put into extracting causal knowledge from text automatically (Blanco et al., 2008;Do et al., 2011;Kontos and Sidiropoulou, 1991;Riaz and Girju, 2013). Previous work has focused on event causality, creating corpora like CausalTimeBank (CTB) (Mirza et al., 2014)  2 Annotation of the corpus is an ongoing effort: We also aim to add information like cause-effect-signal span markings and causal types, and these additional information will also be made public once available. corpora include rich event features like event class attributes and temporal expressions in TimeML annotation format. However, these corpora are typically limited in size. For example, EventStoryLine (ESL) has 1,770 causal event pairs, CaTeRS has 488 causal links, while CTB has 318 causal event pairs. Our work contributes to this strand of literature by providing more data: We have identified 1,957 causal event sentences, each of which can contain multiple event pairs. There is also a discrepancy between such event causality corpora and other causality corpora. Penn Discourse Treebank (PDTB) (Prasad et al., 2008;Webber et al., 2019;Prasad et al., 2006) is a corpus that annotates semantic relations (including causal relations) between clauses, expressed either explicitly or implicitly. This corpus is large (e.g. PDTB-3 has over 7, 000 examples for the CONTINGENCY.CAUSE sense alone) and is potentially useful for training an accurate event sentence classifier. Therefore, we believe that it will be beneficial to align the annotation guidelines of these two corpora types. We did so by constructing event causality annotation guidelines using linguistic rules adapted from PDTB-3. However, our corpus differs from PDTB-3 by focusing on event sentences and accepting more fine-grained arguments that do not necessarily form a clause. This approach of including more varied constructions of causality is similar to the work by BECauSE 2.0 (Dunietz et al., 2017). When discussing causality in text, many corpora only focus on explicit relations (Girju and Moldovan, 2002;Dunietz et al., 2017). An explicit causal relation refers to an example where the cause-and-effect relation is expressed by means of a clear connective or some other causal signal. An example was presented earlier in the first sentence of Figure 1. Rule-based approaches (Girju and Moldovan, 2002;Khoo et al., 2000;Sakaji et al., 2008) have been shown to work well on such constructions. For example, researchers have used such approaches to semi-automatically curate datasets for cause-effect related tasks (Cao et al., 2016;Heindorf et al., 2020;Stasaski et al., 2021). However, in the real world, implicit relations are more common but more challenging to identify (Hidey and McKeown, 2016). Therefore, in our work, we do not limit ourselves to a pre-defined list of connectives but rather, include causal examples in more varied linguistic constructions for more realistic use cases. With respect to this, we differ from CTB, which only annotated explicit causal examples.

Data Source
CNC builds on the datasets featured in a series of workshops directed at mining socio-political events from news articles: Automated Extraction of Socio-political Events from News (AESPEN) in 2020 (Hürriyetoglu et al., 2020b;Hürriyetoglu et al., 2020a) and Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) in 2021 (Hürriyetoglu et al., 2021a;Hürriyetoglu et al., 2021b;Hürriyetoglu, 2021). The data was based on randomly sampled articles from multiple sources and periods, and all annotations were performed by two annotators, adjudicated and spot-checked by a supervisor, and corrected further semi-automatically (Hürriyetoglu et al., 2021c;Yörük et al., 2021). In total, 869 news documents and 3,559 English sentences were available. We annotated all 3,559 sentences in this paper. We utilized the dataset from CASE 2021 directly and some sentences were not well processed. We noticed that a few examples were actually comprised of more than one sentence and titles were still included in the texts. In our next data release, we will redo the parsing of these examples.

Guidelines
CNC includes annotations for binary causal event classification in sentences; We labeled sentences to be Causal or Non-causal. Sentences must contain at least a pair of events, defined as as things that happen or occur, or states that are valid, following TimeML (Saurı et al., 2006). For causality, we utilized the definition of CONTINGENCY from PDTB-3, which assigns this label for samples where "one argument provides the reason, explanation or justification for the situation described by the other" (Webber et al., 2019). The following four senses were recognized as Causal: • CAUSE: Arg1 and Arg2 are causally influenced but not in a conditional relation. Example connectives are "because" and "since".
• PURPOSE: One Arg describes the goal while the other Arg describes the action undertaken to achieve the goal. Example connectives are "in order to" and "so that".
• CONDITION: One Arg presents an unrealized situation, that when realized, would lead to the situation described by the other Arg. Example connectives are "if" and "as long as".
• NEGATIVE-CONDITION: One Arg presents an unrealized situation, that if it does not occur, would lead to the situation described by the other Arg. Example connectives are "till" and "unless".
Since our work is event-based, the senses that provided reasons for the speaker to utter a speech, or the hearer to have a belief (i.e. +SPEECHACT or +BELIEF types) were treated as Non-causal.
A major difference between our approach and PDTB is that we permitted non-clausal elements such as phrases in our annotation scheme as arguments as long as these spans meet our definition of event. PDTB is a discourse bank that restricts its argument spans to discourse relations. As noted by Dunietz et al. (2017), causal spans of multiple linguistic forms are excluded from PDTB.
For instance, in PDTB, verb signals like "caused" are not annotated in the sentence "Speculation about Coniston has caused the stock to rebound from a low of $145.". Consequently, PDTB also did not annotate the spans before and after "caused" as Cause and Effect arguments. Given our focus on news events, more expressions of causality are required for proper interpretation. Therefore, on top of PDTB-approved clauses, we allowed for the following to be recognized as an argument: noun phrases (including nominalizations) 3 , and verb phrases (including ones that fall outside of a coordinated structure, provided that the corresponding span of its relation is a noun phrase that would complete the verb phrase to form a clause). In general, these relaxations of rules allowed us to annotate more fine-grained arguments, including ones within clauses.
To ground our annotations, we utilized the five tests for causality based on the work by Grivaz (2010) and Dunietz et al. (2017). In Table 1, we report some examples to illustrate the applications of the causality tests and the resulting classification labels. Additional details and examples are provided in Appendix A.
Five Tests for Causality 1. Why: The example is not causal if the reader is unable to construct a "Why" question regarding the Effect.

Temporal order:
The example is not causal if the Cause does not precede the Effect in time.

Counterfactual:
The example is not causal if the Effect is equally likely to occur or not occur without the Cause.

Ontological asymmetry:
The example is not causal if the reader can readily swap the Cause and Effect claims in place.

Linguistic:
The example is likely to be causal if it can be rephrased into "X causes Y" or "Due to X, Y." CAUSE types Quoting the PDTB-3 annotation manual (Webber et al., 2019), "CAUSE.REASON (CAUSE.RESULT) is used when Arg2 (Arg1) gives the reason, explanation or justification, while Arg1 (Arg2) gives its effect". Within the CAUSE sense, we noticed that its sub-senses, CAUSE.REASON and CAUSE.RESULT, fit into the five tests framework well. In particular, we were able to construct a question like "Why did <effect>?", and answer with "Because (of) <cause>." in an extractive manner. However, we required some relaxations in the construction of the "Why" test to be able to include other senses into our corpus. We highlight the relaxations as follows: The protests spread to 15 other towns and resulted in two deaths and the destruction of property .
Chale was allegedly chased by a group of about 30 people and was hacked to death with pangas, axes and spears .
The strike will continue till our demands are conceded .
Cond.) Table 1: Examples illustrating the applications of the Tests for Causality (Grivaz, 2010;Dunietz et al., 2017). Cause in pink, Effect in green; potential Cause in gray. Signals are not marked.

PURPOSE types
We permitted the answer to include additional words like "In order to (achieve the goal of) <cause>". In this setting, the cause is a goal, or a justification for the effect action. See Appendix A Example (3).

CONDITION types
We allowed the relaxation of the question to include modal terms, for example: "Why could/would <effect> occur?". See Appendix A Example (7).

CAUSE.NEGRESULT and NEGATIVE-CONDITION
types Quoting the PDTB-3 annotation manual (Webber et al., 2019), "CAUSE.NEGRESULT is used when Arg1 gives the reason, explanation or justification that prevents the effect mentioned in Arg2". For CAUSE.NEGRESULT and NEGATIVE-CONDITION senses, we allowed a further relaxation of the question to include "prevent" or "cause-to-end" terms, as in: "Why was <effect> prevented?", "Why could be <effect> prevented?" or "Why did <effect> end?". See Appendix A Example (8).
For samples with multiple sentences, we annotated each sentence independently, that is, we do not consider cross-sentence causality. We acknowledge that by omitting cross-sentence examples, we potentially lose out between a 1/10 to a 1/5 of additional causal examples, based on proportions estimated from CTB and ESL respectively. Titles were also not annotated.

Workflow
Five annotators and one curator were involved in the annotation effort. Figure 2 summarizes the different annotation iterations to build CNC.

Initial Training
The initial training phase for annotators corresponds to Round 1 in Figure 2. Guidelines were presented at the beginning of the annotation phase, and examples outside of CNC dataset were ran through with annotators. Figure 2: Summary of annotation workflow. n: sample size for the subset. a: number of annotators for the subset. κ: Krippendorff's Alpha score for samples per round. Round 1 was a trial training round where annotations were discarded. All subsequent rounds were for the training set except Round 6 which was for the test set.
We found it difficult to arrive at a consensus on the final label across annotators during this initial discussion. This experience motivated us to update our guidelines after each round of annotation to continually improve annotators' understanding of the guidelines.

Guidelines Refinement
The difficulty in achieving alignment across annotators, especially at the initial stages, arose from the following three issues: Firstly, since our task challenges literacy, comprehension, and cognitive understanding of a sentence, it is a complex and involved task. Thus, a significant amount of time is required for careful reading of the annotation manual and examples to improve familiarity. Secondly, sensitivity to causal cues and signals (including implicit signals) takes time to develop, and might not be apparent to some until later rounds.
Thirdly, different annotators rely on different guidelines or feedback mechanisms to improve their accuracy to identify causal labels faithful to the guidelines.
Therefore, we were motivated to update our guidelines in an iterative fashion after each annotation round to refine the descriptions and include more examples that would help improve alignment across annotators. A group discussion always followed at the end of each annotation round for annotators and curator to share feedback and agree on the final guidelines. We also allowed annotators to skip examples if they were unable to come up with a suitable label. For any major changes in guidelines that might affect previous rounds of annotations, the curator identified such examples and discussed with all annotators before the final annotation is amended. In total, 33 examples had their labels changed in this manner.

Final Annotation
In Round 2, we achieved a Krippendorff's Alpha (κ) score of 29.77% across five annotators. After the second annotation round, we split the five annotators into two groups to cover more data within our time frame. Concurrently, we also noticed an improvement in the agreement scores to 48.08%. By the third annotation round, the guidelines had no further changes. Surprisingly, we noticed agreement scores fell to 26.15% in Round 4. This could be due to the much larger dataset presented to annotators in this round, causing reduced focus. In fact, we noticed the main reason for the drop arose from one annotator in the first group. Excluding his contribution, the agreement increased to a reasonable score of 39.34%. After working out misunderstandings, the subsequent two rounds of annotations achieved consistent agreement scores of 36.95% and 48.55%. Across all rounds, our dataset has an agreement score of 34.99%. In our final corpus, every example was annotated by at least two annotators. If only one annotation is present because the other annotator skipped that example, the curator's vote is final. The curator also resolved any ties present throughout the annotation exercise. In the end, we obtained 3,559 annotated sentences, with 1,957 marked as Causal and 1,602 marked as Non-causal. This is reflected in the first row of Table 2, alongside data sizes for two other external datasets we experimented with in the next section. We followed the traintest split of our data source, and also provide the breakdown of data sizes by class labels in

Experiments
In this section, we present a battery of baseline models and results of causal sentence classification on the CNC data and the two existing news corpora, PDTB-3 (Webber et al., 2019) and CTB (Mirza et al., 2014).

Evaluation Metrics
For all experiments, we evaluated the predictions against true labels using five evaluation metrics: Precision (P), Recall (R), F1, Accuracy (Acc) and Matthews Correlation Coefficient (MCC) (Matthews, 1975). Formulas are provided below in Equations (1) to (5), where T P / T N / F P / F N stands for True Positive/True Negative/False Positive/ False Negative respectively. All metrics were calculated using the scikit-learn (Pedregosa et al., 2011) package.

Dummy Baselines
To create naive baselines, we obtained dummy predictions using two approaches. For the first row indicated by "All Causal" of Table 3, we predicted all examples to be Causal, which is the majority label of our dataset. Notice that the Recall score is perfect (100%), because all Causal examples will be identified if we predict all as Causal. However, Precision is low (56.59%), because all Non-causal examples were wrongly identified as Causal. In the subsequent discussions, we focus our key evaluation metrics to be F1, Accuracy and MCC. For the second row, we randomly assigned the Causal or Non-causal labels to each example based on the ground truth distribution from CNC's training set, averaged over 1, 000 runs.

Neural Network Baselines
We constructed a baseline model by fine-tuning the pretrained Bidirectional Encoder Representations from Transformers (BERT) model (Devlin et al., 2019). In particular, we used bert-base-cased from Huggingface (Wolf et al., 2020)

Training on CNC
From the third row in Table 3, when we trained on our training set and applied on the test set, we achieved a reasonable F1 score of 81.20%, Accuracy of 77.81%, and MCC of 54.52%, each exceeding the dummy baselines scores reflected in the first two rows. This finding demonstrates that our annotations are internally consistent and reliable for a simple baseline model to learn effectively.

Training on External Corpora
In this subsection, we check the transferability of existing event causality and causality corpora on CNC.  Table 3, there is some overlap in the annotation schemes of PDTB and CNC. Therefore, the model was able to achieve Accuracy and MCC scores of 61.74% and 32.09%, which are higher than the first two dummy baselines. However, the F1 score is only at 55.43%, comparable to the Random baseline and worse than the All-Causal assignment. The low F1 score is driven by the low Recall score, which indicates that a model trained on PDTB-3 is unable to detect most of the true Causal examples. However, its high Precision score indicates the model does return high-quality predictions. The low Recall could be explained by the fact that we included guidelines in our annotation schema that are different from PDTB's schema. For example, we were not strict about arguments having to be a proper clause. That is, for a sentence like "John caused the fire and walked away.", we would have marked it as Causal. However, since "John" and "the fire" are not clauses, PDTB would not mark this example as Causal. As a result, a model trained on PDTB would also recognize fewer causal constructions.
To tackle the issue of imbalanced class distribution in PDTB-3, which might affect model performance, we reran an experiment with randomly sampled Noncausal examples to match the number of Causal examples. This corpus is referred to as "PDTB-3 Bal" (n = 19, 834) in Row 5 of Performance was dismal, returning comparable or worse scores for F1 and Accuracy than both dummy baselines. We hypothesized that this poor outcome is a result of CTB's extremely imbalanced class distribution. Indeed, when we retrained on a balanced sample, "CTB Bal" (n = 636), we observed performance improvements across all metrics in the last row of Table 3. A reasonable F1 score of 64.05%, Accuracy of 64.63% and MCC of 32.13% was obtained, similar to or slightly better than PDTB's performance, despite CTB Bal being much smaller in size.
To conclude, we trained models on external corpora and used them directly to make predictions on our proposed CNC corpus. This demonstrates the transferability of existing causality corpora on our corpus. Nonetheless, since both external corpora contain multi-sentence examples and differ slightly in annotation guidelines, some performance differences are expected.

Training & Testing on External Corpora
We continue to study the compatibility of CNC with the two external corpora. In this section, we study if a model trained on CNC will be helpful for event sentence causality prediction on unseen datasets like PDTB-3 and CTB.
For each of the three corpora, we trained on the whole corpus and applied the trained model on the remaining two corpora. For example, we trained on all examples from CNC (3,559 examples) and used the model to predict labels for PDTB-3 and CTB Bal. We experimented with the balanced corpus of CTB because the results of the experiments presented in Section 4.4 shown that the full CTB returns extremely poor performance when used for training. We also included the 5-Fold cross-validation (CV) scores as a reference: For each dataset, we randomly split the examples into five validation sets, and for each round, the corresponding remaining samples were used for training. Table 4 reflects key experimental results. As expected, for every corpus, the CV training setup always returns the best performance. After which, for both PDTB-3 and CTB Bal, we noticed that a model trained on CNC always returns a higher performance across metrics. For example, when testing on CTB Bal, a model trained on PDTB-3 achieved 60.79% while a model trained on our corpus achieved 80.65% F1 score. We introduced a metric called Transferability Rate (TRF), which computes the mean score across test sets. TRF indicates how well a model trained on a given corpus works on average for unseen, external datasets. CNC obtained the highest TRF across the three corpora for both F1 and MCC, reflecting good transferability. Our findings once again demonstrate that there exists some differences in the annotation schemes for causality, which reduces the transferability of trained models between corpora. Nevertheless, our corpus is the most transferable between the event causality (i.e. CTB) and linguistic causality (i.e. PDTB) corpora studied. Therefore, our annotation guidelines and dataset can serve as a bridge between these two strands of causality corpora.

Using CNC for Pre-Training
Since we believe CNC is transferable, we hypothesize that CNC will also be useful for training a pretrained model (PTM). Later, this PTM can be finetuned onto any other causality-based corpora. We term the BERT PTM updated on all examples of CNC as 'CNC-PTM'. CNC-PTM was then fine-tuned on each external dataset (i.e. PDTB and CTB Bal). We again used the 5 CV setup 4 , but reduced the training epochs to 2. We chose this low number in order to keep our model intact. The fine-tuning step only serves to adjust the model's alignment to better fit the new data distribution.
From Table 5, we indeed found improvements in the average performance across folds for both PDTB and CTB Bal. Logically, CNC-PTM is better initialized for causal sentence classification than the general bert-base-cased PTM for two reasons: (1) It had access to more causal examples and, (2) Its structure is more adapted for the classification task. Thus, performance improvements are expected. These findings again support our claim that CNC is generalizable to out-of-distribution datasets. Therefore, it was useful for updating a PTM.

Crowd-Sourcing Annotations
We experimented with crowd-sourced workers using Amazon Mechanical Turk (MTurk . The annotation manual was provided to the MTurk workers, and no restrictions were set on the type of workers we allowed for. We requested for five unique annotators to be assigned to each example. Therefore, 250 annotations were received 5 , of which we had 44 unique workers. Kappa scores were low at 1.62%. Evaluation metrics of these crowd-sourced annotations against our expert labels are reflected in Table 6. The second row reflects predictions taken based on the majority vote (i.e. mode out of five votes per example). The third row reflects outcomes when we compared each of the 250 annotations with our true labels independently. In both cases, performance of crowd-sourced workers is poor.
Since most annotators labelled examples as Causal, the results are similar to the results from an All-Causal dummy baseline (shown in the first row). In fact, in terms of F1, Accuracy, and MCC, the workers' scores are close, if not worse than, the All Causal baseline. As highlighted in Section 3.3, the task is challenging and requires a thorough understanding of the annotation guideline to do well. Given that MTurk provides a low monitoring environment with no room for itera-    tive feedback, it is understandable that crowd-sourced workers perform poorly (close to random assignment) at their tasks. Nevertheless, this experiment provided a few learning points: Firstly, our manual annotations are not easy to identify by a layman, and skilled annotators are required. Similar to our annotation experience, it is necessary to have repeated discussions and feedback sessions with annotators so as to improve agreement scores. Secondly, compared to a BERT model trained on internal examples or external examples annotated with similar logic, the layman identifies causality poorly 6 . Caselli and Inel (2018) argues that discrepancies in the quality of crowd-sourced annotations from experts should be used as an estimate on how complex the task is. Together, these findings highlight that our corpus is a unique and valuable resource, requiring an immense amount of time and effort to create with experts.

Conclusion
Causality is an important cognitive concept that deserves to have a corpus dedicated to the definition and identification. CNC focuses on annotating causality in text, and so far, we have created an annotation schema to identify if event sentences contain causal relations or not. CNC's annotation guidelines are constructed based on linguistic rules, and covers a wider array of causal linguistic constructions than previous works. We also demonstrated transferability between CNC and existing datasets that include causal relations like CTB and PDTB. Finally, the binary causal event classification task is challenging: layman workers achieved poor performance in our crowd-sourced annotation experiment. Therefore, CNC, which has been annotated by experts, is a valuable resource for researchers working in the causality and event causality space. CNC could potentially be relevant to NLU researchers working on downstream tasks too. We are also organizing a shared task using CNC to promote the development of automatic causal text mining solutions 7 . In our next dataset release, we will clean some parts of our current dataset and perform additional curation of mispredicted examples in CV. We are also in the midst of adding fine-grained annotations, such as cause-effect-signal spans and causal concept labels.

Acknowledgements
This project is supported by the National Research Foundation, Singapore under its Industry Alignment Fund -Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.  Table 7: Metrics from predictions with the LSTM model on CNC Test Set using different training sets. F1, Precision (P), Recall (R), Accuracy (Acc) and Matthews Correlation Coefficient (MCC) are reported in percentages (%). Highest score per result column is indicated in boldface.

Bibliographical References
2. Temporal order: The demands must be conceded (Cause) before preventing the strike from continuing (NegEffect).

Counterfactual:
The strike cannot be stopped from continuing (NegEffect) if the demands are not conceded (NegCause).

4.
Ontological asymmetry: Using Cause as an Effect to construct a question: Why was it that "our demands are conceded"? Because "the strike could be stopped" -does not answer the question.

B. Other Baseline Models
As described in Section 4, in addition to the BERT model, we constructed a LSTM model as one of our baselines, considering its ability to account for longdistance relations between words. In this model, firstly, an embedding layer initialized with fastText's Common Crawl 300-dimensional embeddings (Bojanowski et al., 2017). We experimented with GloVe 300dimensional embeddings (Pennington et al., 2014) and a concatenation of GloVe and fastText in our initial experiments and selected fastText because it outperformed others. This layer is followed by two bidirectional LSTM layers with a dense layer on top. The final prediction is obtained via a softmax layer. For our training parameters, we used a maximum sequence length of 128, batch size of 32, a learning rate of 1e −3 with Adam optimizer and epochs of 20 with early stopping patience of 5. The obtained results are summarized in Table 7. We observed that performance of this system is significantly lower than BERT for all scenarios we tested. However, conclusions regarding the transferability of PDTB-3 and CTB Bal onto CNC (based on F1 and MCC scores) holds.