{'fraction_non_alphanumeric': 0.03675015548142704, 'fraction_numerical': 0.011439663783192928, 'mean_word_length': 4.5884149552396, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multimodal Event Graphs: Towards Event Centric Understanding of Multimodal World

Understanding how events described or shown in multimedia content relate to one another is a critical component to developing robust artificially intelligent systems which can reason about real-world media. While much research has been devoted to event understanding in the text, image, and video domains, none have explored the complex relations that events experience across domains. For example, a news article may describe a `protest' event while a video shows an `arrest' event. Recognizing that the visual `arrest' event is a subevent of the broader `protest' event is a challenging, yet important problem that prior work has not explored. In this paper, we propose the novel task of MultiModal Event Event Relations to recognize such cross-modal event relations. We contribute a large-scale dataset consisting of 100k video-news article pairs, as well as a benchmark of densely annotated data. We also propose a weakly supervised multimodal method which integrates commonsense knowledge from an external knowledge base (KB) to predict rich multimodal event hierarchies. Experiments show that our model outperforms a number of competitive baselines on our proposed benchmark. We also perform a detailed analysis of our model's performance and suggest directions for future research.


Introduction
Human life is eventful. We use events to describe what is happening (e.g. protest, flood, etc.), to explain its causes (e.g. police brutality led to protests), to depict our understanding of the world (e.g. during the protest someone burned property) and so on. Understanding the nature of events and their relation is thus a crucial step towards the goal of understanding our world in a nuanced and human-like way.
One way to represent such an understanding of events and their relations is through a graph structure (see Figure 1), where nodes represent events (both textual and visual) and the edges represent eventevent relations. A structured graphical representations offers many advantages. For example, it can be used to summarize text [10], perform question answering [60] and commonsense reasoning [38], predict the next event in a sequence [7], and serve as an input to aid other downstream reasoning tasks [12].
Such an event based understanding of the world and its representation in structured graphical format has been widely explored in the natural language processing (NLP) community [55,56,61]. The task is typically formulated as two distinct steps: 1) event detection [48] and 2) event-event relation prediction [55]. Recently, a similar line of research has appeared in the vision community, tackling Preprint. Under review.  Figure 1: Given a text article, existing methods output an event graph, consisting of text events as nodes and event relations as edges. For the proposed multimodal task, M 2 E 2 R, given a text article and a video, the expected output is a multimodal event graph, consisting of both text events and video events as nodes and their relations. The new multimodal edge is shown with a dashed line. visual event detection [49] and visual event-event relation prediction [44]. Unfortunately, both lines of work remain disjoint -no existing method tackles cross-modal event relation recognition. However, our world is inherently multimodal and unimodal representations fail to capture the rich cross-modal relationships which exist in this plentiful source of data. The proposed task, Multimodal Event Event Relation Prediction, M 2 E 2 R, is an effort towards bridging this gap.
Formally, we propose the novel task of Multimodal Event-Event Hierarchical Relationship (M 2 E 2 R) prediction. Given a text, annotated with textual events, and a video, annotated with video events, the task requires detection of all multimodal 'Hierarchical' and 'Identical' relations among all text event and video event pairs (see Figure 1). To support research on this task, we release a large scale dataset, M 2 E 2 R. The dataset consists of around 100.5K pairs of news article and accompanying video. We annotate a small subset (500 pairs) with all possible multimodal hierarchical and identical relations between text events and video events. This forms the test set for benchmarking and evaluation.
To overcome large scale annotation contraints, we propose a weakly supervised method -1) First generating pseudo labels using existing NLP and Vision techniques 2) Training our proposed, MERP (Multimodal Event Relations Predictor) on these pseudo labels while also leveraging commonsense knowledge from an external Knowledge Base. We evaluate our method against several baselines on our human-annotated benchmark of multimodal event relations. Our results demonstrate that our proposed approach significantly improves over a number of competitive baselines on this challenging task. We include detailed experiments ablating design choices of our model.
The major contributions of this work are: • We introduce the Multimodal Event-Event Relationship (M 2 E 2 R) recognition task.
• We contribute a large-scale dataset of 100.5K news articles and associated videos to facilitate research on this task. We provide a large-number of densely annotated samples by trained annotators to benchmark performance on this task.
• We propose a weakly supervised method for this task which significantly outperforms competitive basselines.

Related Work
Hierarchical Event Relations in Text. Detecting hierarchical event relations (or sub-event relations) is a long-standing problem in the text domain [35,16]. Early works mainly rely on heuristic phrasal patterns. For example, Badgett et al.
[2] found some characteristic phrases (e.g., "media reports" in new articles) always contain sub-events with hierarchical relations. To further enrich hierarchical event relation instances, recent works [62] rely on generative language model to generate subevent knowledge among different commonsense knowledge [5,45], then incorporate knowledge into event ontology. In this paper, we focus on the hierarchical event relations between multi-modal events.
Relation Understanding in the Vision Domain. Scene graphs [24,58,21] have been densely studied for images/videos to parse the visual scene into a graph. However, the relationships studied in scene graphs are not between two events. To the best of our knowledge, the only pioneering work has discussed the event-event relationship in video domain is VidSitu [44]. Unfortunately, to simplify the research problem on this topic, they have made several assumptions: 1) All events are manually cutted into fixed interval (2-second). 2) All event types are from predefined event ontology. Instead, in this work, we focuses on open-vocabulary event types, and variable-length video events. Besides, we focus on multimodal relations in contrast to their vision-only event relations.
Multimodal Event Understanding. Since single-modality event extraction is well studied [34,47,28,29,63,27,31,36], understanding events from multiple modalities [25,8,39,26,66,52,57] has attracted extensive research interests because different modalities usually provide the complementary information for comprehensively understanding the real-world complex events. Two important benchmarks [25,8] have been established for image + text and video + text settings. Li et al. first introduced the task of jointly extracting events and labeling argument roles from both text articles and images. An unannotated image-article dataset and a manually labeled evaluation image-article dataset are collected to train and evaluate models, respectively. Chen et al. further defined the task of joint multimedia event extraction from video and text to exploit the rich dynamics from videos. However, both the works focus on event detection in comparison to the event-event relation task explored in this work.

Task
Rrepresenting our world in terms of events and their relations (event graphs) provides a powerful abstraction towards the overarching goal of understanding our world, especially from multimodal data (text + visual). We propose the novel task, M 2 E 2 R, to generate such multimodal graphs. Multimodal event graph consist of textual and visual events as nodes and their relations as edges. Inducing such an event graph is composed of two steps -(1) detecting events from text and visual data (2) predicting relationships between these events. Event detection from both modalities has been explored before [48,49] and we use the established SOTA techniques to detect these events. The proposed task, M 2 E 2 R, focuses on multimodal event-event relationship prediction. While there are many event relations types that have been explored in NLP and Vision literature [19,61], we focus on two types of relations in this work -Hierarchical and Identical.
Events have many facets to them and are referred to on multiple granular levels in text and visual data. For example in Figure 1, we find that during the event "e 1 : Protest", subevents "v 2 : Burning of public property" and people "e 2 : Flooding" streets occurred. Hierarchical event relationship prediction is aimed at resolving such event granularities. Such an event graph containing Hierarchical and Identical relationships have many downstream applications -summarization, question answering, commonsense reasoning etc [10, 60,38,12].
Formal Task Definition Given a text article, T , annotated with text events, {e i } m i=1 , and a video, V annotated with video events {v j } n j=1 , M 2 E 2 R requires the model to predict all possible Hierarchical and Identical event-event relations, {r k } K k=1 , from a text event, e i , to a video event, v j , among all possible m × n pairs, where r k ∈ {'Hierarchical', 'Identical'} and K ≤ m × n. We will now discuss definitions for different components of the task and justifications for task design choices.
Text Event Definition Text event has been defined quite thoroughly in different NLP works on information extraction [20,41,18]. As such, we closely follow ACE Corpus's [20] definition of events. Very broadly it's defined as: 'a change of state or the changed state of a person, thing or entity.' However, ACE corpus has a very detailed definition addressing many linguistic nuances which is not required for our application. Secondly, the ACE corpus is restricted to fixed ontology of event types, whereas we annotate events in open-domain. As a result of aforementioned points, we came up with a lightweight and modified event definition and annotation instructions (detailed in Appendix A.1).
Video Event Definition Like text events, we don't restrict video events to a fixed ontology, rather we deal with them in an open-domain setting. Precisely defining events in such a setting is quite challenging due to multiple granularities on which events are depicted in videos. For example, during a 'clash' event, we can see 'pulling out baton' event and 'throwing a punch' event. This makes it difficult to pick salient event boundary in video clips. Sadhu et al. [44] circumvent this ambiguity by defining temporally fixed event boundary of duration 2 seconds. However, pre-defining the boundary duration is difficult and application specific. Additionally, it can often divide salient events into multiple segments. We address these issues by defining event boundary to be shot change, partly following Shou et al. [49]. This gives us a good trade-off between ease, clarity, consistency and non-segmentation of events, based on qualitative analysis and annotators feedback.

Relation Types
We define two types of event-event relations in this work -Hierarchical and Identical. These relation types are well defined in NLP [16] and we follow them to define the relations for our task as below: Hierarchical: 'A parent event A is said to be hierarchically related to a subevent B, if event B is spatio-temporally contained within the event A.' Identical: 'An event A is said to be identical to another event B if both events denote exactly the same real-world events.' Relation direction from text to video The relations in our multimodal event graphs are directed from text event to video events. The logic behind this design choice is that text events are often more abstract while video events are often atomic. For example, we are likely to observe abstract events such as war and election in text while their atomic subevents -fighting and voting -are more likely to be visible in the video.
Difference from Video Grounding Task One question that may be raised is -video grounding task also localize text (events) to video segments (events); so how is M 2 E 2 R different from the grounding task? The answer is that video grounding task can only provide us with 'Identical' relations. It will be difficult to expect grounding to extract 'Hierarcihcal' relations. For example, a parent event, 'strom' can be hierarchically related to subevent 'cancelled flight' [55]. However, 'strom' looks nothing like 'cancelled flight'. As such, it will be difficult for a video grounding model to predict these 'Hierarchical' relations.

Dataset
To support research on the proposed task, we introduce a large scale M 2 E 2 R dataset. We collected the dataset from news domain as the content is rich in event mentions and their relations [16,19]. In order to mitigate data bias, we chose news sources rated 'Center' by the media rating website allsides.com, resulting in a total of 9 news media sources (Appendix B.1). We scraped Youtube for news video in the channel ids of these 9 sources, collecting a total of 100.5K videos.
For each video, we also collected the associated description and closed captions. We filtered videos whose 1) duration was greater than 14 minutes; and 2) descriptions had less than 10 words. This was done to prune videos which may be too computationally expensive to process or whose description may be too short to have meaningful events. We split the data two ways -1) 100K unannotated train split for self-supervised/weakly supervised training and 2) 526 annotated test split -249 validation set and 277 test set -for benchmarking and evaluation. We annotate only a small set because annotations for this task is extremely challenging and resource consuming -two popular NLP Hierarchical event-event relations dataset [19,56] contain 100 articles each (including train split).

Train Split
The large scale train split containing 100k videos with a total duration of more than 4K hours and descriptions, which form our text data, totalling 28M words allows for training self-supervised/weakly supervised models on the task. We analyse and compare our training split against 4 popular videolanguage datasets in Table 1. Our dataset is orders of magnitude larger than any other dataset, especially in terms of video length duration and text article length (20x), allowing for rich events representation. Additionally, our dataset is the only one that includes Automatic Speech Recognition (ASR) script along with article descriptions of videos. We provide additional details in Appendix B.2.

Test Split
Annotation Procedure As the first step, following definition of video event from Section 3, we extract video events using an off-the-shelf video segmentation model -PySceneDetect. To make text event annotation easier, we provide automatically extracted text events (using [48]) to the annotators along with the instruction to add/omit events according to the definition in Section 3. Next, we task the annotators to mark all possible relations, ∈ 'Hierarchical', 'Identical', from the annotated text events to the provided video events in a video-article pair. We provide screenshots of annotation tool and additional details in Appendix B.3.
We train 5 annotators for this task through a series of short seminars, multiple rounds of feedback and consultation with all the annotators to improve consensus.
Inter Annotator Agreement (IAA) We measure the quality of the annotations using IAA. Inspired by Glavaš et al. [16] and GLAVAŠ and ŠNAJDER [17], we formulate IAA j = r 1(xrj≥2) Si| , where j ∈{'Hierarchical', 'Identical'}, S i = set of all relation annotated by annotator i as j, r ∈ ∪ 5 i=1 S i and x rj = # annotators who marked relation r as j. The intuition behind this formulation is to calculate the percentage of relations which have been annotated by at least 2 annotators. Using this formulation, we get IAA Hierarchical = 47.5 and IAA Identical = 48.9. While the IAA score may seem low, the IAA Hierarchical for NLP datasets HiEve and IC is 69 and 62 respectively. This denotes that the task in itself is quite challenging and adding multimodal aspect to it makes it even more demanding, lowering the IAA score even further.
Dataset Analysis We compare our dataset against two popular NLP event-event relations dataset -in Table 2 -as there is no existing dataset on multimodal event-event relations. M 2 E 2 R has comparable number of 'Hierarchical' and 'Identical' relations with the added novelty of being first-of-its-kind multimodal (text + vision) event-event relations dataset.

Multimodal Event-Event Relations Detection
Acquiring large scale labelled dataset for M 2 E 2 R task is time and resource consuming; not to mention the time and effort investment required to train multiple annotators (Section 4.2). As such, we propose a weakly supervised method ( Figure 2) -first, we generate pseudo labels using existing NLP and Vision techniques and secondly, we use these pseudo labels for training. We describe the method in detail below.

Pseudo Label Generation
Event Detection in Text and Video The first step is detect events in text and video separately. We use the same automatic methods to detect them as used on the test data -Open Domain IE [48] and open source library PySceneDetect 1 for text event and video event detection respectively.
Text Event-Text Event Hierarchical Relation Detection Lets assume we detected m text events, , in an article T and n video events, {v j } n j=1 , in the accompanying video V . The next step is to detect Hierarchical relations from text event, e u , to text event, e us , using Wang et al. [56], from all possible m × m pairs. Lets say we detected {e u e us } u=p,s=q u=1,s=1 hierarchical pairs, where the parent event is e u and the subevent is e us and p, q ≤ m.

Video Event Retrieval
The final step is to retrieve video events, {v us l } r l=1 and r ≤ n, from video V , which depicts the same real world event as the text subevent, e us . This step essentially simplifies to a video retrieval task. As CLIP [37] model has demonstrated state-of-the-art performance in multimodal retrieval tasks [30,11], we use it for this step. We discuss more details in Appendix C.1.
As a result of applying CLIP we get all possible video events which are identical to the text subevent: {e us v us l } r l=1 . Since e u was the parent event of e us and e us depicts the same event as {v us l } r l=1 , we can conclude that e u is the parent event of {v us l } r l=1 by transitivity. As a result, we get a total of {e u v us l } u=p ,s=q ,l=r u=1,s=1,l=1 Hierarchical pairs and {e us v us l } u=p ,s=q ,l=r u=1,s=1,l=1 Identical pairs, where p < p, q < q.
We collect additional identical pairs by directly comparing all text events {e i } m i=1 in the article T to all video events in v j n j=1 in the paired video V using CLIP. This gives an aggregate of , where m < m, n < n, identical pairs. In total, we collect 57910 multimodal hierarchical event pairs and 390149 mutimodal identical event pairs from the 100K video-article pairs training set. This forms our pseudo label.

Training
Once we obtain the pseudo labels, we proceed to training ( Figure 3) using our model, MERP (Multimodal Event-Event Relations Predictor). Given, text event, e i , and video event, v j , having a label from the pseudo label set, r ij , we follow the below described training method and loss objective.

Input Representation & Feature Extraction
The text event is represented as a word, e i , in a sentence, se i = [w 1 , w 2 , ...e i , ..w j , ...w n ]. The video event, v j , is essentially a video clip in the video consisting of n video events, {v j } n j=1 . It is comprised of a stack of frames sampled uniformly at f s frames per second rate, v j = {F t y } Z y=1 . We use CLIP to extract text event features, s image encoder and f t is a minor modification of CLIP's text encoder, f t (Appendix C.1).
Contextualizing Video Event Features So far, we have extracted video event features independent of other events in the video. But, this is limiting because a video event such as 'building destruction' could have happened because of a 'storm' event or an 'earthquake' event. As such, we use Contextual Transformer (CT) to contextualize the event features with respect to other events in the video. CT is essentially a stack of multi-headed attention layers [54]. All the video events' features from video V , {f v j } n j=1 , forms the input token to CT. We get the resulting contextualized features of video event Commonsense Features To aid learning the relationship between open domain text and video events, we incorporate commonsense knowledge from an external knowledge base, ConceptNet [51]. Inspired by Wang et al. [55], we extract events related by relations 'HasSubevent', 'HasFirstSubevent' and 'HasLastSubevent' from ConceptNet as positive pairs and random events as negative pairs. We embed the event pairs using CLIP and then, leverage the embeddings to train a feature extractor CS(., .) -a MLP (Multi Layer Perceptron) -using contrastive loss (Appendix C.2). Once trained, we freeze it and use it as a black box commonsense feature extractor while training MERP, cs ij = CS(f t i , cf v j ). Although while training MERP, one of the event is from visual modality, we are still able to use CS as it is because we extract embeddings for the video event using CLIP as well and CLIP's image embeddings and text embeddings are in the same embedding space.
Embeddings Interactions (EI) Following Wang et al. [55], we also add additional text event and video event features interactions for better representation. We add two types of features interactions -(1) Subtraction of text and video event features (SF), sf ij = f t i − cf v j and (2) Hardamard product of text and video event features (MF), mf ij = f t i * cf v j .

Multi Layer Perceptron (MLP) & Loss
We concatenate the text event feature, f t i , contextualized video event features, cf v j , commonsense features, cs ij , and embedding interactions, sf ij and mf ij , to form the input to a 2 layer MLP. The MLP layer is a 3-way classifier, outputting probabilities for e i and v j being classified as 'Hierarchical', 'Identical' or 'NoRel' (Not Related). The output is represented as . We train the model using Cross-Entropy loss between p ij and the label, r ij .

Inference
During inference, use CLIP with MERP, as an ensemble model to prune false positives for Identical relations. The rationale is to leverage CLIP's strong multimodal feature matching ability to prune out some of the video and text events falsely predicted as 'Identically' related, which CLIP can confidently predict to be not 'Identical'.

Implementation Details
Our best model uses 1 layer of multi-headed attention in CT. We make a note that the majority of text event and video event pairs are not related (94.52% in train and 93.23% in validation set). To address  Table 4: Ablation studies on components and features. "MERP Basic" represents the basic MERP model. "CT" denotes the contextual transformer. "CS" denotes the common-sense knowledge features. "EI" denotes the embedding interactions, including the subtraction and element-wise product of textural and visual features. this label bias, we weight the the labels in Cross-Entropy Loss by the inverse ratio of their count in train set as done by Wang et al. [56].
We train our model for 15 epochs using a batch size of 1024 and a learning rate of 1e-5 on 4 NVIDIA Tesla v100 gpus for a total training time of around 34 hours. We provide ablation study of our model architecture in Section 6.3 and additional details on training hyperparameters in Appendix E.

Evaluation Metric
Following event-event relations work in NLP ( [19,16], we evaluate each type of relations (Hierarchical and Identical) using Precision (P), Recall (R) and F 1 score (more details in Appendix D.1). We also report the macro average of F 1 scores (Avg. F 1 ) of Hierarchical and Identical relation types.

Baselines
Prior Baseline (Prior Base.) We use a random weighted classifier which randomly predicts a relation type (from the set {'Hierarchical', 'Identical', 'NoRel'}) for any text and video event based on the prior distribution of the relation type in the annotated labels.
Text Only Baseline (Text Base.) To evaluate the contribution of visual data in generating these multimodal event graphs, we compare our model against a Text only baseline. We construct this baseline by using ASR provided with video as a proxy for video events Appendix D.2). Proxy for a video event v j is the ASR found within the timestamps of v j , let's say X j . Now, we extract events from X j , {e jz } w z=1 as discussed in Section 5.1. Next, we use the NLP model [56] to predict relationship type, r ijz between text event in article, e i , and proxy video events {e jz } w z=1 found in video event v j . If any of the predicted relationship r ijz ∈ {'Hierarchical', 'Identical'}∀z, we propagate the relation r ijz from e i and e jz to e i and v j because e jz is a proxy for v j .
Multimodal Baseline (MM Base.) We discussed a method to predict multimodal event-event relations in Section 5.1, using SOTA NLP and vision methods, in the form of pseudo labels. This is currently the best performance that NLP and Vision can combine to give. As such, we compare the performance of our model against this multimodal baseline. We apply the aforementioned method on the validation and test to predict relations and evaluate it against the ground truth labels (Table 3).  6.3 Results

Comparison against baselines
The comparison between our MERP and above-mentioned baselines on the validation and test set are reported in Table 3, respectively. From both two tables, we have following observations: 1) For the most comprehensive metric Avg F 1 score, MERP obviously outperforms all baselines with significant performance gains (e.g., 18.0 vs. 11.4 on the validation set). 2) For Hierarchical relation type, our model achieves 4x recall over multimodal baseline (MM Base.). This is because our model can directly predict multi-modal 'Hierarchical' relationship from text parent event to video subevent. In comparison, MM Base. relies on finding the subevent in text first before retrieving the matching video subevent (Section 5.1). Absence of subevent mention in text causes it to miss numerous multimodal event relations. Meanwhile, since the multimodal baseline only produces quite a few predictions, it achieves higher precision scores. We also note the performance improvement of MM Base. over Text Base. demonstrating the criticality of visual data to this task.

Ablation Study
We conducted a set of ablations to verify the importance of different features used in our model. All results are reported in Table 4. We make several observations based on the results: 1) contextualizing video event features with respect to other video events (using CT) helps. 2) External Knowledge base (via CS features) improves understanding of open domain event-event relations. 3) All three network components -CT, CS and EI -combine to give the best performance.

Qualitative Analysis
We show some qualitative results from our model in Figure 4. The text-text event relations are derived using method described in Section 5.1 and MERP predicts the multimodal relations. These results demonstrate the advantage of MERP -we are able to predict multimodal relations in which the video subevent does not have a corresponding mention in the text article: v 2 :'shooting water cannon' on left and v 1 :'punching' on the right. This is extremely difficult with the even the best baseline, MM Base. (cf. Section 6.3.1). We provide more examples and analysis in Appendix F.

Conclusion and Future Work
We proposed the novel task of M 2 E 2 R for generating multimodal event graphs which is a powerful way to understand, represent and reason about our world. Along with the task, we introduced a large scale unannotated video-language dataset for training and a smaller annotated set for benchmarking and evaluation. We proposed a weakly supervised method to predict these multimodal event relationships, achieving an improvement of around 3x on recall and 50% on F 1 score (Hierarchical relations) over the strongest baseline, without requiring any annotations. Our method makes significant advances on the proposed task; still the best performance of 12.8 Avg F 1 score implies the immense difficulty of the task and consequently, avenues for future work. Some of them can be: 1) explicit contextual matching/contrast for text-video event; 2) dealing with open domain setting and 3) better video event detection. Further, adding more relationship types like Temporal and Causal will make event graphs more comprehensive.
As is the case with any large scale dataset, there are privacy, distribution and social bias concerns with M 2 E 2 R. We address these concerns in Appendix B.4.

A Events and Events-Events Relations
Multimodal event graphs consists of events (text and video) as nodes and relations between them as edges. For the purpose of clarity and consistency, we provide additional details on the definition of each type of events and their relations.

A.1 Text Events
Events in text denote a change of state or the changed state of a person, thing or entity. An event trigger is the word which most clearly expresses the occurrence of the event. For the purpose of annotating events in text article, we task the annotators to annotate all possible event triggers. Events triggers can be (underlined text denote event triggers): • Verbs. (eg. They have been married for three years) • Nouns. (eg. The attack killed 7 and injured 20) • Nominals. Nominals are created from verb or adjective, eg. decide -> decision.
• To resolve the cases where both a verb and a noun are possible event triggers, we select the noun whenever the noun can be used by itself as a trigger word (eg. The leaders held a meeting in Beijing). • To resolve the cases where both verb and an adjective are possible event triggers, the adjective is selected as the trigger word whenever it can stand alone to express the resulting state brought about by the event (eg. The explosion left at least 30 dead and dozens injured). • In cases where multiple verbs are used together to express an event, the main verb is chosen as the event trigger (eg. John tried to kill Mary). • In case the verb and particle occur contiguously in a sentence, we will annotate both the verb and particle as event trigger (Jane was laid off by XYZ Corp). • Sometimes, multiple events are present withing a single scope ie. the sentence. In such cases, each sentence should be annotated with trigger words corresponding to all the events (eg. The explosion left at least 30 dead).
Additionally, we ask the annotators not to annotate the following type of events (underlined text denote events not to be annotated): • Negative Events. An event is negative when it is explicitly indicated that the event did not occur (eg. His wife was sitting on the backseat and was not hurt). • Hypothetical Events. This includes the events which are believed to have happened or are hypothetical in nature (eg. Chapman would be concerned for his safety if released). • Generic Events. A generic event is any event which is contained in a generic statement (Terrorists often flee to nation-states with crumbling governments to avoid interference).
There have been violent and chaotic scenes at Manchester United's Old Trafford stadium as fans (e₁: invaded) the pitch to (e₂: protest) against the club's American owners.
T im e

A.2 Event Relation Types
There are primarily four types of relations that have been explored in the literature: identical [19,16], hierarchical [56], temporal [55] and causal [61,44]. While each of these relations are important towards building a comprehensive event graph, exploring all of them together is extremely challenging due to the annotation time required and the difficulty at distinguishing the different relation types for annotators. Despite the event relation prediction being a longstanding and established problem in NLP, no single large scale dataset exists containing all four relation types. Because the annotation and task difficulty increases substantially in the multimodal setting, we focus on the two most frequently found multimodal relation types in our human annotated data: Hierarchical and Identical relations.
Identical events are those events which denote exactly the same real world event. In contrast, two events are said to be hierarchically related if one of the event (parent event) spatio-temporally contains the other event (subevent). For this task, we constrain parent event to be a text event only to simplify the task and limit disagreements between the annotators. We provide additional illustration of the difference in these two types of relations in Figure 5.
As a corollary, a video event is 'Identical' to a text event if and only if the text event begins and ends in the video event and there is no more aspect to it, otherwise the text event is 'Hierarchical' to the video event. For example, a text event, 'meeting', will have a 'Hierarchical' relation to a video event showing two people shaking hands because the event 'meeting' has aspects other than just shaking hands like 'discussion', 'interview' etc.
Although, the direction of our Hierarchical relation if from text event to video event, we make a note that there could be event-event relations directed from video event to text event. However, we do not deal with them in this work to make the task more tractable.

B Dataset Details B.1 Dataset Curation
We consider the news domain for our dataset collection as news stories often contain rich events information along with their nuanced event-event relations. However, news articles are often opinionated and run the risk of being biased. One way of adjudging this bias is to rate the political leaning of news outlet as being left, center or right. To minimize such bias and collect only factual news, we employ media rating website, allsides.com, to select news agencies rated as 'Center'. Using this strategy, we end up with 9 news media sources: Associated Press, Axios, BBC News, Christian Science Monitor, Forbes, NPR, The Hill, Wall Street Journal and Reuters. Having finalized the news sources, we collect YouTube channel ids for all these sources. Next, we scrape videos, associated descriptions and closed captions from the collected channel ids. Videos longer than 14 minutes or having a description length smaller than 10 words were filtered out. We end up with dataset of size 100.5K samples: 100K for training and 526 for evaluation. We further divided this evaluation set into two sets: 249 for validation and 277 for testing.

B.2 Dataset Exploration
Data Statistics We provide additional data statistics on distribution of video duration, text article #words and #sentences in Figure 6. We observe that most of the videos in our dataset are at least 100 seconds long and most of the text articles have at least 10 sentences and 50 words. We compare these statistics against other video language dataset below.
We compare our dataset against 10 related video language datasets in Table 5. Our dataset is around 20x bigger in terms of total video duration and number of sentences. Datasets such as HowTo100M [33] and MERLOT [64] which only contain closed captions as textual data without accompanying   Data Topic Exploration To discover the range of news topics covered by our dataset, we employed LDA topic modeling [4] implemented in Mallet [32] following Zellers et al. [64]. Precisely, we grouped our dataset into 100 topics using a vocabulary of size 22K consisting of words which occurred in at least 25 articles but no more than 10% of articles. Further, we removed those topic which contained media source names (eg. Reuters, WSJ etc.).
We report the top 10 topics in Table 6. We see that a diverse range of topics are covered by our dataset: Election, Justice, War, International News, Entertainment etc. For a random 1000 data samples, we visualize the topic distribution via tsne plot in Figure 7 and a wordle of most prominent words in Figure 8.

B.3 Annotation Tool
For annotating the evaluation set, we used the Label Studio 2 annotation tool. We did not use the vanilla tool; rather we customized it according to our use case. The screenshots of annotation interface is shown in Figure 9.
The top row shows an overview of our interface. The text article, pre-populated with automatically extracted text events (depicted by red boxes), is on the left. The video along with a scroll bar and play/pause button is in the center. It has also been pre-populated with automatically extracted video events. These video events are shown by segmented green bars in center middle. The dark green segment shows the selected video event. The rightmost panel shows information on selected event, the list of all annotated text and video events and finally, the list of all event-event relation annotations.
The annotation task is divided into two subtasks -1) adding/deleting text events; and 2) annotating all possible 'Hierarchical' and 'Identical' multimodal relations from text event to video event. The middle rows shows the steps involved in adding a text event in blue and deleting a text event in pink.
To add text event, the annotator first needs to click on the box marked 'Text Event' at top left corner. Then, they needs to select word(s) from article to be labelled as a text event. To delete a text event, the annotator needs to first click on the red box containing the event to be deleted. Then, they need to either hit backspace key or click on the delete icon on the rightmost panel.
The bottom row illustrates the process of labelling a multimodal relation from text event to video event. To annotate a relation, the annotator needs to first click on the text event that they want to label the relation from. Then, they need to click on the relation icon on the rightmost panel. Next, the video event segment, to which the relation is directed to, needs to be clicked. Finally, the type is relation needs to be selected from the drop down menu displayed at the bottom on rightmost panel.
Once the annotation is complete, we export the labels in json format.

B.4 Privacy, Distribution and Social Impact
Privacy As discussed, we download videos from the official YouTube channels of news media companies. As such, the videos are publicly available as well as originating from a public entity. Further, we manually verified on~50 videos that the youtube descriptions do not contain any reference to individual authors; rather they contain generic links to news agency website and their social media pages. These steps taken minimizes the risk of compromising privacy/identity of any individual.
Distribution and Licensing Following data release strategy from prior work [64,33], we will be only releasing urls of the YouTube videos. This enables us to publicly release the data quickly while also minimizing potential consent and privacy issues. The usage of the data is for non-commercial research and educational purposes, which we believe constitutes 'fair use'. However, our data release strategy is aimed at further reducing any potential licensing issues.
Social Impact Our dataset have been sourced from news domain to ensure richness of events. However, as news media tend to be opinionated, and sometimes sensationalized, the dataset risks suffering from racial, gender, cultural and religious bias. As discussed in previous sections, we consciously took steps to reduce these biases by making an effort to collect factual news from 'Center' rated news media sources.

C Model Components Details C.1 CLIP For Video Event Retrieval
As discussed in the main paper, we employ a weakly supervised training method due to supervised training data constraints. As the first step in this strategy, we collect pseudo labels. Given a paired text article and video, we first extract all text events {e i } m i=1 and all video events {v j } n j=1 . Next, we detect all hierarchical pairs in text, {e u e us } u=p,s=q u=1,s=1 , using Wang et al. [56]. Here, e u is the parent event and e us is the subevent. Finally, we employ CLIP to match text subevent e us with a video event so that we can propagate the hierarchical relation from text subevent to video subevent, giving us the  Table 7: Ablation studies on components and features. "MERP Basic" represents the basic MERP model. "CT" denotes the contextual transformer. "CS" denotes the common-sense knowledge features. "EI" denotes the embedding interactions, including the subtraction and element-wise product of textural and visual features. 18.0 required multimodal hierarchical relation (pseudo label) from text parent event to video subevent. We describe the precise approach for this matching step below.
CLIP is a multimodal transformer model which encodes text into an embedding using a transformer architecture, f t (.), and separately encodes image into an embedding, in the same embedding space as the text embedding, using another transformer architecture, f i (.). It then scores the alignment of text and image using a distance scoring function -typically cosine -between encoded text embedding and encoded image embedding.
While encoding text, CLIP brackets the input sentence with [SOS] and [EOS] tokens and outputs the activation of the [EOS] token from the highest layer of text transformer as the embedding representation for the whole sentence. However, we want the embedding of the event word contextualized by words around it. To this end, we change the attention masking of [EOS] token in the last layer of text transformer from being uniform to all words in the sentence to focusing on the event word with a polynomially decreasing attention mask depending on the distance of the words in sentence from the event word. We represent this modified text encoder by f t (.).
We encode the text event e us by inputting the sentence containing the event word, se us = [w 1 , w 2 , ..., e us , ..., w i , ...w n , to f t to get f t (se us ). To encode the video event, v us l , we first represent it as a stack of frames sampled at some f s frames per second, v us l = {F us l y } Z y=1 . Next, we encode each frame using the image encoder, f i (.), and aggregate the contribution of each frame to get the video event embedding: 1 Z Z y=1 f i (F us l y ). We say that the text event, e us , represent the same event as the video event, v us l , if: λ is a threshold that we arrived at by fine-tuning on the multimodal event pairs which had 'Identical' relations from 100 video article sample pairs in the validation set. The threshold found, λ = 30.39 gave a precision value of 75% on the validation set. Using eq. (1), we arrive at the required text event -video event matching pairs, {e us v us l } r l=1 .

C.2 Commonsense Features from Knowledge Base
We leverage commonsense knowledge in the world to help us predict multimodal relations between open domain events. To this end, we use ConceptNet [51], a large scale knowledge base containing concepts, entities and events as nodes and the relation between them as edges. We extract positive and negative event pairs from ConceptNet and use it to train the commonsense feature extractor, CS(., .). CS(., .) is a single linear layer which concatenates the embeddings of the event pairs to form its input. It outputs a 512 dimensional embedding as commonsense features, which is trained using contrastive loss. These extracted features characterize how 'Hierarchical' event pairs should relate to each other.

C.3 Implementation Details
We provide additional implementation details in this section. • To extract video event features, we sample frames uniformly from the video event clip at a rate of f s . f s in this work was fixed at 3 frames per second.
• The video event embeddings extracted using CLIP forms the token embedding to be fed to the Contextual Transformer (CT). These embeddings are concatenated by positional embeddings, as is common with transformer architectures ( [54]). Further, we cap the maximum number of video events that CT can process at 77 to limit the computational cost.
• As CLIP has demonstrated strong perormance in co-relating multimodal data, we use CLIP to remove false positive 'Identical' relations from model prediction. To implement it, we use eq. (1) with a threshold of 28.0. Event pairs scoring below this threshold are pruned. This threshold was again arrived at using the validation set.

D.2 Text Only Baseline Details
We construct text only baseline by using video ASR as proxy for video itself -each video event is represented by the ASR content within the timestamps of the video event. Now, events and eventevent relations with text article can be extracted from ASR using NLP models. However, we don't use raw ASR directly. We found it difficult because they are unpunctuated and the NLP models to detect events and event-event relations have been trained on punctuated data. As such, we punctuate ASR using [1].
We observed the Text baseline to have a low recall in Table 7. On manual analysis, we found three reasons for it: 1) a number of video events do not associated ASR content; 2) some video events are very short in duration which limits the ASR content to a very few words; and 3) text events detected in ASR sometimes don't provide useful information about the events in video (eg. an event such as 'mentioned' detected in ASR). We also observe that the contribution of contextual transformer is most significant towards the full model performance. This makes sense because contextualizing a video event with respect to other video event can significantly change its interpretation. For example, a video event showing crowd of people on streets could be demonstrators agitating against the government or cheering on their newly elected leader depending on the context.

Influence of Number of Layers in Contextual Transformer.
To further investigate the influence of different number of layer n in the contextual Transformer, we conducted a set of ablations by setting n = {1, 2, 4, 6}, and results are reported in Table 8. As can be observed, all different layer settings can achieve good Avg F 1 score, which demonstrates robustness of our model. Meanwhile, n = 1 achieves slightly better results. The reason could be that more layers cause some over-fitting. As such, we use n = 1 in setting in our best model.
Pruning Strategy As shown in Table 9, the pruning strategy can slightly improve the F 1 score on identical event relations, and maintain the performance on hierarchical relations.
Dealing with Noisy Labels As discussed in the main paper, our training labels are actually pseudo labels derived using other NLP and Vision models. As such, the labels are naturally noisy. To deal with this noise, we employed Multi Instance Learning (MIL) [6] training objective (Table 10). However, we found that it didn't improve performance. One possible reason could be the fact that our large scale training data compensates for the inherent noise in labels. Such outcome have also been observed previously by Jia et al. [22].

F Qualitative Analysis on predicted samples
We provide additional qualitative results and visualizations in Figure 10. As can be seen, our model is able to enrich text event graph with new visual events nodes and different multimodal relations from diverse topics (explosion and protest). This is especially emphatic because our model was trained in an open domain setting and was not specifically trained to detect events from these topics.
Further, we compute heatmap visualization for illustrating the attention of the model while making predictions. We use Grad-CAM for this purpose [46,15]. Through these visualizations, we provide evidence that the model is indeed learning meaningful co-relations between text and visual events instead of learning some statistical prior. For example, in top row we can see that the model is focussing on smoke while predicting it's hierarchically related to explosion.

G End-to-End System Evaluation: Event Detection + Relationship Prediction
Multimodal Event graphs consists of text and video events as nodes and the event-event relations relations as edges. The proposed task in the paper, M 2 E 2 R, focuses on prediction and evaluation of multimodal event-event relations, given a text and video event.
The assumption is that we are e₁: explosion Radical groups in Georgia tried to prevent a (e₂: rally)… Riot police were (e₃: stationed) in between the two opposing rallies to prevent violence... provided with ground truth text and video events. However, the assumption doesn't hold for 'in the wild' text-video pair. To generate multimodal event graphs for such an input, we need to predict text and video events first and then predict the relations between them. As such, we evaluate our model against this combined task as well -event detection and event-event relation prediction.
In this setting, the model is first required to predict text event and then predict multimodal eventevent relation between the predicted text event and given video event. We don't task the model to predict video events because our video events are actually camera shots by definition and are not annotated by the annotators. The evaluation is then done between the predicted multimodal event-event relations and ground truth multimodal event-event relations. The evaluation metric remains the same -Precision, Recall and F 1 . We call this setting IETe2Ve (Information Extraction + Text event to Video event relation prediction) and the earlier setting of predicting only multimodal event-event relations Te2Ve(Text event to Video event relation prediction).
In our model, we use Shen et al. [48] to extract text events and then predict the multimodal relations the usual way. For Text Base. and MM Base. as well, we again use Shen et al. [48] as the first step to detect text events. The results are reported in Table 11. MERP obviously performs worse now than in Te2Ve setting (7.8 vs 12.8 on test set). However, it still performs better than next best baseline by 1%.
We note the extreme challenge posed by this setting -predicting open domain events and then predicting relations between it and open domain video events. This is also evidenced by extremely low performance of Prior Base. -0.1 Avg F 1 down from 1.62 in Te2Ve setting. This is so because in IETe2Ve, it must consider all words in text article as possible events. This makes the prior of 'NoRel' very high, significantly reducing number of relations predicted as 'Hierarchical' and 'Identical'. We also note the low performance of Text Base. -this is due to the same reasons discussed in Appendix D.1.

H Discussion
As discussed in the main paper, one of the biggest advantage of our proposed model, MERP, over even the best baseline, is the ability to predict multimodal relations where the video subevent is not mentioned in text article. We provide quantitative results as further evidence -MERP has a recall of 15.92 for such multimodal Hierarchical relations as against 2.14 for the baseline. We arrive at this result by evaluating the models against those multimodal Hierarchical relations for which the video subevent was not annotated with a corresponding Identical text event.
Performance Bottleneck. One major reason is that we are dealing in an open domain setting. While, there are many advantages to having an open domain of video and text events, it makes understanding new events and then predicting their relationships very challenging.

I Datasheet for M 2 E 2 R
In this section, we present a DataSheet [13, 3] for M 2 E 2 R, synthesizing many of the other analyses we performed in this paper.

Motivation For Datasheet Creation
• Why was the dataset created? In order to study cross-modal event event relations.
• Has the dataset been used already? No.
• What (other) tasks could the dataset be used for? Possibly visual grounding and multi-modal schema induction.

Data composition
• What are the instances? The instances that we consider in this work are pairs of news articles and associated videos . • How many instances are there? We include 100.5K news articles and associated videos.