{'fraction_non_alphanumeric': 0.03390045306212971, 'fraction_numerical': 0.014257199885942401, 'mean_word_length': 4.605398685846208, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Scene Graph Modification as Incremental Structure Expanding

A scene graph is a semantic representation that expresses the objects, attributes, and relationships between objects in a scene. Scene graphs play an important role in many cross modality tasks, as they are able to capture the interactions between images and texts. In this paper, we focus on scene graph modification (SGM), where the system is required to learn how to update an existing scene graph based on a natural language query. Unlike previous approaches that rebuilt the entire scene graph, we frame SGM as a graph expansion task by introducing the incremental structure expanding (ISE). ISE constructs the target graph by incrementally expanding the source graph without changing the unmodified structure. Based on ISE, we further propose a model that iterates between nodes prediction and edges prediction, inferring more accurate and harmonious expansion decisions progressively. In addition, we construct a challenging dataset that contains more complicated queries and larger scene graphs than existing datasets. Experiments on four benchmarks demonstrate the effectiveness of our approach, which surpasses the previous state-of-the-art model by large margins.


Introduction
A scene graph is a structural representation that captures the semantics of visual scenes by encoding object instances, attributes of objects, and relationships between objects. (Johnson et al., 2015). As shown in Figure 1, the scene graph encodes objects (e.g. "Boy", "Racket"), attributes (e.g. "Girl is standing"), and relations ("Boy holding racket"). Scene graphs are able to capture the interactions between text and images by associating objects in the graph with regions of an image and modeling the relations between objects. Therefore, it has been used in the cross modality task such as image retrieval, image captioning, and visual question answering (Schuster et al., 2015;Shi et al., 2019;Wang et al., 2020b).
Recently, modifying the scene graph based on the input becomes an emerging research direction as cross-modal systems may need to resort to an interactive process through multiple iterations (Ramnath et al., 2019;He et al., 2020). Take text-based image retrieval as an example, users start with a query describing the main objects or topics they are looking for, then modify the query to add more constraints or provide additional information based on previous search results. Instead of directly manipulating images, scene graphs can be used to convert the image-editing problem into a graphediting problem, conditioned on the textual query. As shown in Figure 1, given a retrieved image from the last turn, if the user wants to see a girl rather than a boy holding a racket, he will enter the query "I would like to see a girl holding racket" to the system. According to the query, the object "Boy" in the original scene graph will be substituted with the object "Girl". The target image can be retrieved Figure 2: Examples of basic operations INSERT and DELETE for scene graph modification. Q denotes the textual query, G S denotes the source scene graph, G T denotes the target scene graph and G I is the extended graph.
given the updated scene graph. The key challenge in this process is how to modify the corresponding partial structure in the original scene graph based on understanding the natural language query.
Prior effort framed this scene graph modification (SGM) task as conditional graph generation (He et al., 2020), where the scene graph is generated from the scratch condition on the original graph and query (You et al., 2018;Guo et al., 2019;Cai and Lam, 2020b). However, rebuilding the entire scene graph may not be an optimal solution, as the model has to generate the partial structure of the original graph that should be unmodified. Moreover, nodes and edges of the scene graph are constructed separately in their proposed framework, which generates all the nodes first then attaches edges between generated nodes in the second pass. Such an approach may lead to the lack of the modeling capability of interactions between node prediction and edge prediction.
Instead of rebuilding the whole scene graph, we introduce a novel formulation for SGM -incremental structure expanding (ISE), which is able to build the target graph by gradually expanding the original structure. At each step, ISE generates the connecting edges between the existing nodes and the newly generated node, upon which the type of the new node is jointly decided. Based on the formalism, our proposed model is able to iterate between finding the relevant part in the query and reading the partially constructed scene graph, inferring more accurate and harmonious expansion decisions progressively. Experiments on three SGM benchmarks demonstrate the effectiveness of the proposed approach, which is able to outperform previous stateof-the-art models by large margins. To test the ability of a model under a complex scenario, we further construct a more challenging dataset from the remote sensing domain (Lu et al., 2017), which has much more modification operations based on the more complicated queries compared with the existing scene graph modification datasets. Our key contributions are summarized as follows: • We propose a novel formulation for scene graph modification, allowing incremental expansion of the source scene graph rather than the regeneration of the target graph.
• We further construct a challenging dataset that contains more complicated queries and larger scene graphs. Extensive experiments on four SGM datasets show the effectiveness of our proposed approach.
• Experiments on four benchmarks demonstrate the effectiveness of our approach, which surpasses the previous state-of-the-art model by large margins.

Incremental Structure Expanding
In scene graph modification, a node or multiple nodes can be inserted to, deleted from or replaced with other nodes in the scene graph. He et al. (2020) defined the scene graph modification task as a conditional graph generation problem. Formally, given the source scene graph G S and the natural language query Q, the target scene graph G T is generated by maximizing the conditional probability p(G T |G S , Q). Instead of generating the entire target graph G T , we frame the task as an incremental structure expanding, which extends the source scene graph G S one node at a time, as well as the edges associated with the node. Such a formulation does not require the model to rebuild the unmodified structure of the source scene graph.
Under this formulation, we first define two basic operations: INSERT and DELETE. Scene graph modification can be viewed as combining and applying these two operations multiple times. For- mally, given the query Q, a sequence of n operations a 1 , a 2 , ..., a n are selected from a set of graph modification operations A = {INSERT, DELETE}. After applying the operations to the source scene graph G s , the target scene graph G t is derived. Each operation is defined as: • INSERT: A new node o is added to G s , and edges are attached between o and existing nodes in G s . As shown in Figure 2 (a), the node "Ball" is added to G s and an edge between "Ball" and "Holding" is attached, according to the query "holding a racket and a ball".
• DELETE: As shown in Figure 2 (b). A node o is removed from G s , as well as its associated edges. As shown in Figure 2 (b), the node "Blue" is removed from G s and the edge between "Racket" and "Blue" is removed either, according to the query "a racket not a blue".
Inspired by incremental parsing (Nivre, 2004;Dyer et al., 2015;Cai and Lam, 2020a;Zhang et al., 2021Zhang et al., , 2022, we design a data structure called extended graph G I , which can be used to model INSERT and DELETE under the graph expansion setting. As shown in Figure 2 (a), the extended graph G T is identical with the target graph G T after applying INSERT. As for DELETE, we introduce a dummy node "Delete", which is attached to the node in the source graph G S that should be removed. For example, the dummy node "Delete" is attached to the node "Blue" in G T . In the postprocessing stage, nodes attached with the dummy node "Delete" will be removed. Using this formulation, we are able to model scene graph modification by incrementally expanding the source graph G S to the extended graph G I , which can be converted to the target graph G T without any losses.
If the modification requires multiple operations, there will exist multiple node orderings. Take node substitution as an example, replacing a node o i with o j in G s can be viewed as DELETE the node o i first, then INSERT the node o j , or vice versa. In practice, we impose that the DELETE operation always comes before INSERT, then the breadthfirst search is used to define a deterministic node ordering.

Model Architecture
In this section, we will present the model based on the incremental structure expanding formulation. Figure 3 gives an overview of the proposed model, which consists of five components including query encoder, graph encoder, feature fusion, edge decoder and node decoder.
Query Encoder This module is used to encode the query Q by generating the representation of each token of it.
Graph Encoder This module is used to encode the graph by generating the representation of each node of it. Note that the representations of the graph are constructed incrementally during the expanding progresses based on the updated graph of the last time step. The graph is the source graph G S at the first timestep.
Feature Fusion this module aims to combine the representations from query and graph encoder, then served as a writable memory, which is updated based on the information from edge and node decoder during the incremental expansion.
Edge Decoder this module is used to predict the edges between the newly generated node and existing nodes of the graph, then update the memory of the feature fusion module with edge information.
Node Decoder this module is used to generate a new node of the graph, then update the memory of the feature fusion module with node information.

Query Encoder & Graph Encoder
For fair comparisons with the previous work (He et al., 2020), our query encoder and graph encoder are based on the vanilla transformer (Vaswani et al., 2017), which consists of multi-head self attention (MSA) and position-wise feed-forward network (FFN) blocks. The FFN contains two layers with a ReLU non-linearity. Layer normalization (Ba et al. 2016) is applied before every block, and residual connections (He et al., 2016) after every block.
Formally, given an input query Q with n tokens, each token embedding is randomly initialized and positional encoding is added to the token embedding to retain positional information. The resulted embeddings are denotes as x = {x 0 , x 1 , ..., x n }. Similar to BERT (Devlin et al., 2019), a special token is appended to the query as x 0 for sentence encoding. Transformations in the query encoder can be denoted as: After stacking L blocks, we obtained the contextualized token representations from the query encoder, denoted as {x L 0 , x L 1 , ..., x L n }. The first vector x 0 is treated as the sentence-level representation of the query and will be used as the initial state during expansion. For clarity, we denote the vectors as x∈R (n+1)×d , where d is the dimension.
As for the graph encoder, we treat the input graph as a sequence of nodes in the chronological order of when they are inserted into the graph as discussed in Section 2. Formally, given the graph G t at the time step t, we take its node sequence {o 1 , o 2 , ..., o t−1 } as the input. A transformer architecture is also applied to obtain the contextualized node embeddings. Notice that the contextualized representation of the graph is constructed incrementally as the expanding progress. Therefore, we apply the vanilla transformer with masked self-attention as the graph encoder, which only allows each position in the node sequence to attend to all positions up to and including that position. For brevity, we denoted the resulted contextualized node representations as y∈R m×d .

Feature Fusion
Unlike the conventional sequence-to-sequence model that only has one encoder, our model contains two encoders. Previous work (He et al., 2020) proposed to use gating mechanism and cross attention to combine the representations of resulted representations from query and graph encoders. We choose to use vanilla multi-head attention mechanism (Vaswani et al., 2017) to fuse the features from these encoders. Formally, at each time step t, the feature fusion component combines the query and graph representations for gradually locating and collecting the most relevant information for the next expansion: z l Finally, the edge decoder passes the edge information to the feature fusion module by updating the expansion state h t+1 as follows:

Node Decoder
The node decoder needs to look at the input query and determine which tokens are the most important ones. This choice is a weighted matrix that gives an attention probability between each token in the query and generated nodes in the target graph.
Concretely, a single-head attention α s t is computed based on the state h t and the sentence representation s 1:n , where α s t denotes the attention weight of the word w i in the current time step. This component then updates the parser state with the alignment information via the following equation: We then compute the probability distribution of the new node through a hybrid of two channels. The new node can either be a DELETE node or a token copied from the input query. First, h t is fed through a sof tmax to obtain a probability distribution over a pre-defined vocabulary, which contains the DELETE node and other dummy nodes such as EOS. The probability of the new node is calculated Second, we used the attention scores α s t as the probability to copy a token from the input query as a node label similar to the copy mechanism See et al., 2017). Therefore, the final prediction probability of a node o is defined as: where [i] indexes the i-th element, and T (c) are index sets of tokens respectively that have the surface form as o. P (gen) and P (copy) are the probabilities of generating and copying a node, respectively. They are computed by using a single layer neural network with sof tmax activation as: The whole expanding procedure is terminated if the newly generated node is the special node EOS.

Statistics
User Generated MSCOCO GCC RSICD

Dataset Construction
Existing SGM datasets are synthetically constructed based on scene graphs from MSCOCO (Lin et al., 2014) and GCC (Sharma et al., 2018a), and via crowd sourcing. To construct scene graphs, He et al. (2020) used an in-house scene graph parser to parse a random subset of MSCOCO description data and GCC captions, thus the constructed scene graph is relatively simple. In Table 1, the average numbers of nodes and edges for each graph are limited to 2.9 and 1.9 respectively. GCC is more complicated than MSCOCO with a larger graph, but the percentage of nodes and edges from the development/test set that does not appear in the training set (OOV Nodes, OOV Edges) are still low, which will cause the model easily overfit to the dataset. To verify the generalization ability and the scalability of the model to handle more complex scene graphs, we constructed our own Scene Graph Modification dataset based on the Remote Sensing Image Captioning Dataset (RSICD) (Lu et al., 2017) in the remote sensing field for remote sensing image captioning task. Inspired by the modification methods proposed by He et al. (2020). First, we adopt the parser (Schuster et al., 2015) to parse the caption for each graph and generate the original scene graph x. Then we define three types of graph modification operations A = {INSERT, DELETE, SUB-STITUTE}, and randomly apply them to the original scene graph to generate query (q) and modified scene graph (y). The data in RSICD consists of the triples (x, y, q). 2 Compared with the existing SGM dataset, each graph of RSICD has more nodes and edges, with an average of 5.9 and 3.7 on the training/development/test set, which is almost twice that of User Generated and MSCOCO. In addition, the dataset comes from the field of remote sensing. Due to the large

Data
We evaluated our model on four benchmarks, including User Generated, MSCOCO and GCC proposed by He et al. (2020), and RSICD dataset proposed in this work. MSCOCO, GCC and RSICD are constructed synthetically from publicly available datasets (Lin et al., 2014;Sharma et al., 2018b;Lu et al., 2017), while the User Generated dataset is created via crowd sourcing. Detailed statistics of datasets are shown in Table 1.

Setup
For fair comparisons, we used the same data splits for User Generated, MSCOCO and GCC datasets as in Weber et al. (2021). For RSICD, we randomly split the data into 8K/1K/1K for training/development/test. Following Weber et al. (2021), we use three automatic metrics for the evaluation, including node-level and edge-level F1 score, and graph-level accuracy. Graph-level accuracy is computed based on exact string match, which requires the generated scene graph to be identical to the target scene graph for a correct prediction. We reported the mean score and standard deviation by using 5 models from independent runs. We refer to the Appendix A.2 for the detailed implementation.

Baselines
For comprehensive comparisons, we include six baselines as follows. Except for the CopyGraph, all of them aim to rebuild the target scene graph.
CopyGraph This baseline directly copies the source scene graph as the target scene graph, which can be viewed as the lower bound.
Text2Text This baseline is introduced by He et al. (2020). They used the standard sequenceto-sequence architecture by linearizing the scene graph based on depth-first search.
GRNN Graph RNN (You et al., 2018) is used as the graph encoder and edge decoder. Specifically, the edges are represented by an adjacency matrix, which is then generated in an auto-regressive manner. Both the query encoder and node decoder are based on Gated Recurrent Units (Cho et al., 2014).
DCGCN Densely-Connected Graph Convolutional Networks (Guo et al., 2019) are used as the graph encoder. Other components are kept the same as the GRNN.
GTran Graph Transformer (Cai and Lam, 2020b) is used as the graph encoder, while other modules are the same as GRNN and DCGCN.
STran The sparsely-connected transformer (He et al., 2020) is used to encode the source graph. In addition, a cross-attention mechanism is applied to fuse the features from graph encoder and query encoder. Node decoder and edge decoder are the same as GRNN.
EGraph This is the state-of-the-art model on graph modification task. Concretely, Weber et al. (2021) considerably increases performance on the graph modification by phrasing it as a sequence labelling task.   (Weber et al., 2021).

Main Results
According to Table 2, our proposed approach (ISE) significantly outperforms the state-of-theart model (Weber et al., 2021) on three datasets. Specifically, ISE outperforms EGraph 1.81, 1.11 and 1.33 percentage points in terms of graph accuracy on User Generated, MSCOCO and GCC datasets, respectively. We observe that the improvement is especially prominent on the User Generated dataset, which is more challenging than the other two synthetic datasets in terms of the diversity in graph semantics and natural language expressions. All baseline models suffer from performance degradation as it is much harder to rebuild the entire target scene graph on this dataset. On the other hand, ISE constructs the target scene graph by incrementally expanding the source scene graph without changing the unmodified structure. We believe this formulation is able to effectively cope with this difficulty. We also observe that both EGraph and ISE achieve lower graph accuracy on the GCC dataset. The main reason is the difficulty of predicting the correct edges between generated nodes. For example, EGraph achieves 98.62 Node F1 score on GCC, higher than 97.62 Node F1 score on the User Generated dataset. However, EGraph only achieves 75.01 Edge F1 score on GCC, while it can attain 88.26 Edge F1 score on User Generated. Our proposed model has larger improvements upon EGraph in terms of Edge F1 score on the same dataset (93.06 vs. 91.64). We attribute this stronger improvement to iterations between nodes prediction and edge prediction, which allows more accurate and harmonious expansion decisions progressively. On the other hand, EGraph predicts nodes and edges at two independent stages. Such an approach may lead to the lack of the modeling capability of interactions between node prediction and edge prediction.
We further compare our model with EGraph on  Table 4: An ablation study for ISE. Rebuild denotes that we regenerate the scene graph rather than extend it. -Copy denotes model without using the copy mechanism the newly constructed dataset RSICD as shown in Table 3. ISE is able to achieve a graph accuracy of 44.20% and improves upon the EGraph model by 21 percentage points. However, the graph accuracy of all the models is much lower than the one attained on the previous three SGM datasets. One reason is that RSICD has more complex queries paired with larger scene graph, which brings a challenge to existing models. The RSICD dataset also suffers from the data sparsity issue where many words (39%) and nodes (42%) only appear once in the training data. Incorrect node prediction will further propagate the errors to edge prediction. Our iterative node and edge prediction paradigm help to alleviate this issue. Specifically, ISE only outperforms EGraph 9.69 percentage points on Node F1 score, while the improvement on Edge F1 score is 13.05%. Therefore, ISE is able to achieve a higher accuracy. In order to further address this data sparsity issue, one potential solution is transfer learning, where the model is pretrained on User Generated dataset first then fine-tuned on RSICD. However, this approach may suffer from a domain-shift problem, as RSICD is constructed based on the remote sensing domain. We leave this direction as future works.

Analysis and Discussion
In this section, we provided a fine-grained analysis of our proposed model. We reported all the results on the development set by using the ISE model without contextualized embeddings from BERT.
Ablation Study As shown in Table 4, we examine the contributions of two main components used in our model. The first one is the incremental structure expanding. We use the same model architecture but try to rebuild the target scene graph   similar to previous efforts. We can observe significant drops on three SGM datasets, which further confirms the effectiveness of the extending strategy. The second one is the copy mechanism, which directly copies the token from the query as nodes in the target scene graph. It plays a significant role in predicting nodes especially when the training data is limited (User Generated). Table 5 shows the performance of STran and ISE against different training settings on MSCOCO dataset. We considered four training settings (20%, 40%, 60%, 80%, 100% training data). ISE consistently outperforms STran under the same amount of training data. When the size of training data decreases, we can observe that the performance gap becomes more obvious. Particularly, using 40% of the training data, ISE is able to achieve a graph accuracy of 88.64%, higher than STran trained on the whole dataset. These results demonstrate that our model is more effective in terms of using training resources and more robust when the training data is limited.  Performance against Query Length Table 6 shows the results of STran and ISE under different query lengths on GCC dataset. We partitioned the sentence length into three classes (<5, [5, 10), ≥10).

Performance against Training Data Size
In general, ISE outperforms STran against various sentence lengths. When the length of the query increases, we can observe that the performance gap becomes more obvious in terms of graph accuracy. Intuitively, with the increase of the query length, it is more challenging for the model to comprehend the sentence. This suggests that ISE is able to handle more complex instructions. Table 7 shows the results of STran and ISE against different target scene graph sizes on GCC dataset. We partitioned the scene into three classes (<5, [5, 10), ≥10). Based on the formulation of extending the source scene graph, our model is required to deal with larger graphs. For example, deleting a node in the scene graph becomes adding a special "Delete" node in the extended graph. However, ISE is able to consistently outperform STran against various target graph sizes, even when the target scene graph is large. This result suggests the superiority of the proposed formulation. 3

Performance against Graph Size
Case Study We give two cases in Figure 4. STran generates scene graph from the scratch conditioned on the original graph and query may lead to the lack of the modeling capability of interactions between node prediction and edge prediction. For example, in Figure 4 (a), STran omitted the attribute: "Velvet" during the node prediction. In addition, during the edge prediction, STran redundantly generated the relation: "Of" in Figure 4 (b). However, these structures do not need to be modified in the source scene graph. ISE can infer more accurate target graph by incrementally expanding the source graph without changing the unmodified structure. : Two cases of STran and ISE for scene graph modification on User Generated. Q denotes the textual query, G S denotes the source scene graph, G T denotes the target scene graph generated by STran and ISE.

Related Work
We refer to the Appendix A.3 for the detailed related work of scene graph. Scene graph builds a bridge between image domain and text domain. Vision and natural language are all tremendously promoted by studying into scene graphs. Recently, scene graph modification becomes an emerging research direction. Chen et al. (2020) proposed a framework based on scene graph editing for textbased image retrieval. On the other hand, He et al. (2020) took the scene graph and the textual query as inputs and modified the source graph according to the query. They viewed the task as conditional graph generation, which is further decomposed into node prediction and edge prediction. For node prediction, all the nodes in the target scene graph is generated based on a graph-to-sequence model with dual encoder Beck et al., 2018;, then a graph RNN is adopted to predict the edges between generated nodes (You et al., 2018). More recently, Weber et al. (2021) developed an alternative formulation of this problem in which they model the modification as an auto-regressive sequence labelling task. Instead of rebuilding the entire target graph, we framed the scene graph modification task as incremental graph expansion. This formulation is related to incremental parsing, where a sentence is scanned from left-to-right and the structured is built incrementally by inserting a node or attaching an edge. Incremental parsers are widely used in semantic parsing (Zhou et al., 2016;Cheng et al., 2017;Guo and Lu, 2018;Naseem et al., 2019;Liu et al., 2022a) and syntactic parsing (Huang and Sagae, 2010;Dyer et al., 2015;Liu and Zhang, 2017), as they are computationally efficient, and can use machine learning to predict actions based on partially generated structures. Our feature fusion module can be viewed as the parser state as it carries the structural information and serves as a writable memory during the expansion step. Unlike Weber et al. (2021) linearize the scene graph and label it in an auto-regressive manner, our model iterates between finding the relevant part in the query and reading the partially constructed scene graph, inferring more accurate and harmonious expansion decisions progressively.

Conclusion
In this paper, we designed a novel formulation for scene graph modification, which allows us to incrementally expand the source scene graph instead of rebuilding the entire graph. Based on the formalism, we further propose a model that is able to leverage the mutual causalities between node prediction and edge prediction. Experiments on three SGM benchmarks demonstrate the effectiveness. To test our model under a complex scenario, we constructed a more challenging dataset from the remote sensing domain, which has more modification operations based on the more complicated queries compared with existing SGM datasets. For future work, we would like to explore how to integrate the model into the text-based image retrieval task.

A.1 Operations in RSICD
We introduce three operations in RSICD in details: • DELETE: The original scece graph is x. We randomly select a node o in x, and delete it both with related edges. The deleted graph is defined as y. We choose a random sentence from the DELETE Template (Manuvinakurike et al., 2018), for example, " I do not want **." We replace ** with o to get modification operation q.
• INSERT: It is the reverse process of DELETE.
The graph before deleting the node is regarded as y, and the corresponding graph after deletion is treated as x. The modification operation is randomly selected from the INSERT Template (Manuvinakurike et al., 2018), for example, " Show me **." We replace ** with o to obtain query q.
• SUBSTITUTE: We randomly select a node o, use the AllenNLP toolkit (Gardner et al., 2018) to find the three most similar semantics nodes compared with o. We randomly choose a node m, and select a sentence from the SUBSTITUTE Template (Manuvinakurike et al., 2018), for example, " I prefer @@ to **, modify ** to @@." We replace ** and @@ with o and m, and get modification operation q. Note that SUBSTITUTE operation could be viewed as DELETE the node o first and then INSERT the node m, or vice versa. In

A.2 Implementation Details
Hyper-parameters of the model are tuned on the development set. All transformer (Vaswani et al., 2017) layers share the same hyper-parameter settings. Following He et al. (2020), we randomly initialized the word and node embeddings. We also report results with contextualized embeddings from BERT (Devlin et al., 2019). Specifically, we used the BERT-base-uncased implemented by (Wolf et al., 2020). The parameters in BERT are fixed during training. To mitigate over-fitting, we apply dropout (Srivastava et al., 2014) with the drop rate 0.2 between different layers. Following Cai and Lam (2020a), we use a special UNK token to replace the out-of-vocabulary lemmas of the input query and remove the UNK token in the generated graph. Parameter optimization is performed with the ADAM optimizer (Kingma and Ba, 2015) with β 1 = 0.9 and β 2 = 0.999. The learning rate schedule is similar to that in Vaswani et al. (2017), where warm-up steps being set to 2K. We used early stopping on the development set for choosing the best model. Please refer to Table 9 for the detailed hyper-parameters settings for ISE.

A.3 Scene Graph and Application
Deep learning has significantly promoted the advancement of computer vision (Liang et al., 2017;Ren et al., 2021). Simple visual understanding tasks such as object detection and recognition are no longer sufficient. To depict the relationship between objects in the scene as a driving force,