{'fraction_non_alphanumeric': 0.039242636746143054, 'fraction_numerical': 0.013744740532959326, 'mean_word_length': 4.858833196384552, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement

Grounded Situation Recognition (GSR) aims to generate structured semantic summaries of images for"human-like"event understanding. Specifically, GSR task not only detects the salient activity verb (e.g. buying), but also predicts all corresponding semantic roles (e.g. agent and goods). Inspired by object detection and image captioning tasks, existing methods typically employ a two-stage framework: 1) detect the activity verb, and then 2) predict semantic roles based on the detected verb. Obviously, this illogical framework constitutes a huge obstacle to semantic understanding. First, pre-detecting verbs solely without semantic roles inevitably fails to distinguish many similar daily activities (e.g., offering and giving, buying and selling). Second, predicting semantic roles in a closed auto-regressive manner can hardly exploit the semantic relations among the verb and roles. To this end, in this paper we propose a novel two-stage framework that focuses on utilizing such bidirectional relations within verbs and roles. In the first stage, instead of pre-detecting the verb, we postpone the detection step and assume a pseudo label, where an intermediate representation for each corresponding semantic role is learned from images. In the second stage, we exploit transformer layers to unearth the potential semantic relations within both verbs and semantic roles. With the help of a set of support images, an alternate learning scheme is designed to simultaneously optimize the results: update the verb using nouns corresponding to the image, and update nouns using verbs from support images. Extensive experimental results on challenging SWiG benchmarks show that our renovated framework outperforms other state-of-the-art methods under various metrics.


INTRODUCTION
Understanding complex events in a way that obeys human cognitive habits is one of the core tasks of computer vision and multimedia. As shown in Figure 1, "human-like" event understanding goes beyond traditional object-and action-centric detection and recognition tasks [11,13,24,30,31,39]. Different from image captioning [9,32,43,78] and scene graph generation [20,67,68,71], which use natural language or object-relational graphs to describe scenes, human-friendly event understanding must be event-centric, that is, not only to identify what activities happened, but also to recognize how objects participate in activities, i.e., answer questions like "who is doing what with some tools at someplace. " To meet the demands of "human-like" event understanding, inspired by previous research on semantic role labeling [21,45,53,59] in text, Grounded Situation Recognition (GSR) [47,73] is proposed for event understanding in the multimedia domain. As illustrated in Figure 1, GSR not only detects the salient activity (verb) in the image (e.g., sprinkling), but also recognizes all semantic roles (e.g., agent is man, source is cup). To further determine semantic roles in images, GSR also provides visually grounded information (i.e., bounding boxes) for noun entities. From the perspective of the theory of frame semantics [26], GSR task can be considered as a multimedia extension to earlier lexical databases such as FrameNet [3] and PropBank [33]. By describing activities with verbs and grounded semantic roles, GSR can provide a visually grounded verb-frame, which benefits many downstream scene understanding tasks, such as information retrieval [15,16,44,56], question answering [1,7,35,67], recommended system [10,12,14,55], and multimedia understanding [29,51,77,78].
To sum up, the essence of GSR task is using semantic relations to generate verb-frames for event understanding. However, inspired by object detection and image captioning, almost all current GSR methods [19,36,42,47,54] adopt a two-stage framework. As shown in Figure 3 (a-b), two-stage framework 1) first blindly pre-detects verbs to reduce the search space, and then 2) predicts the semantic roles in an auto-regressive (RNN) or parallel (Transformer) manner depending on the detected verbs. Such two-stage frameworks obviously neglect the semantic relations among verbs and semantic roles. On the one hand, pre-detecting the verb without noun entities inevitably fails to distinguish some similar daily activities. For example, as shown in Figure 2 (left), it is hard to distinguish similar verbs (e.g., buying and giving) without the help of any noun entities. On the other hand, based on pre-identified verbs, applying auto-regression in a closed space would accumulate errors and thus miss the semantic relationships. As shown in Figure 2, once verb buying is wrongly predicted as giving in the first stage, the semantic roles payment: credit and place: shop could be neglected in the second stage.
Similar to our starting point, previous work CoFormer [18] and SituFormer [65] also argue that the existing two-stage framework is problematic. As shown in Figure 3 (c), a three-stage framework is thus proposed to further optimize verbs with predicted nouns. Following traditional two-stage works, the first two stages perform the verb and noun entities detection, respectively. Then, in the third stage, the predicted noun entities are used to refine the verb. However, such disentangled framework still has the following flaws. First, the bidirectional semantic relations between verbs and noun entities cannot be fully exploited. They either use the noun roles to refine the verbs (SituFormer), or only use verbs to refine the noun roles (CoFormer) while ignoring each other. Second, the framework is redundant. It has two parallel transformer verb and noun detectors, but does not learn semantic relations between them during the encoding phase. Third, the refinement process is not scalable. It can be treated as a one-time noun-to-verb optimization, which apparently cannot be expanded.
To address these issues, we focus on how to exploit such bidirectional semantic relations within verbs and noun entities, which can constrain each other. Rather than explicitly pre-detecting the verbs at the first stage, we postpone decision verbs, thus simply assuming a pseudo-category and learning an intermediate representation for each noun entity. We then devise an iterative framework to capture the semantic relations among verbs and nouns and alternately learn their features. As such, we streamline the redundant structures and make them flexibly handle semantic roles in parallel.
Technically, our proposed method, called GSRFormer, is built based on the transformer structure [62] due to its parallel processing capability. As shown in Figure 4, GSRFormer adopts a two-stage architecture that consists of an encoder and a decoder. In the encoder part, we first utilize stacked Multi-Head Attention (MHA) layers to learn the feature of the verb. By assuming a pseudo category of the verb, we further learn the intermediate representations of the corresponding noun entities from the image. In the decoder part, MHA layers are employed to mine the implicit relations among both verbs and nouns. By leveraging a set of support images, the model alternately optimizes the verbs and nouns in a loop: update the features of verbs using nouns, and vice versa. Our framework successfully learns the semantically rich representations for both verbs and nouns and thus performs accurate recognition. To conclude, our contributions are mainly three folds: • We reveal the problems of existing frameworks and point out that learning the bidirectional semantic relations is the core for accurate role recognition.
• We propose a two-stage framework with transformer structures to iteratively refine activity verbs and noun entities. It flexibly mines the potentially open semantic relations within verbs and nouns and alternately updates their features.
• Extensive experiments on challenging SWiG benchmarks fully demonstrate that our proposed framework outperforms other state-of-the-art methods under various metrics.
To address this problem, image captioning [1,23,49,52,69,75] and scene graph generation [8,20,37,66,68,71,76] attempt to reason and describe scene content through natural language or relational graphs of objects. However, these efforts have still failed to understand events consistent with human cognition, i.e., identifying what happened and who was involved in what roles. In this context, Yatskar et al. [73] first proposed Situation Recognition (SR) task and annotated imSitu dataset as the benchmark. However, the original SR task cannot point to where the involved noun entities are located in the image. To further address the visual grounding of the entities, Pratt et al. [47] redefines Grounded Situation Recognition (GSR) task and proposes SWiG dataset by providing bounding box annotations on imSitu dataset. GSR task can be seen as a further extension of SR task.  (c) Three-stage transformer [18,65] adopts a coarse-to-fine refinement process. It still obeys the two-stage idea, where the first stage detects a set of similar verbs and the second stage recognizes noun entities. The third stage utilizes noun entities to refine verbs.
Challenges of GSR tasks. The challenges of GSR task are to handle the semantic relations in the scene. Yatskar et al. [73] proposed a conditional random field (CRF) model in the initial phase. However, follow-up work [72] pointed out that CRF method cannot effectively utilize semantic relations. Since then, a lot of works have started to investigate how to model the relations among semantic roles. The previous technical routes mainly include Recurrent Neural Network (RNN) [42,47,63,63], Graph Neural Network (GNN) [36,54] and Relational Reasoning [4,22], etc. Transformers in GSR task. After great success in NLP tasks, transformer structure [62] was introduced to solve various computer vision problems, including image generation [17,46], image recognition [25,41,60], object detection [5,80], object segmentation [74], image captioning [23] and scene graph generation [20], etc. To further exploit the strength of the transformer in GSR task, Cho et al. [19] proposed the first transformer framework (GSRTR) by replacing the object queries in transformer object detector (DETR [80]) with semantic role queries. Wei et al. [65] recently proposed SituFormer, which uses two transformer-based verb and noun detectors to improve the performance. In addition, Cho et al. [18] proposed CoFormer that tends to leverage the semantic relations between the verb and noun roles to refine the prediction. Problem of framework. To elaborate, as shown in Figure 3, almost all existing GSR methods [4,19,22,42,47,54,63] adopt a two-stage framework based on RNN or transformer. In RNN structure, Pratt et al. [47] proposes a Joint Situation Localizer (JSL) model, which consists of a verb classifier in the first stage and an RNNbased object detector in the second stage. In transformer structure, GSRTR [19] uses a transformer encoder to detect verbs in the first stage and a transformer decoder to predict semantic roles in the second stage. These two-stage frameworks evidently cannot exploit the semantic relations. First, recklessly detecting verbs in the first stage will inevitably misidentify similar activities. Second, the closed auto-regressive strategy of the second stage not only fails to correct misrecognized verbs, but leads to more mispredicted semantic roles. Although recent works (SituFormer [65] and CoFormer [18]) are also aware of framework issues and adopt a three-stage framework to refine prediction results from coarse to fine, they are still unable to optimize the results with bidirectional semantic relationships (i.e., from both verbs and nouns).

PROPOSED METHOD 3.1 Problem Formulation
Definition of GSR task. Given an image , GSR aims to generate a structured verb frame F = { , R } 2 . As shown in Figure 1, GSR not only recognizes the salient verb ∈ V, but also detects all corresponding semantic roles .., } is the set of role types for verb . For instance, the verb-frame in Figure 1 can be instantiated as . Semantic roles R is a collection of triples, where each role contains the role type , the noun entity ∈ N and the corresponding bounding box b ∈ R 4 . Note that not all semantic roles have corresponding nouns and bounding boxes, i.e., or b can be equal to {∅}. Problems of existing frameworks. Inspired by object detection and image captioning, as shown in Figure 3, the existing GSR methods [19,36,42,47,54] widely adopt a two-stage framework, i.e., 1) identifying salient verbs in the first stage, and 2) detecting the corresponding semantic roles R in the second stage: (1) There are multiple highly similar activities in GSR tasks, such as buying and giving, as shown in Figure 2. This two-stage framework is inherently unable to utilize the noun entities in the first stage to distinguish similar verbs, let alone misidentify semantic roles due to the verb prediction errors accumulated in the second stage.
To address these issues, Wei et al. [65] recently proposed a threestage framework (SituFormer) as, where the main idea is to refine verb predictions in a coarse-tofine manner. As shown in Figure 3 (c), it uses the first two stages to identify a set of candidate verbs V and corresponding noun entities {R } . Then the third stage uses a ranking loss to refine the verb with a set of support images through similarity retrieval. ) . . .  Similarly, Cho et al. [18] also proposed a three-stage framework (CoFormer). From the framework perspective, CoFormer is similar to SituFormer. 1) The first stage coarsely predicts noun roles (Glance). 2) The second stage uses the predicted noun roles to help predict the verb (Gaze-Step1).
3) The third stage refines the candidate noun roles in Gaze-Step1 using the predicted verb (Gaze-Step2).
Obviously, these revised frameworks are still unreasonable due to the following issues. 1) It uses only noun entities to refine the verb, but totally ignores semantic constraints in the opposite direction (i.e., from verbs to noun entities). 2) Although it has two redundant transformer-based noun and verb detectors, it does not learn the semantic relations between them at the encoding stage.
3) The coarse-to-fine refinement process can only be done once, and it is impossible to iteratively optimize both verbs and nouns.

Overview of GSRFormer
To solve these problems, we reconstruct a framework (GSRFormer) with transformers to fully exploit the semantic relations of GSR tasks. We reformulate GSR task as, where˜is the assumed pseudo verb category, and H is the intermediate feature for the verb and associated roles. The godsend is that it yields the essence of the GSR task, which is to extract comprehensive semantic relations and then iteratively refine the results. As shown in Figure 4, GSRFormer consists of a transformer-based encoder and decoder, respectively. In the first step (Sec. 3.3), instead of pre-detecting verbs, the transformer encoder learns intermediate representations H of verbs and corresponding semantic roles from images, respectively. In the second step (Sec. 3.4), we first use the features obtained in the encoder to retrieve the representations of Top-similar verbs as the support verb set {H } . To mine the various semantic relations, the neural message passing mechanism [28] is then used to flexibly associate relevant relations to each verb or role, thus effectively updating their representations. Finally, we take full advantage of the transformer structure to perform multiple iterations to refine verbs and semantic roles R .

Transformer-based Encoder
The transformer encoder is devised to learn intermediate representations of verbs and semantic roles from images using two multi-head attention modules, respectively. Representation of verbs. As shown in Figure 4, given an image , CNN backbone first extracts the feature map X img ∈ R ×ℎ× . Since the input to the transformer encoder is a sequence of tokens, a 1 × 1 convolutional layer and a flatten operator are used to convert the X img into a sequence of "visual" tokens [f 1 , ..., f ℎ ], where each token f ∈ R is compressed as a -dim visual feature. Inspired by the classification token used in ViT [25], we initialize a learnable verb token f ∈ R to stands for verb . Then all visual and verb token sequences are fed into the first encoding module as, where ⊕ is element-wise addition and E is positional embedding used to distinguish relative positions in the sequence. MHA verb img is a set of stacked multi-head self-attention blocks. As shown in Figure  4, each block consists of a multi-head self-attention layer and a feedforward network, and the layer normalization [2] (Add & Norm) is used before both of them 3 . The role of MHA verb img module is to use multi-head self-attention to learn the intermediate representation of the verb from image features. The output can be divided into optimized 1) image features E img = [e 1 , ..., e ℎ ] ∈ R ×ℎ and 2) verb feature e ∈ R . Here verb feature e is fed to a classifier to determine a pseudo verb category˜. After obtaining˜, we can fetch the corresponding semantic roles ℛ and initialize them to a sequence [r 1 , ..., r ], where each initialized role r ∈ R is a -dim visual feature.
Representation of semantic roles. We further learn intermediate representations of corresponding semantic roles plus verb ] from images. Specifically, E img and e are used as the input to the second encoding module as, where we concatenate verb feature e and corresponding semantic role embeddings [r 1 , ..., r ] as query. As shown in Figure 4, similar to the previous encoding module, MHA roles img is also a set of stacked multi-head attention blocks. The output representation H is then utilized in the decoder part. After GSRFormer encoder, we assume that the feature H has captured the semantics from the image.

Transformer-based Decoder
The transformer decoder aims to utilize the semantic relations among a set of similar verbs and corresponding semantic roles to simultaneously refine their features. While transformer itself is a powerful model that can mine the implicit relations of all pairwise interactions, it still struggles to leverage additional domain-specific knowledge. In our framework, we expect to exploit a set of support verbs (excluding their corresponding nouns) to profit the noun refinement, while only using the nouns in one single image to refine its verb. This a priori knowledge can hardly be included when directly applying the transformer since it considers ALL pairwise relations. To this end, we borrow ideas from the neural message passing [28], where the "messages" are first computed by transformers, and then each verb or noun entity is updated using "messages" arising from the relations to other appropriate entities. As shown on the right of Figure 4, the transformer decoder is mainly composed of the following four parts. Support verbs set. There are many very similar verbs in GSR tasks, as shown in Figure 2. Inspired by previous work [ where D T is the set of all training images. The similarity score S(·, ·) is the average cosine similarity of semantic roles as, where sim(·) is cosine similarity. h → and h → are the noun entity features for the verbs and , respectively. , where ⊕ is element-wise addition, and implies the -th iteration. h ( ) and h ( ) → (·) are the learned verb and noun entity features, respectively. Here we replace the positional encoding in the original MHA layer with the semantic role embedding r ( ·) . Thus MHA roles verb is actually a multi-head self-attention model. We perform the relation message computation for the entire support set. The obtained verb message p ( ) and the noun entity message p ( ) → (·) contain all the semantic information within a single image. Below we will consider semantic relations in multiple verbs (i.e., support verbs set). Refine noun entity with verbs. We utilize the semantic relations (messages) from the verbs of support set to refine the noun entities. To update the representation of a noun entity h can be any permutation-invariant function (e.g., element-wise sum and max), and here we employ a gated update function [38]. After the messages are fused, a transformer sublayer (FFN and LN) are used to updated representation with residual connection: where LN(·) is layer normalization [2] and FFN(·) is a feed-forward neural network (commonly with one large intermediate layer).
Refine verb with noun entities. Similarly, when updating the verb feature, we utilize the messages of nouns only from the single associated image, as shown in Eq. 12-14, Note that the above two refining processes can be accomplished alternately. Unlike the previous work [65], which only optimized from rough noun entities to verbs, our framework takes full advantage of the flexibility of the transformer structure to efficiently perform multiple refinement iterations. After completing the refinement of noun entities and verb features for iterations, we employ a lightweight MLP over the verb and noun entity features, respectively, to achieve the classification of verbs and the regression of noun entities with bounding boxes, We discuss the training process in detail in the following section.
In the ablation studies, we discuss the effect of the number of loops and refinement order on the results in detail.

Training Objectives
We use the same data augmentation and batch training strategy as previous work [19]. The training details for the encoder and decoder are as follows.
Training of encoder. We use the cross-entropy loss function to train the encoder to obtain the pseudo-verb category as, where the ground-truth verb category is denoted as and the predicted pseudo-verb category is˜. Note that the first multi-head attention module of the encoder performs gradient back-propagation only when training the encoder, and does not participate in parameter updates when training the decoder. Training of decoder. When training the decoder, we need to optimize the categories of verbs and nouns as well as the bounding boxes of nouns. Following previous work [19,65], the losses of the decoder are calculated as, where we use the cross-entropy loss function to train the decoder to get the true verb categories . For the noun loss function, and b denote the ground-truth noun category and bounding box, while and b are the predicted noun category and bounding box. L CE is the cross-entropy loss for noun classification. L consists of the generalize IoU loss [50] and the 1 regression loss. Process of inference. Similar to previous methods [19,42,47,65], GSRFormer also requires inference in the encoder and decoder separately. Compared to the three-stage SituFormer [65], our inference process is more straight. At inference time, GSRFormer first predicts a pseudo-verb category and then constructs the corresponding semantic roles to learn the representations using the encoder. Based on the output features from the encoder, a set of similar support verbs is then retrieved in the training set as the input to the decoder. Finally, the decoder produces the verb and noun predictions.

EXPERIMENT 4.1 Experimental Settings
Datasets. Our experiments are carried out on the challenging SWiG benchmark [47]. SWiG dataset builds on the original Situation Recognition (SR) imSitu dataset [73] by adding bounding box (bbox) annotations for all visible semantic roles (63.9% of roles have bbox annotations). Since each image in imSitu is annotated with three verb frames by three annotators, SWiG contains 126,102 images with 504 verbs and 190 semantic role types, and each verb is followed by 1 to 6 semantic roles (3.55 on average). We followed the official splits to construct the training/validation/testing set with sizes of 75K/25K/25K, respectively. Evaluation metrics. We use the same five evaluation metrics as Pratt et al. [47], including 1) verb: the accuracy of verb prediction, 2) value: the accuracy of noun prediction for each semantic role, 3) val-all: the accuracy of noun prediction for the whole semantic role set, 4) grnd: the accuracy of noun prediction with correct grounding (bbox) for each semantic role, 5) grnd-all: the accuracy of noun prediction with grounding (bbox) for the whole semantic role set. Note that we consider a grounding is correct if the IoU between the predicted and ground-truth bbox is above 0.5. Meanwhile, we report the above metrics in three evaluation settings: 1) Top-1-verb, 2) Top-5-verb and 3) Ground-Truth-Verb, which select verbs based on top-1 prediction, top-5 predictions, and corresponding ground truth, respectively. If verb predictions are incorrect in the Top-1-verb and Top-5-verb settings, the corresponding semantic role predictions are also considered false.

Comparisons with State-of-the-Arts
Baseline models. Existing SR models can be classified into: 1) CRF [73]: CRF-based model, 2) CRF+DataAug   Results under Ground-Truth-Verb setting. The Ground-Truth-Verb setting evaluates whether the system can understand events in a human-cognitive manner. The numerical value of this setting describes how well the machine predictions match the human annotations. The experimental results are shown in Table 1 and Table  2. In general, our GSRFormer outperforms other state-of-the-art methods. Compared with SituFormer [65], which also adopts the transformer structure, GSRFormer improves the accuracy of noun prediction in single (value) and all semantic roles (val-all) by 2.96% and 2.55%, respectively. Furthermore, GSRFomer achieves similar improvements under the vision grounding setting (i.e., grnd and grnd-all), which shows that GSRFormer can effectively learn visual information from natural scenes. Results under Top-N-Verb settings. We use the Top-N-Verb setting to evaluate the accuracy of predicting verb categories. The results in Table 1 and Table 2 show that our GSRFormer outperforms other state-of-the-art methods. It is well known that SituFormer [65] adopts a three-stage framework to improve the accuracy of verb prediction. Compared to SituFormer, GSRFormer can further push the verb prediction accuracy by 2.33% (absolute) under the Top-1-Verb setting. In addition, GSRFormer also achieves a more splendid improvement on SituFormer in single noun prediction (2.24% on value) under the Top-1-Verb setting. Unlike SituFormer, which only uses nouns to improve verbs, our GSRFormer adopts a bidirectional refinement strategy to iterative optimize the results. The 2.24% increase in noun prediction accuracy fully reveals the effectiveness of our proposed alternative semantic refinement.

Ablation Studies
Effectiveness of encoder. We perform ablation studies on two multi-head attention modules of the encoder (denoted as Encoder-1st and Encoder-2nd), as shown in Table 3. Experimental results show that both structures significantly improve performance (over 10% absolute improvement). This fully demonstrates the effectiveness of the encoder module. Encoder-1st is valuable in understanding visual grounding details (comparable on grnd-all metrics), while Encoder-2nd is more conducive to mining verbs and semantic roles. Effectiveness of decoder. We validate the entire decoder module. By removing the decoder, the modified model resembles a simple two-stage approach [19]. The results in Table 3 show that the absolute improvement is also over 10%, which fully demonstrates the effectiveness of the decoder module. Below we will analyze each part of the decoder in detail.
Effectiveness of iterative refinement. We verify iterative refinement. Without iterative refinement (i.e., only 1 transformer layer in the decoder instead of 5), the performance (2%-8% drop) will be similar to traditional RNN-based methods (e.g., JSL [47]). It hints that the real improvement to the transformer structure is the ability to make iterative improvements. Effectiveness of alternate optimization. We verify alternate optimization. By removing this, we mean to update the verb with support verbs, and update nouns with other nouns. We are surprised to find that alternate optimization resulted in more performance drop than iterative refinement, except for the visual relevant grnd-all metrics. This fully illustrates that the core of iterative refinement is to exploit the semantic relationship between verbs and nouns (roles) in both directions. Effectiveness of message computation. We compare with that removes the message passing mechanism, i.e., directly using features of transformer encoder to learn the semantic relations. As  Figure 5: (a) Comparison of GSRFormer and GSRTR [19], where GSRFormer predicts more correct verbs and nouns under Top-5-Verb setting.
(b) An example demonstrates the application of GSRFormer, which can serve cross-modal semantic question answering.  shown in Table 3, message passing not only effectively uses semantic relations to improve verb and noun prediction, but also learns the visual grounding information (grnd and grnd-all metrics). Effects of refinement order. Table 4 shows that refining the noun first gives better results. This is the exact opposite of the previous work [19,42,47]. We speculate that this is because the set of noun entities is more relevant to the visual groundings, so predicting nouns first leads to fewer errors than verbs. For example, humans naturally recognize noun entities as evidence for judgments when stating verbs. This inspires us to prioritize noun entities. Effects of aggregate functions. We test four aggregate functions. Table 5 shows that Gated Function [38] achieves the best results and Element-Wise Sum does not reach satisfactory results. This contrast points out that aggregation functions should strive to highlight semantically relevant features while ignoring unnecessary noise.
In future work, we will explore more fusion strategies.
Effects of the number of stacked layers. We validate the effect of stacking layers in three multi-head attention structures. As shown in Figure 6, the best performance is achieved when six layers are stacked in the encoder and five layers in the decoder. As the number of stacked layers increases, the performance first increases and then decreases. We attribute this performance change to the noise of stacking too many layers.
Effects of the size of support verb set. As shown in Figure 6 (d), the best performance is when the support set size is 5. The performance drops when the size is 1, indicating that similar verbs can support semantic understanding. Moreover, when the size is 10, the performance does not further improve, which means that expanding support verbs also raises noise.

Visualization and Application
We visualize the results of GSRFormer in Figure 5 (a). With the help of alternate semantic refinement, GSRFormer predicts some almost indistinguishable action verbs and semantic roles (see red font). We also show an example of the semantic question-answering application in Figure 5 (b). For example, when we ask questions like "How are women's eyebrows made?", GSRFormer can not only utilize the generated verb-frame to understand the question, but also provide structured answers with rich image facts.

CONCLUSION
In this paper, we first reveal the problems of existing frameworks and point out that the use of semantic relations is the root of the GSR task. To this end, we propose GSRFormer, a two-stage transformer framework that utilizes bidirectional semantic relations to iteratively refine predictions of verb and noun entities. Experimental results on SWiG dataset show that it outperforms other state-of-the-art methods. In the future, we will further explore to explain the semantic structure of GSRFormer and extend it to other semantic analysis tasks. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.

A IMPLEMENTATION DETAILS
Network structure. We use ImageNet pre-trained ResNet-50 as the backbone. To facilitate computation, we use 1×1 convolution to compress the dimension of image features to 512, which is consistent with the hidden dimension of each semantic role query and verb token. To further speed up inference, the dimension of verb embedding and role embedding is 256. Correspondingly, we use learnable 2D embeddings for positional encodings, which have the same dimensions as sequences of visual and verb tokens. The number of heads for all multi-head attention blocks is 8. The sizes of the hidden dimensions of the four structures are 2048, 1024, 1024, and 1024, and the dropout rates are 0.15, 0.3, 0.3, and 0.2, respectively. The bounding box regressor is three fully connected layers with ReLU activation function and 1024 hidden dimensions, using a dropout rate of 0.2. Label smoothing regularization [57] is used for target verb and noun labels with label smoothing factors of 0.3 and 0.2, respectively. We set the support verb set size to 5. The number of stacked layers of the encoder is set to 6, and the number of stacked layers of the decoder is set to 5. Training details. Although using the data augmentation strategies similar to DETR [5] can improve the experimental results, for a fair comparison, we employ the same data augmentation procedure as previous work [19]. Specifically, Random Color Jittering, Random Grayscale Scaling, Random Scaling, and Random Horizontal Flipping are employed. The Hue, Saturation, and Lightness Scales in random Color Jittering are set to 0.1. Random Grayscale Scaling is set to 0.3. Random Scaling is set to 0.5, 0.75, and 1.0. The probability of Random Horizontal Flip is set to 0.5. The number of semantic roles varies from 1 to 6, depending on the verb category. To speed up training, inspired by previous work [19], we utilize zero padding for each output of the noun prediction branch to ensure batch training. Because there are as many semantic role queries as semantic roles, we directly ignore the padding output in the loss computation. When training the decoder, we need to alternately compute verb and noun losses separately. Since there are three nouns per semantic role, the final noun loss is the sum of the three noun losses. In addition, we also illustrate more visualization comparisons and application examples, as shown in Figure 7-8. Stirring -PLACE kitchen Figure 7: Comparison between GSRFormer and GSRTR [19]. The first two columns compare the verbs detected by GSRTR and GSRFormer, respectively. The last two columns are the semantic roles predicted by GSRFormer. With the help of an iterative refinement mechanism, GSRFormer predicts more correct verbs and nouns (marked in red font) under the Top-5-Verb setting. These examples also clearly illustrate the importance of semantic relations for GSR tasks, i.e., bidirectional semantic ties can mutually refine the predictions of verbs and nouns.  understanding. For example, when questioned "How do people cook? What do people cook with, and where do people cook?", GSRFormer can list cooking procedures and steps. Compared to image captioning and scene graphs, GSRFormer can not only utilize the generated structured verb-frame to apprehend the questions, but also provide answers with intuitive image facts to help users understand.