{'fraction_non_alphanumeric': 0.054552676876903385, 'fraction_numerical': 0.011144458666195877, 'mean_word_length': 4.542441291585127, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering

Today's VQA models still tend to capture superficial linguistic correlations in the training set and fail to generalize to the test set with different QA distributions. To reduce these language biases, recent VQA works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on diagnostic benchmarks for out-of-distribution testing. However, due to complex model design, these ensemble-based methods are unable to equip themselves with two indispensable characteristics of an ideal VQA model: 1) Visual-explainable: The model should rely on the right visual regions when making decisions. 2) Question-sensitive: The model should be sensitive to the linguistic variations in questions. To this end, we propose a novel model-agnostic Counterfactual Samples Synthesizing and Training (CSST) strategy. After training with CSST, VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST). CSS generates counterfactual samples by carefully masking critical objects in images or words in questions and assigning pseudo ground-truth answers. CST not only trains the VQA models with both complementary samples to predict respective ground-truth answers, but also urges the VQA models to further distinguish the original samples and superficially similar counterfactual ones. To facilitate the CST training, we propose two variants of supervised contrastive loss for VQA, and design an effective positive and negative sample selection mechanism based on CSS. Extensive experiments have shown the effectiveness of CSST. Particularly, by building on top of model LMH+SAR, we achieve record-breaking performance on all OOD benchmarks.


INTRODUCTION
V ISUAL Question Answering (VQA), i.e., answering natural language questions about the given visual content, is one of the essential abilities of advanced AI agents. With the release of multiple large-scale VQA datasets (e.g., VQA v1 [3], VQA v2 [4], CLEVR [5], and GQA [6]), VQA has received unprecedented attentions and hundreds of VQA models have been developed. However, since the inevitable annotation artifacts in the real image datasets, today's VQA models always over-rely on superficial linguistic correlations between the questions and answers (a.k.a., language biases) [4], [5], [7], [8]. For example, a model naively answering "2" for all "How many X?" questions can still get satisfactory performance regardless of "X". To disentangle the bias factors and clearly monitor the progress of VQA research, several diagnostic benchmarks have been proposed, such as VQA-CP [9] and GQA-OOD [10]. These benchmarks deliberately keep different question-answer distributions in training and test sets. Moreover, the performance of many state-of-the-art VQA models [11], [12], [13], [14], [15] all consistently drop significantly on these diagnostic benchmarks compared to their counterparts (i.e., VQA v2/v1 and GQA).
Currently, the most prevalent solutions to mitigate the based models [17], [18], [19]: They train two models (i.e., the question-only and targeted VQA model) in an adversarial manner [20], [21], i.e., minimizing the loss of VQA model while maximizing the loss of question-only model. Since these two models are typically designed to share the same question encoder, these adversary-based methods aim to reduce the language biases by learning bias-neutral question representations. Unfortunately, this adversarial training scheme brings significant noise into gradient calculations, which results in an unstable training process [18]. 2) Fusionbased models [1], [22], [23], [24], [25]: They late fuse the predicted answer distributions of the two models, and derive training gradients based on fused answer distribution. The design philosophy of these fusion-based models, is to let the targeted VQA model focuses more on samples, which can't be easily answered correctly by the question-only model.
Although the ensemble-based methods have dominated the performance on these diagnostic benchmarks, it is worth noting that current methods fail to equip themselves with two indispensable characteristics of an ideal VQA model: 1) Visual-explainable ability: The VQA model should rely on the right visual regions when making decisions, i.e., right for the right reasons [26]. As shown in Fig. 1 (a), although both two models can predict the correct answer "surfing", they actually refer to different visual reference regions when making their respective answer predictions. 2) Questionsensitive ability: The VQA model should be sensitive to the linguistic variations in questions. As shown in Fig. 1 (b), for two questions with similar sentence structure (e.g., only replacing the word "luggage" with "bus"), if the meanings of the two questions are different, the model should perceive the discrepancy and make corresponding predictions.
In this paper, we propose a novel model-agnostic Counterfactual Samples Synthesizing and Training (CSST) strategy. CSST serves as a plug-and-play component to improve VQA models' visual-explainable and question-sensitive abilities, even for complex ensemble-based models. Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST).

Image
Question Answer The synthesized training sample by V-CSS. It masks ciritcal objects (e.g., ) in image and assigns different GT answers (e.g., "not green").

What color is the man's [MASK] NOT green
(c): The synthesized training sample by Q-CSS. It replaces critical words (e.g., "tie") with special token "[MASK]" in the question and assigns different GT answers (e.g., "not green").
For each original VQA training sample, CSS can generate a corresponding counterfactual sample. As shown in Fig. 3, CSS consists of two different types of sample synthesizing mechanisms: V-CSS and Q-CSS. For V-CSS, it synthesizes counterfactual images by masking critical objects in the original image. By "critical", we mean that these objects are important for answering a certain question. For example, object (i.e., green tie) is a critical object for question "What color is the man's tie?". Then, the counterfactual image and original question compose a new image-question (VQ) pair. For Q-CSS, it synthesizes counterfactual questions by replacing critical words in original question with a special token "[MASK]". Similarly, the counterfactual question and original image compose a new VQ pair. Meanwhile, to avoid expensive manual annotations, we design a dynamic answer assigning mechanism to approximate ground-truth answers for all synthesized VQ pairs (e.g., "not green").
Based on our general sample synthesizing mechanism CSS, we also propose a new CST training strategy for VQA, which helps the VQA models to focus on the critical objects and words, i.e., improving VQA models' visual-explainable and question-sensitive abilities. Specifically, the CST consists of two different training objectives: XE loss and contrastive loss (cf. Fig. 2 (c)). For XE training, we use CSS to generate a counterfactual sample for each original sample, and feed both two samples into a same VQA model. After training with these complementary sample pairs simultaneously, the VQA models are forced to focus on masked critical objects and words. For contrastive training, we sample (or generate) a set of positive/negative samples for each original sample, and propose two variants of supervised contrastive loss [27] for optimization. After training with the contrastive objective, the VQA models can further distinguish the original samples and superficially similar counterfactual ones. For more effective contrastive training, we further design a novel positive and negative sample selection mechanism based on CSS. For each original training sample, we regard all samples with the same question type and ground-truth answers as positive samples, and utilize CSS to generate corresponding counterfactual samples of each positive sample as negative samples. Since CSS can effectively find the critical objects in images or words in questions, these positive and negative samples are superficially similar but semantically different. Compared to directly generating negative samples from the original sample [28], our strategy can significantly improve the diversity of visual content at contrastive training, which is empirically important for contrastive training.
We evaluated CSST on two challenging datasets: VQA-CP and GQA-OOD. Extensive ablations have demonstrated the effectiveness of CSST. CSST can be seamlessly incorporated into any VQA architecture, which not only improves their both visual-explainable and question-sensitive abilities, but also consistently boosts their performance. Particularly, by building on top of model LMH+SAR [1], [2], we achieve new record-breaking results on both benchmarks. This paper is a substantial and systematic extension of our previous CVPR version [16] with several improvements: 1) We design two variants of supervised contrastive loss for VQA. 2) We extend CSS into contrastive training (i.e., CSST), and design a novel positive and negative sample selection mechanism based on CSS. 3) We make two improvements on CSS, which achieves better performance than the original version. 4) We achieve new state-of-the-art performance on all benchmarks, and conduct more experiments to further verify the generalization and superiority of CSST, including more datasets, backbones, ablations, and visualizations.

RELATED WORK
Visual Question Answering (VQA). VQA, i.e., understanding both the visual content and natural language question, and making the answer prediction, is an important research task in both computer vision and natural language processing. Benefits from the deep learning techniques and largescale VQA datasets [3], [4], [5], [6], VQA has realized impressive progress and achieved good results in real images. With the advance in large-scale multimodal representation pretraining, today's VQA performance is mainly dominated by pretrained multimodal BERT models [15], [29], [30], [31].
Language Biases in VQA. Despite VQA is a typical multimodal task, a large body of research [4], [7], [8], [32] has shown the existence of language biases in VQA. There are two main solutions to reduce language biases: 1. Balancing Datasets to Reduce Biases. The most straightforward solution is to create more balanced datasets. For example, Zhang et.al. [8] and Goyal et.al. [4] collected complementary images with opposite answers for all questions. Although these "balanced" datasets have reduced biases to some extent, the statistical biases from questions still can be leveraged [9]. As shown in VQA-CP, the performance of numerous models drop significantly compared to these "balanced" datasets. In this paper, CSST follows the same spirit of dataset balancing and trains VQA models with more complementary samples. Especially, we don't need any extra manual annotations.
2. Designing Models to Reduce Biases. Another solution is to design specific debiasing models. So far, the most effective debiasing models for VQA are ensemble-based methods [1], [17], [18], [19], [22], [23], [25]. In this paper, we propose a novel CSST strategy, which can be seamlessly incorporated into the ensemble-based models to further reduce the biases. Visual-Explainable Ability in VQA Models. To improve visual-explainable ability, early works [33], [34], [35] directly apply human attention as supervision to guide the models' attention maps. However, since the existence of strong biases, even with appropriate attention maps, the remaining layers of network may still disregard the visual signal [36]. Thus, some recent works [36], [37] utilize Grad-CAM [38] to obtain the private contribution of each object to correct answers, and encourage the rank of all object contributions to be consistent with human annotations. Unfortunately, these models have two inherent drawbacks: 1) They need extra human annotations. 2) The training is not end-to-end. Question-Sensitive Ability in VQA Models. If VQA systems really "understand" the question, they should be sensitive to the linguistic variations in question. Surprisingly, to the best of our knowledge, there is only a few works [39] have studied the influence of linguistic variations in VQA. Specifically, it designs a cycle-consistent loss between two dual tasks, and utilizes sampled noises to generate diverse questions. However, Shah et.al. [39] only consider the robustness to different rephrasings of the same question. In contrast, we also encourage models to perceive the difference of questions when changing some critical words. Data Augmentation in VQA. Some concurrent works and following works after our CSS [16] also utilize data augmentation to improve VQA performance. Specifically, there are two directions: 1) Images Augmentation. Almost all other image augmentation works [40], [41], [42] resort to GAN [20] to generate corresponding counterfactual images, which is notorious for unstable training. Meanwhile, photo-realistic image generation itself is still an open challenge. 2) Questions Augmentation. The most straightforward question augmentation strategy in VQA [43] is back-translation [44], which translating a sentence from one language to another and then translating it back using a pair of translation models.
In contrast, in this paper, our proposed CSST only masks critical objects or words, which is easier and more adoptable Contrastive Training in VQA. Contrastive learning techniques have achieved unprecedented success in vision com-munity, especially for self-supervised representation learning [45], [46], [47]. The core idea of contrastive learning is to maximize the mutual information between the input samples and positive samples, and minimize the one between negative samples. Currently, there are several VQA works [28], [43], [48] also utilize contrastive training to distinguish the original training samples and superficially similar counterparts. Different from all existing works, we equip our CSS mechanism into positive and negative selection for contrastive training, which not only increases the sample diversity, but also meets the semantic requirements.

APPROACH
We follow the common formulation and regard the VQA task as a multi-label classification problem. Without loss of generality, given a dataset D = {I i , Q i , a i } N i consisting of triplets of images I i ∈ I, questions Q i ∈ Q and answer sets a i ∈ {a j i |a j i ∈ A}, the VQA task learns a mapping f vqa : I × Q → [0, 1] |A| , which produces an answer distribution given image-question pair. I, Q, and A denote the set of images questions, and answers, respectively. For each ground-truth answer a j i ∈ a i , it has a soft target score t j i , and we use the same ground-truth soft target scores as in prior works [14]. For simplicity, we omit subscript i in the following sections.
In this section, we first introduce the UpDn baseline [14] and a typical ensemble-based framework in Sec. 3.1. Then, we introduce the details of Counterfactual Samples Synthesizing (CSS) in Sec. 3.2. Last, we introduce the details of Counterfactual Samples Training (CST) in Sec. 3.3. [14]. For each image I, the UpDn uses an image encoder e v to output a set of object features: V = {v 1 , ..., v nv }, where v i is i-th object feature. For each question Q, the UpDn uses a question encoder e q to output a set of word features: Q = {w 1 , ..., w nq }, where w j is j-th word feature. Then, both V and Q are fed into the model f vqa to predict answer distributions:

Bottom-Up Top-Down (UpDn) Model
Typically, model f vqa contains a soft attention mechanism, and it is trained with a (binary) cross-entropy (XE) loss. Ensemble-Based Models. As we discussed in Sec. 1, the ensemble-based models can be grouped into two sub-types: adversary-based and fusion-based. Since adversary-based models [17], [18], [19] always suffer severe unstable training and relatively worse performance, in this section, we only introduce the typical fusion-based framework [1], [22], [23]. As shown in Algorithm 1, they introduce an auxiliary questiononly model f q which takes Q as input and predicts answers: Then, they combine these two answer distributions and obtain a new answer distributionP vqa (a) by a function M : In the training stage, the XE loss is computed based on the fused answer distributionP vqa (a) and the training gradients are backpropagated through f vqa and f q . In the test stage, only the model f vqa is used as plain VQA models.

Algorithm 1 Ensemble-based Model (fusion-based)
Inputs: original training sample (I, Q, a), update cond is a control condition for parameter updates. Outputs: images features V , question features Q, and predicted answer distribution P vqa (a).

Counterfactual Samples Synthesizing (CSS)
The overall structure of the CSS training scheme is shown in Algorithm 2. Specifically, for any VQA model, given an original training sample (I, Q, a), CSS can generate two types of counterfactual samples: (I − , Q, a − ) by V-CSS or (I, Q − , a − ) by Q-CSS. In the following, we mainly introduce the details of V-CSS and Q-CSS. As shown in Algorithm 2, for each specific training sample, we only use one certain synthesizing mechanism, and δ is the trade-off weight (see Fig. 5 (c) for more details about the influence of different δ).

V-CSS
We sequentially introduce all steps of V-CSS following its execution path (i.e., line 5 to 8 in Algorithm 2), which consists of four main steps: initial objects selection (IO SEL), object local contributions calculation, critical objects selection (CO SEL), and dynamic soft answer assigning (DSA ASS).

Initial Objects Selection (IO SEL).
In general, for any specific QA pair (Q, a), only a few objects in image I are related. To narrow the scope of critical objects selection, we first construct a smaller object set I, and assume all objects in I are possibly important in answering this specific question. Since we lack manual annotations about the critical objects for each sample, we followed [37] to extract objects which are highly related to the QA. Specifically, we first assign POS tags to each word in the QA using the spaCy POS tagger [49] and extract nouns in QA. Then, we calculate the cosine similarity between the GloVe [50] embeddings of object categories 1 and the extracted nouns, the similarity scores between all objects in I and the QA are denoted as SIM. We select |I| objects with the highest SIM scores as the initial object set I.

Object Local Contributions Calculation.
After obtaining the initial object set I, we start to calculate the local contribution of each object to the predicted probability of ground-truth answer. Following recent works [36], [37], [51] which utilize modified Grad-CAM [38] to derive the local if cond ≥ δ then execute V-CSS 5: a − ← DSA ASS(I + , Q, VQA, a) 9: return (I − , Q, a − ) 10: where P vqa (a) is the predicted answer probability of the ground-truth answer a, v i is i-th object feature, and 1 is an all-ones vector. Obviously, if the score s(a, v i ) is higher, the contributions of object v i to answer a is larger.

Critical Objects Selection (CO SEL).
After obtaining the private contribution scores s(a, v i ) for all objects in I, we select the top-K objects with highest scores as the critical object set I + . The K is a dynamic number for each image, which is the smallest number meets Eq. (5): where η is a constant, we set η = 0.65 in all experiments (See Fig. 5 for more details about the influence of dynamic K setting). Since the visual features V = {v i } are extracted from a pretrained Faster R-CNN, i.e., they always have a lot of repetitions due to overlapped image regions. To avoid visual clue leakage from the other object visual features [52], we also regard the object that overlaps any critical objects by 60% IoU or more as critical objects. Then, the counterfactual input I − is the absolute complement of set I + in set I, i.e., I − = I\I + . We show an example of I, I + and I − in Fig. 4.

Dynamic Soft Answer Assigning (DSA ASS).
Given the counterfactual visual input I − and original question Q, we compose a new VQ pair (I − , Q). To assign ground-truth answers for the VQ pair (I − , Q), we design a new dynamic soft answer assigning (DSA ASS) mechanism. The details of DSA ASS are shown in Algorithm 3. Specifically, we first feed another VQ pair (I + , Q) into the VQA model, and obtain the predicted answer distribution P + vqa (a). We directly use the same set of ground-truth answer categories as the pseudo ground-truth for the corresponding counterfactual sample. Meanwhile, we re-assign the ground-truth , , P + vqa (a) ← VQA(I + , Q + , a, False) 3: a − := a same gt answer set 4: for a i− in a − do 5:  For I + and I − , they are two mutual exclusive object sets. For Q + and Q − , we show the example when word "kite" is selected as critical word. For clarity, we omit some objects.
probability for each answer category of a − , e.g., the i-th In an extreme case, if the model predicts all ground-truth answers correctly for the VQ pair (I + , Q) with 100% probabilities, then a − is a ∅, i.e., zero for all answer candidates. The basic motivation is that if the current model can predict groundtruth answers for (I + , Q) (i.e., I + contains critical objects and I − not), the ground-truth for (I − , Q) should not contain original ground-truth answers anymore, e.g., "not green" in Fig 2. Similarly, the higher confidence for (I + , Q), the less probability for this answer as the ground-truth for (I − , Q).

Q-CSS
All steps in Q-CSS are similar to V-CSS. Following its execution path (i.e., line 11 to 13 in Algorithm 2), it consists of word local contribution calculation, critical words selection (CW SEL), and dynamic soft answer assigning (DSA ASS).

Word Local Contribution Calculation.
Similar with the V-CSS (cf. Eq. (4)), we calculate the contribution of i-th word feature to the ground-truth answer a as:

Critical Words Selection (CW SEL).
In this step, we first extract question-type 2 words for each question Q (e.g., "what color" in Fig. 4 is the question-type of question "What color is the kite?"). Then, we select top-K words with the highest scores from the remaining sentence (except question-type words) as critical words. The counterfactual question Q − is the sentence by replacing all critical words in Q with a special token "[MASK]". Meanwhile, the Q + is the sentence by replacing all other words (except question-type and critical words) with "[MASK]". We show an example of Q, Q + , and Q − in Fig. 4.

Dynamic Soft Answer Assigning (DSA ASS.)
This step is identical to the DSA ASS in V-CSS, i.e., Algorithm 3. For Q-CSS, the input for DSA ASS is the VQ pair (I, Q + ).

Highlights of Two Improvements on CSS [16]
Compared to the initial CSS in the conference version [16], we make two important modifications and improvements: 1) Object Overlapping in CO SEL. Different from the words features in questions, which are relatively "independent", object features from the same image always have a lot of repetitions due to their overlapped image regions. We avoid this visual clue leakage by considering the IoU overlaps between all objects. This strategy helps models rely more on the calculated critical objects. 2) "Soft" Answer Assigning in DSA ASS. In initial CSS work [16], we select top-K predictions based on P + vqa (cf. Algorithm 3), and assign ground-truth answers based on these top predictions. Compared to this "hard" answer selection, DSA ASS is more robust to the selection of hyperparameter K, and is more sensitive to perceive the changes of answer predictions (i.e., P + vqa ). To distinguish these two types of CSS, we denote the current improved version as CSS + in the following sections (See results in Sec. 4 for comparisons between CSS & CSS + ).

Counterfactual Samples Training (CST)
To further benefit from the counterfactual samples generated by CSS, we propose a CST strategy for model training (cf. Algorithm 4). Specifically, it consists of a cross-entropy (XE) training loss and a contrastive (CR) training loss.

XE Training on Counterfactual Samples
For XE training, we directly follow other state-of-the-art VQA models (e.g., UpDn [14]) and use binary cross-entropy loss as training objective, i.e., where σ denotes the sigmoid activation function, and t i is the soft target score of i-th ground-truth answer for this training sample (cf. line 5 in Algorithm 3). Different from the most prevalent VQA framework which only uses original samples in the XE training (cf. Fig. 2 (a)), in our XE training, we feed both original training samples and their corresponding counterfactual samples into the same VQA model (i.e., line 3 to 5 in Algorithm 4).

Contrastive Training on Counterfactual Samples
For contrastive training, we propose a novel and effective positive and negative samples selection mechanism based on our CSS, and two variants of contrastive loss for VQA. # contrastive training 12: , , P p ← VQA(I p , Q p , , False) for pos. sples. 13: , , P i n ← VQA(I i n , Q i n , , False) for all neg. sples. 14: L CR ← CR(P a , P p , {P i n }) CR training 15: end function Positive & Negative Samples Selection Strategies. Without loss of generality, we regard each original training sample as the anchor sample, and we regard all samples with the same question-type 2 and ground-truth answer categories as the candidate positive set 3 . In each training step, we randomly sample one sample from the candidate set as the positive sample (i.e., POS SEL in Algorithm 4). For each positive sample, we compose four different types of negative samples, which can be categorized into two groups based on their sampling strategies. The first group of negative samples is the counterfactual samples of the positive samples corresponding to V-CSS and Q-CSS, respectively. The second group of negative samples is randomly sampled samples: including 1) a random sampled sample with the same question-type 2 but different ground-truth answer; and 2) a composed sample by replacing the original image with a randomly sampled image from the same batch. All four types of negative sample selection strategies constitute the NEG SEL step in Algorithm 4. Detailed quantitative results of different samples are reported in TABLE 2.
Advantages over Existing Positive/Negative Sampling Solutions. Currently, there are two types of positive and negative sample selection strategies. The first type also builts on top of our CSS [28]. For each sample (I, Q), they directly use the (I − , Q) and (I, Q − ) in our CSS (cf. Algorithm 2) as negative samples. The second type is directly randomly replaced an image or question [43] to compose negative samples (e.g., (I rand , Q) or (I, Q rand )). Instead, we benefit from the ability of CSS, and generate counterfactual samples for positive samples (our first group), i.e., negative samples consists of Global Contrastive Loss (CR-G). Given the predicted answer distributions (before sigmoid activation function) of anchor sample, positive sample, and negative samples (denoted as P a , P p and {P i n } respectively), the CR-G loss is: where s(P i , P j ) is the cosine similarity of answer distributions P i and P j , and τ is the hyperparameter temperature. Local Contrastive Loss (CR-L). Instead of calculating similarity based on the whole answer distributions, there is another alternative choice which only focuses on probabilities of ground-truth answers. Thus, the CR-L loss is: where m is the answer index of ground-truth answer with the highest target score, andP * (·) is the answer probabilities after sigmoid activation function (i.e.,P * (·) = σ(P * (·))). In addition, we addP i n (m) as weights which helps models to focus more on hard negative samples with largerP i n (m). Differences between CR-G and CR-L. Since CR-L only optimize models along a single ground-truth answer direction, it can directly suppress models output over-high scores for negative samples on original ground-truth answers, which is important for vanilla VQA models (e.g.. UpDn). But for ensemble-based VQA models (e.g., LMH), since we overlook the accuracy of fused answer distributions (cf.P vqa (a) in Eq. (3)), it would be better to use CR-G, which calculates the sample similarity over the global answer distributions.

Datasets and Evaluation Metrics
Datasets. We evaluated CSST on two challenging diagnostic VQA benchmarks: 1) VQA-CP [9]. It is a re-organization of the training and test sets of widely-used VQA v1 [3] and VQA v2 [4], where the answer distribution of each question type in the training set is made explicitly different from the one in the test set. There are two VQA-CP datasets: VQA-CP v2 and VQA-CP v1. We followed the official splits [9] for both two datasets. We also reported our results on the VQA v2 validation set for more complete comparisons. 2) GQA-OOD [10]. It is a fine-grained re-organization of GQA [6]. They share the same training set, but GQA-OOD introduces fine-grained shifts into both the validation and test sets. Evaluation Metrics. For model accuracies on VQA-CP, we followed standard VQA evaluation metric [3]. Similarly, we reported accuracy on all test samples (All) and three differ-

Experimental Settings
Data Preprocessing Details. For fair comparisons, we did all the same data preprocessing steps with the widely-used UpDn model [14] using the publicly available reimplementation 4 . Specifically, for image feature extraction, we used Faster R-CNN [53] pretrained on VG to extract the top-K objects. Following the convention of prior works, we set K = 36 for VQA-CP and VQA, and K = 100 for GQA-OOD. For question feature extraction, there are slight differences for the models with different backbones. For the UpDn-based models, we set the maximum length of questions as 14, and the word embeddings are initialized with GloVe [50] vectors with the dimension of 300. A singlelayer GRU is used to obtain question embedding vectors with the dimension of 1,024. For the LXMERT-based models, we used the official tokenizer in LXMERT to segment each question into word tokens. A pre-trained LXMERT model is used to obtain question features with the dimension of 768. Training Details and Hyperparameters. We trained UpDnbased models or LXMERT-based models for 30 epochs with 1: Accuracies (%) of different VQA architectures on VQA-CP v2 test set and GQA-OOD testdev set. CSST-G and CSST-L denote the CSST with CR-G and CR-L losses, respectively. * denotes the results from our reimplementation using official codes. Q-CSS + /V-CSS + /CSS + denote the results of this improved version of CSS, and Q-CSS/V-CSS/CSS denote the results from [16].   batch size 512 on all datasets. We used the Adamax algorithm as the optimizer following the public reimplementation 4 . All parameters were initialized from scratch and the random seed was set to 0. Loss weights of L XE , L CR-G , and L CR-L were set to 1, 1, and 8 in most of experiments. τ in the contrastive loss was set to 1 in all experiments. Results about the influence of hyperparameters are reported in Sec. 4.3.1 Adapting Ensemble-based Methods to GQA-OOD Benchmark. In original GQA-OOD paper [10], the authors claim none of the typical ensemble-based models (e.g., RUBi [22] and LMH [1]) can improve the Acc-T performance on GQA-OOD, and they even deteriorate Acc-H performance. In this paper, we argue that it is not suitable to apply these debiasing methods on all samples, because "biases" only reside in imbalanced local groups [10]. Therefore, in our experiments, we only utilized these debiasing methods on imbalanced local groups. Since the local groups of samples are different in the training and test sets, in the training stage, we only selected the imbalanced local groups from the training set based on the same Shannon entropy and threshold, i.e., we don't use any extra information about test set in advance. Adapting LXMERT Backbone for VQA. LXMERT [15] itself is a multimodal BERT model, which can be easily used for VQA. Similar to other multimodal BERT models, after pretraining, it typically utilizes the output of the [CLS] token as a multimodal fused feature, and trains a linear classifier for answer prediction. Unfortunately, this setting is slightly different from the UpDn backbone, whose inputs are both visual and question features. To benefit from the pretrained weights and seamlessly equipped LXMERT into other existing ensemble-based methods with UpDn backbone, in this paper, we utilized the outputs of visual and language embedding tokens as the visual and question features, respectively (i.e., replacing original inputs of ensemble-based methods). The pretrained weights of LXMERT keep fixed.
(See results of models with LXMERT in following sections.) Adapting SAR [2] strategy to LMH-CSST. In our experiments, we also equipped the model-agnostic SAR [2] to our LMH-CSST to further improve performance. Specifically, for the Select And Rerank (SAR) module, we used SAR to refer to SAR+LMH (i.e., incorporate LMH into SAR). We chose LMH-CSST as Candidate Answer Selector (CAS), and used top-20 answers as candidates. We utilized the "R → C" strategy to combine question and answer into a synthetic dense caption. We trained SAR for 10 epochs, and batch size was set to 32. For Question Type Discriminator, we selected 1 or 2 candidates for Y/N questions and 12 candidate answers for non-Y/N questions when testing on VQA-CP. Meanwhile, Since we lack manual question type annotations for samples on GQA-OOD, we removed Question Type Discriminator when testing on GQA-OOD, and selected 12 candidates for all questions. We refer readers to SAR [2] for more details.

Influence of Different Hyperparameters of CSS
We run a number of ablation studies to analyze the influence of different hyperparameters of CSS + (i.e., V-CSS + and Q-CSS +5 ). Specifically, we conducted all ablations by building  3: Accuracies (%) on VQA-CP v2 test set and VQA v2 val set of state-of-the-art models. The gap (other) represents the accuracy difference between the other category of VQA v2 and VQA-CP v2. "Pre." represents "Pretraining", i.e., these models use extra datasets at their pretraining stage (e.g., LXMERT backbone [15] or SAR [2]). "Ann." represents "Annotation", i.e., these models rely on extra manual anntations. † represents the ensemble-based methods, * indicates the results from our reimplementation using official released codes, ‡ denotes these models use the same adapting strategy for LXMERT backbone mentioned in Sec. 4  on top of a typical ensemble-based VQA model LMH [1]. To disentangle the influence of our contrastive training, we only use the XE loss as training objective and both original samples and counterfactual samples as the inputs (cf. Fig. 2 (b), similar with [16]). All results are illustrated in Fig. 5. Size of I in V-CSS + . The influence of different size of I is shown in Fig. 5 (a). We can observe that the model's performance gradually decreases with the increase of |I|. Size of critical objects in V-CSS + . The influence of masking different numbers of critical objects is shown in Fig. 5 (a). We compared dynamic K (cf. Eq. (5)) with some fixed constants (e.g., 1,3,5). From the results, we can observe that the model with dynamic K achieves the best performance. Size of critical words in Q-CSS + . The influence of replacing different sizes of critical words is shown in Fig. 5 (b). From the results, we can observe that replacing only one word (i.e., top-1) achieves the best performance. Proportion δ between V-CSS + and Q-CSS + . The influence of different δ is shown in Fig. 5 (c). From the results, we can observe that the performance is best when δ = 0.5. In all the following experiments (including experiments on different benchmarks, models with different backbones, and models with contrastive training), we used the same best hyperparameter settings for both V-CSS + and Q-CSS + .

Architecture Generalization of CSST
Settings. Since our CSST is a model-agnostic training strategy, which can be seamlessly incorporated into any different VQA architecture. To evaluate the effectiveness of CSST to boost the debiasing performance of different backbones, we incorporated the CSST into multiple architectures including: UpDn [14], RUBi [22], LMH [1]. Especially, RUBi and LMH are ensemble-based methods. For VQA-CP v2, we followed the same settings as prior works. For GQA-OOD, we used the adapting strategies mentioned in Sec. 4.2. All results are shown in TABLE 1. For more clear comparisons, we used superscript + to distinguish this improved version of CSS (Q-CSS/V-CSS) with their respective initial counterparts [16].
Results. Compared to these baseline models, our CSST (i.e., both CSST-G and CSST-L) can consistently improve the performance for all architectures. For different architectures, the behaviors of CSST-G and CSST-L are slightly different: CSST-G is more suitable for complex ensemble-based models (e.g., 9.21% absolute gains in LMH on VQA-CP v2), and CSST-L is more suitable for vanilla VQA models (e.g., 16.87% absolute gains in UpDN on VQA-CP v2). Meanwhile, when both two types of CSS are used (i.e., Q-CSS + and V-CSS + ), models often achieve better performance than single type CSS. Furthermore, the model with CSST (i.e., CSST-G or CSST-L) always achieves the best performance.

Influence of Positive & Negative Sample Selection
Settings. To demonstrate the effectiveness of our proposed samples selection strategy, we compared CSST with a strong baseline CL [28], which has shown effectiveness under our CSS mechanism. Meanwhile, to further show the importance of counterfactual samples in the sample selection, we separate these four negative samples into two groups: 1) "CSS": the first two negative samples from V-CSS + and Q-CSS + ; 2) "Random" (Rand): the last two randomly composed negative samples (cf. NEG SEL in Algorithm 4). All results are reported in
Results. The results are reported in

Performance on VQA-CP v1
Settings. We also compared the LMH-CSST with state-ofthe-art models on the VQA-CP v1. Similarly, we group these models into: 1) AReg [17], GRL [18], RUBi [22] and LMH [1] are all ensemble-based models. 2) GVQA [9] and UpDn are vanilla VQA models, and GVQA is with SAN [13] backbone. All settings are same as the ones on VQA-CP v2. Results. The results are reported in TABLE 4. Compared to baseline models, both LMH-CSST and LMH-CSST+SAR achieve new state-of-the-art performance on two different backbones over all metrics. Particularly, the CSST improves the performance of LMH with a 7.89% (63.16% vs. 55.27%) absolution performance gains on UpDn backbones.
Results. The results are reported in TABLE 5. For fair comparisons, the baselines (UpDn and LMH) with the LXMERT backbones were reimplemented by using the same adapting strategy. Compared to baselines, LMH-CSST+SAR achieves new state-of-the-art performance on most of the metrics on both two backbones. Especially for the most important metrics Acc-T [10], our achieve the best performance over others (e.g., 47.98% vs. 43.37% and 48.07% vs. 47.2%).

Improving Visual-Explainable Ability
We will validate the effectiveness of CSST to improve visualexplainable ability by answering the following questions: Q1: Can existing visual-explainable models be incorporated into the ensemble-based framework? Q2: How does CSST improve the model's visual-explainable ability? 4.5.1 CSST vs. SCR (Q1) Settings. We equipped the existing state-of-the-art visualexplainable model SCR [37] into the LMH framework, and compared it with CSST. Results are reported in TABLE 6 (a). Results. Since the training of all SOTA visual-explainable models (e.g., SCR [37], HINT [36]) are not end-to-end, for fair comparisons, we used a well-trained LMH (i.e., 52.45% accuracies on VQA-CP v2) as the initial model. However, we observe that its performance continues to decrease from the start, which shows that the existing visual-explainable models can not be easily incorporated into the ensemble-based framework. In contrast, the proposed CSST can consistently improve performance on different backbones.  Fig. 7: Visualization exmaples of question-sensitive ability from VQA-CP v2 test set. Different shades of green color in the question denotes the relative values of s(â, w), i.e., the word with darker green denotes the word has larger contribution to final predictions.

Evaluations of Visual-Explainable Ability (Q2)
Settings. We evaluated the effectiveness of CSS to improve the visual-explainable ability on both quantitative and qualitative results. For quantitative results, since we lack human annotations about the critical objects for each question, we regard the SIM scores (cf. IO SEL in Sec. 3.2.1) as pseudo ground-truths. Thus, we design a new metric Average Importance (AI): the average SIM score of the top-K objects with highest |s(a, v)|. We formally define AI as: where k is the index of top-K objects, SIM (I,Q) k is the SIM score of k-th object for the sample (I, Q), and 1 is an indicator function. The results are shown in TABLE 6 (b). We further demonstrate some qualitative results in Fig. 6. Results. From TABLE 6 (b), we can observe that CSST dramatically improves AI scores, which means the influential objects for predictions are more related to the QA pairs. From Fig. 6, we can find that both CSS (LMH vs. LMH-CSS) and CST (LMH-CSS vs. LMH-CSST) help the model to make predictions based on critical objects (i.e., green boxes) and suppress the influence of irrelevant objects (red boxes).

Improving Question-Sensitive Ability
We will validate the effectiveness of our CSST to improve the question-sensitive ability by answering the following questions: Q3: Does CSST help to improve the robustness to diverse rephrasings of questions? Q4: How does CSST improve the model's question-sensitive abilities?

Robustness to Rephrasings of Questions (Q3)
Settings. As discussed in previous work [39], being robust to diverse rephrasing of questions is one of key behaviors of a question-sensitive model. To more accurately evaluate the robustness, we re-splited the existing dataset VQA-Rephrasings [39] with the same splits as VQA-CP, and denoted it as VQA-CP-Rephrasings. For evaluation, we used the standard metric Consensus Score CS(k). Results are reported in TABLE 6 (c) (left). We refer readers to [39] for more details about the VQA-Rephrasings and metric CS(k). Results. From TABLE 6 (c), we can observe that Q-CSS + dramatically improves the robustness to diverse rephrasings of questions. Furthermore, V-CSS + can help to further improve the robustness, i.e., CSS + achieves better performance.

Evaluations of Question-Sensitive Ability (Q4)
Settings. We evaluated the effectiveness of CSST to improve the question-sensitive ability on quantitative and qualitative results. For quantitative results, since there is no standard evaluation metric, we design a new metric Confidence Improvement (CI): Given a test sample (I, Q, a), we remove a critical noun in question Q, and obtain a new test sample (I, Q * , a) 6 . Then we fed both two samples into the evaluated model, and calculated the confidence decreases of the ground-truth answer. We formally define CI as:  6 Test set is released at: https://github.com/yanxinzju/CSS-VQA.
whereâ is the predicted answer for sample (I, Q), 1 is an indicator function. The results are reported in TABLE 6 (c). We further demonstrate some qualitative results in Fig. 7.
Results. From TABLE 6 (c), we can observe that CSST helps the model to benefit more from the critical words, i.e., removing critical words results in more confidence drops for the ground-truth answers. From Fig. 7, we can find that both CSS (LMH vs. LMH-CSS) and CST (LMH-CSS vs. LMH-CSST) help the model to make predictions based on critical words (e.g., " bus" or "objects"), i.e., forcing the model to understand the whole questions before making predictions.

CONCLUSIONS AND FUTURE WORKS
In this paper, we proposed a model-agnostic Counterfactual Samples Synthesizing and Training (CSST) strategy to improve the VQA model's visual-explainable and questionsensitive abilities. The CSST is composed of two main components: CSS and CST, where CSS generates counterfactual training samples by masking critical objects or words, and CST consists of a XE training loss and a contrastive training loss. The CSST (i.e., CSS and CST) can consistently boost the performance of different VQA models. We validate the effectiveness of CSST through extensive comparative and ablative studies on three benchmarks and multiple different backbones. Moving forward, we are going to 1) extend CSST to other visual-language tasks that suffer severe language biases or other types of biases; 2) design a specific VQA backbone that benefits from the design of CSST.