{'fraction_non_alphanumeric': 0.03716702347945109, 'fraction_numerical': 0.014003439441266276, 'mean_word_length': 5.036864406779661, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data augmentation by morphological mixup for solving Raven's Progressive Matrices

Raven's Progressive Matrices (RPMs) are frequently used in testing human's visual reasoning ability. Recent advances of RPM-like datasets and solution models partially address the challenges of visually understanding the RPM questions and logically reasoning the missing answers. In view of the poor generalization performance due to insufficient samples in RPM datasets, we propose an effective scheme, namely Candidate Answer Morphological Mixup (CAM-Mix). CAM-Mix serves as a data augmentation strategy by gray-scale image morphological mixup, which regularizes various solution methods and overcomes the model overfitting problem. By creating new negative candidate answers semantically similar to the correct answers, a more accurate decision boundary could be defined. By applying the proposed data augmentation method, a significant and consistent performance improvement is achieved on various RPM-like datasets compared with the state-of-the-art models.


Introduction
Visual recognition has been widely and deeply explored, while visual reasoning [1][2][3][4][5][6][7][8] recently becomes an increasingly important research topic and needs further investigation. Visual reasoning often consists of two tasks: visual recognition and logical reasoning. It leverages on the development of visual recognition while pushing beyond it. Visual reasoning infers the relationship between objects based on the perceived visual attributes. In computer vision community, data augmentation techniques have been proved effective for many problems, including the latest developments in image classification [9,10], object detection [11,12], synthetic data generation [13,14] and many others [15][16][17][18][19]. In this paper, we aim to improve the performance of visual reasoning system through data augmentation, especially for the problem of solving visual IQ tests [7,8].
Raven's Progressive Matrix (RPM) [20,21] problem is one of the frequently-used tests on human visual reasoning in cognitive science. An RPM problem is formed by a 3×3 pictorial matrix with the last one left blank. The task is to identify the missing entry from 8 candidate answers. One illustrative example is shown in Fig. 1. As the RPM test assesses human's visual reasoning ability using pictorial matrices containing visually simple patterns, it minimizes the impact of language barrier and culture bias [20].
Researchers have made considerable efforts to develop a system to automatically solve the RPM problem as part of broad research initiatives for advanced automated visual reasoning [7,8,[22][23][24][25]. Santoro et al. [22] constructed the first RPM-like dataset named Procedurally Generated Matrices and developed * Corresponding author. Tel.: +86 (0)574 8818 0000-8805 Email address: jianfeng.ren@nottingham.edu.cn (Jianfeng Ren) (a) Question (b) Candidate answers Figure 1: An example of the I-RAVEN dataset [8] with question panel (a) and candidate answer set (b). The correct answer is marked in red.
a reasoning algorithm named Wild Relation Network (WReN). Zhang et al. [7] developed the dataset called Relational and Analogical Visual rEasoNing (RAVEN), and a ResNet architecture assembled with a Dynamic Residual Tree (DRT) module for reasoning. Later Zhang et al. [23] developed the CoPINet architecture under the principle of contrasting, achieving the state-of-the-art performance on the RAVEN dataset. Very recently, Hu et al. [8] argued that the RAVEN dataset [7] has a shortcut to solutions, and developed an Impartial-RAVEN dataset and a solution model named Stratified Rule-Aware Network (SRAN).
In RPM problems such as RAVEN, besides the correct answer, 7 incorrect answers often serve as the negative candidate answers to confuse humans. From the machine-learning point of view, these negative candidate answers will result in an increasing number of training samples, which is beneficial to the model training process. More specifically, more negative candidate answers will define the decision boundary more accurately.
Thus, we develop a data augmentation algorithm by generating more negative candidate answers, train the models with more samples and consequently improve the model performance.
The common image manipulations in data augmentation such as image rotation, cropping and flipping are beneficial to visual recognition of natural images [26]. However, these approaches are not suitable for RPM problems, as they only complicate the visual recognition process, but do not help the reasoning process. For example, the visual recognition system needs additional efforts to identify the image rotation angle, while it is an irrelevant attribute in RPM problems, which does not help the reasoning process and may even confuse the logical reasoning system.
In this paper, we propose a data augmentation method named Candidate Answer Morphological Mixup (CAM-Mix), a graylevel pixel-wise morphological operation that convexly combines candidate answers, by mixing up the correct answer candidate with negative candidates. The generated negative samples are semantically similar but different from the correct answers, and hence could define a more accurate boundary around the space spanned by the correct answers. In addition, many visual attributes remain unchanged in the proposed CAM-Mix, which avoids the problems of introducing additional attribute values by the traditional image mixup [27]. To the best of our knowledge, this is the first research attempt of introducing data augmentation in solving the RPM problems.
The main contributions of this paper are two-fold: 1) We propose a data augmentation mechanism by mixing up the candidate answers, which could define a more accurate decision boundary around the space spanned by the correct answers. 2) As a plug-and-play tool, the proposed method achieves a significant and consistent improvement compared with the stateof-the-art models on RPM problems.

Data augmentation
With great advances in recently developed deep convolutional neural networks, remarkable performance has been achieved for various computer vision tasks [28][29][30][31][32][33]. Sufficient training data often serve as the foundation to guarantee the performance of deep learning models [15]. To improve the generalization ability of the models, data augmentation has been widely used as an effective scheme to assist the model training process. According to the survey by Shorten et al. [34], image data augmentation techniques can be categorized into two main types, basic image manipulations and deep learning approaches.
In the first category, geometric transformations like image rotation, flipping and cropping perform well on natural images [26], but they are generally task-dependent and may not be safely applied in specific shift-variant tasks, e.g. applying image rotation in digit recognition tasks of 6 verses 9. Later, the development of mixing images and random erasing [35] guarantees better generalization ability on various tasks. Specifically, the idea of mixing images can be classified into nonlinearly mixup, i.e. assembling of sample instances [36,37] or sub-images [38], and a linear interpolation of image pairs such as alpha-blending [27]. Vanilla mixup [27] uses a simple linear interpolation function which may suffer from overfitting to corrupted samples. To overcome this challenge, researchers have developed mixup variations including Manifold Mixup [39], which leverages semantic interpolations inside layers to train neural networks, and MetaMixUp [40], which utilizes the concept of meta-learning to determine an optimized interpolation policy to regularize network models.
Data augmentation based on deep learning usually refers to adversarial approaches, such as transfer learning [41] or Generative Adversarial Network (GAN) [16][17][18]. They can generate more synthetic data visually similar to existing training data and help to improve the classification performance. Such data augmentation techniques related to GAN models are usually more task-specific, and hence the generalization ability across different tasks may be limited. Fu et al. [16] developed a GAN-based data augmentation model for fine-grained classification problems, which can produce class-dependent synthetic images with fine-grained details. GAN-based data augmentation adopted in [17] aims to handle the problem of sample inadequacy and class imbalance, and consequently facilitates the training of LSTM network in real-time visual tracking problems. Liu et al. [18] developed a GAN model for pedestrian detection transferring pedestrians from other datasets into the target scene, to cope with the challenge of insufficient data and heavy pedestrian occlusions.
Visual question answering (VQA) is a conventional visual reasoning task that measures the machine understanding of scene-level images. The objective is to derive an accurate natural language answer, given an image and a related natural language question [2]. Recent advances on various solution models lead to a better machine reasoning capability. Hong et al. [4] developed a neural model to exploit intermediate CNN layers and low-level layer features to answer low-level semantic questions. Gao et al. [5] further explored the relationship between question semantics and fine-grained object information, which improves the VQA performance.
Raven's Progressive Matrices (RPMs) are originally designed as a non-verbal assessment for human intelligence. By using pictorial matrices containing visually simple patterns, it minimizes the impact of language barrier and culture bias. Recently, large-scale RPM-style datasets, RAVEN [7] and its variants [8], were developed to extend the RPM-related study from cognitive science to computer science. The RPM-style datasets [7,8] are often automatically generated, and the problems are solved with minimal prior knowledge about the internal construction rules.
Santoro et al. [22] developed the Wild Relation Network (WReN), which applies a Relation Network to model the rela-tionship between the question panel and the candidate answer. Zhang et al. [7] developed a Dynamic Residual Tree (DRT) module that can help logical reasoning in neural networks [7]. They later developed an improved network architecture called CoPINet [23] under the principle of contrasting. Hu et al. [8] argued that RAVEN has a shortcut to solutions, and hence developed an I-RAVEN dataset balancing the candidate answer sets. Their developed solution model, SRAN [8], achieves the state-of-the-art results on the I-RAVEN dataset.
Various models have been developed in literature to solve the RPM-like problems [7,8,[22][23][24]. However, to the best of our knowledge, no work has considered the problem of insufficient data for precisely conducting relational reasoning in RPMs. In this paper, we propose a data augmentation method that can effectively improve the visual reasoning capabilities of various state-of-the-art solution models.

Problem formulation
An example of RPM-like problems is shown in Fig. 1. Given a 3×3 RPM-like question matrix with the last one missing Q = {q 0 , q 1 , · · · , q 7 }, the aim is to find the missing image a * from the candidate answer set A = {a 0 , a 1 , · · · , a 7 }.
We follow the formulation widely adopted in recently developed solution methods based on deep neural models [8,[22][23][24], i.e. by training a customized model containing functional reasoning modules using sufficient labelled training data. The model is able to generalize on testing data with input Q and each of candidates a i ∈ A, and output one predicted answer a * from candidates. The underlying principle for aforementioned algorithms is to make a selection from the candidate answers with maximum probability of being the correct answer given Q, A: The probability of each candidate P(a i |Q, A) is derived through the deep convolutional neural network. The model takes a group of 8 images Q and one candidates a i from A as the input. In CoPINet [23], for instance, where the model is built upon the Perception Branch F , the Contrast Module C, and the Multilayer Perceptron M, the probability scores s i for candidate answers are calculated as: 3.2. Proposed CAM-Mix by gray-level morphological operation The idea of mixing up images, which is counter-intuitive to data augmentation [34], has proven its effectiveness by a line of works [36,37,39,40,42,43]. In the traditional vanilla image mixup, the following linear interpolations [27] are applied on pairs of samples x i , x j and their one-hot labels y i , y j in training: Visually, the hyper-parameter λ determines how two images x i and x j are mixed. From human's perspective, if λ ≈ 0.5 the two images are more likely to be displayed in balanced transparency. The mixed-up images may confuse humans when making decisions, as the generated synthetic images are visually messy and chaotic from human's perspective. However, it may help machine learning models to learn some hidden relationships in undiscovered search space, and consequently gain a stable performance improvement.
The RPM-like problems are distinct from general image recognition tasks in the following aspects, which may limit the performance of traditional mixup defined in Eqn. (3). 1) In RPM problems, the emphasis is on logical reasoning, and hence the input images are formed up by visually simple patterns, e.g. combinations of regular polygons in a gray-scale intensity form. The image mixup should help more in logic reasoning rather than its original objective in visual recognition. 2) The visual attributes in RPM-like problems come from a finite set. The vanilla mixup defined in Eqn. (3) alpha-blends a pair of image samples, which will produce new visual attribute values such as image intensities outside the finite set. These newly introduced values may lead to incorrect reasoning rules and unnecessarily complicate the visual reasoning process. 3) Finally, the vanilla mixup method interpolates the image content as well as the associated labels, which is contradictory to the nature of visual reasoning on RPMs that each question sample can only have exactly one correct answer.
To address these challenges, we propose a mixup algorithm significantly different from previous ones, based on the concept of gray-level image morphological operations [44]. The proposed image mixup method does not generate images with new visual attribute values outside the finite set, but mixes up the visual attributes of two images, without explicitly extracting the visual attributes. Formally, we define the following gray-level morphological operations for a pair of gray-scale images I 1 and I 2 of the size n×m: AND(I 1 , I 2 ) = max i∈n, j∈m One example of the proposed image mixup operation is shown in Fig. 2. We can see that no new visual attribute values (intensity values, shape variations, etc.) are generated during this process. The new images are generated without additional scalars or interpolation functions like vanilla mixup, and the attributes of the generated images are from one of the two original images, without explicitly extracting the attributes.

Data augmentation by proposed CAM-Mix
As a data augmentation method for RPM problems, we aim to increase the training samples by mixing up the candidate answers, i.e. taking pixel-wise morphological operation OR and AND of negative candidates and the correct candidate. Formally, given the training samples (Q, A), where A contains the correct candidate a c and negative candidates A N : A = {a c }∪ A N , the objective is to generate more negative candidate answersÂ N using the proposed gray-level morphological operations: We build a set of extra training samples (Q,Â) in Fig. 2, whereÂ = {a c } ∪Â N . By mixing up the negative candidate answers with the correct one, those newly generated answers have partial information from the correct answer. As a result, the newly generatedÂ N could define a more precise decision boundary around the space spanned by the correct answers, as evidenced later in Fig. 3 and Fig. 4.
The proposed data augmentation method generates extra synthetic training data by partially introducing the information of the correct answer into negative candidates. Such a formulation allows the models to explore possible interpolation functions on the 'uncharted space' that was not covered by the original training data, as illustrated by Fig. 3. By utilizing the proposed data augmentation scheme, extra negative candidate answers (orange dots) introduce extra decision support and a better estimation of the decision boundary of the model. In addition, the proposed method generates negative candidate answers using the original finite set of visual attribute values without explicitly extracting them, and the generated visual attribute values are still within the finite set. The proposed method hence does not significantly complicate the visual recognition problem, while the newly generated samples could provide additional decision support to the subsequent logical reasoning problem.

Experimental results
We evaluate the proposed data augmentation method on two benchmark datasets: RAVEN [7] and I-RAVEN [8]. We show that by utilizing the proposed CAM-Mix augmentation, we achieve a significant and consistent performance improvement compared with state-of-the-art approaches.

Dataset description
RAVEN dataset [7] has 70,000 samples. Each question contains 16 images, including a 3×3 image matrix with the last one missing, and 8 images of candidate answers. Given the first 8 images of the question, the objective is to deduce the reasoning rules from the first 8 images and choose the most suitable image from the 8 candidate answers as the correct one. The candidate answers are generated by permutation of visual attributes of the correct answer image, and each permuted image is derived by randomly shifting one attribute value. The dataset consists of 7 problem configurations and each sub-patch in a question matrix is constructed using 6 attributes (Angle, Number, Position, I-RAVEN dataset [8] is a variant of the RAVEN dataset [7]. Hu et al. argue that there exist defects in choice panels of the original RAVEN dataset, which may lead to a shortcut to the correct answer, and hence a dataset with impartial candidate answers is built. The negative candidate answers of the I-RAVEN dataset are generated by hierarchically permuting one attribute of the correct answer in three iterations. In each iteration, two child nodes are generated, where one node remains the same with the parent node and the other permutes once. The I-RAVEN dataset shares the same settings and configurations with the original RAVEN dataset except the distinction in generated negative candidate answers. As shown later, the I-RAVEN dataset is more difficult than the original RAVEN dataset, after removing the shortcut.

Experimental settings
We apply the proposed data augmentation to the current best methods in the literature, i.e. CoPINet [23] and SRAN [8], for RAVEN and I-RAVEN, respectively. Both models are compared to various recently published methods [7,22,24]. We test and compare the performance of the vanilla mixup [27] and the proposed CAM-Mix (OR, AND) on both models. For the vanilla mixup, we follow the original formulation and hyperparameter settings in [27], which alpha-blends the candidate images as well as the associated labels with λ ∼ Beta(α, α), α ∈ (0, ∞). The synthetic dataset is combined with the original dataset to train the two models. Specifically, both datasets originally consist of 42,000 samples for training, 14,000 for validation, and 14,000 for testing. With the proposed data augmentation method, we produce 42,000 extra synthetic samples, forming up to 84,000 training samples in total. Neither validation nor testing data are modified. In Fig. 6, we firstly demonstrate that the proposed CAM-Mix (OR) data augmentation could benefit the model training process, as the training loss converges faster than the situation where the data augmentation is not applied.

Visualization of data augmentation
Secondly, we show that the proposed CAM-Mix can help to define a clearer decision boundary around the cluster of correct answer samples with the new synthetic samples. For a better visualization, we plot mini-batches of normalized training data for three different data augmentation methods: vanilla mixup, proposed CAM-Mix (OR) and CAM-Mix (AND). The features extracted from the last hidden layer of applied solution model CoPINet [23] can be visualized as a 2-D plot as shown Table 1: The reasoning accuracy on the RAVEN dataset. The results for other models are referred from [23]. Overall, all three data augmentation methods could improve the classification performance of the state-of-the-art method, CoPINet. Among these methods, the proposed two methods outperform the vanilla mixup [27], the original CoPiNet [23] and other solution models.

Methods
Accuracy in Fig. 4 after dimension reduction to obtain the first two principle components. We plot the original data and the synthetic data generated by the vanilla mixup [27], the proposed CAM-Mix (OR) and CAM-Mix (AND), shown in three subplots of Fig. 4 respectively. As the optimal model parameters of solution models for different sets of training data are different, the data distributions are shown in different subspaces and not directly comparable. From Fig. 4(a), we can see that although the vanilla mixup can generate a certain amount of data falling into the uncharted space around the original candidate answers, the correct candidate answers cannot be well separated from either the original negative candidate answers or the synthetic negative candidate answers. The generated synthetic data do not help the model converge to generate a subspace where the correct candidate answers could be well separated from the negative candidate answers.
Contrastingly, the proposed CAM-Mix could not only generate a remarkable amount of data that fills in the uncharted space around the correct candidate answers, but also form up a clearer decision boundary around the space spanned by the correct candidate answers, as visualized in Fig. 4(b) and Fig. 4(c). This demonstrates the effectiveness of the newly-generated synthetic data to help the model converge to generate a subspace where positive answers and negative answers can be well separated. The synthetic data well fall into the uncharted space around the correct candidate answers. As a plug-and-play tool, the proposed CAM-Mix is advantageous to the visual reasoning systems and various RPM solution models.

Experimental results on the RAVEN dataset
On the RAVEN dataset, Table 1 demonstrates an maximum improvement of 1.9% on the mean accuracy by applying the proposed CAM-Mix (OR) on top of the state-of-the-art model CoPINet [23], and consistent improvements can be witnessed on all the problem configurations. Particularly, on Center, 2×2Grid, 3×3Grid configurations, more than 2.7% of performance improvement can be achieved compared with CoPINet. The proposed method also significantly outperforms other solution models on the RAVEN dataset.
Compared to traditional vanilla mixup [27], the proposed CAM-Mix (OR) achieves a performance gain of 1.3% on the RAVEN dataset. Serving as a data augmentation technique, the vanilla mixup generates extra training samples that may assist the training process, and consequently gains an improvement of 0.6% compared to CoPINet. However, it introduces new attribute values by pair-wise linear interpolation of intensity values as well as the associated labels. As a result, it can only have limited positive effect and it is far less effective compared with the proposed CAM-Mix, which truly mixes up the visual attributes of candidate answers, without explicitly extracting them or generating new attributes values. The proposed CAM-Mix can hence generate a better decision boundary around the space spanned by the correct samples, and hence perform better.
We plot the error rate comparison in Fig. 7 with and without the proposed CAM-Mix (OR), and consistent improvements is demonstrated on all 7 problem configurations. Notably, the proposed CAM-Mix can even strengthen the model reasoning capability on configurations that was almost perfectly resolved (e.g. Left-Right, Up-Down and Out-InCenter).  [23]. After utilizing the proposed data augmentation method on the CoPINet [23], the error rates on all 7 configurations drop.

Experimental results on the I-RAVEN dataset
The I-RAVEN dataset has balanced candidate answer sets and cannot be resolved by building Contrast Module C on candidate answers, and hence is more complicated for existing solution models as shown in experimental results in Table 2. The Table 2: The reasoning accuracy on the I-RAVEN dataset. The results for other models are referred from [8]. All three data augmentation methods significantly improve the accuracy of SRAN [8]. Among these, the proposed CAM-Mix (AND) significantly improves the mean accuracy by 12.9%, and significantly outperforms the vanilla mixup by 9.3%.

Methods
Accuracy proposed data augmentation method is applied on top of the state-of-the-art model, SRAN [8], and gains a remarkable improvement on the I-RAVEN dataset, i.e. the proposed CAM-Mix (AND) results in a performance gain of 12.9% on the average accuracy. On all 7 different configurations, the proposed data augmentation method could significantly boost the performance of the previous best model, SRAN [8], on the I-RAVEN dataset. The largest performance gain on the Out-InCenter configuration is more than 20%.
The proposed CAM-Mix (AND) achieves a performance gain of 9.3% on the RAVEN dataset compared to traditional vanilla mixup [27]. The results further show the limited capability that vanilla mixup brings to existing solution models compared to the proposed CAM-Mix. We plot the error rate comparison in Fig. 8 with and without the proposed CAM-Mix. By utilizing the proposed CAM-Mix (AND) on top of the state-of-the-art solution model, SRAN [8], the error rates on all 7 configurations are significantly reduced. As a generalized plug-and-play tool, the proposed method could boost the performance of various state-of-the-art models, as evidenced in Table 1 and 2.

The impact of image size
The results reported in [8] are based on the images of size 224×224, but other models [7,[22][23][24] are evaluated on the images of size 80×80. For a fair comparison, we conduct a series of experimental studies on the I-RAVEN dataset with images of these two different sizes. We also evaluate the performance of CoPINet model on the I-RAVEN dataset with image size of 224×224. The mean accuracy arises from 46.1% to 50.9%, which shows the positive effect of a larger image size. SRAN outperforms CoPINet on the I-RAVEN dataset by 9.9% using the image size of 224×224. The accuracy is further boosted to 60.6% and 73.0% by utilizing the proposed CAM-Mix (OR), and 60.9% and 73.7% by utilizing the proposed CAM-Mix (AND) on the I-RAVEN dataset with image size of 80×80 and 224×224, respectively. All the experimental results demonstrate the effectiveness of the proposed CAM-Mix method.

Conclusion
In this paper, we propose a data augmentation method, CAM-Mix, for RPM problems by generating more negative candidate answers using gray-level morphological operations. The proposed method mixes-up the visual attributes of original images, without explicitly extracting them. The generated negative candidate answers are semantically similar to the correct answers, and hence a better decision boundary could be defined around the space spanned by the correct answers. The proposed approach is systematically evaluated on benchmark datasets and outperforms the state-of-the-art models. CAM-Mix can serve as a plug-and-play tool to existing solution models and receive a consistent accuracy improvement. Additionally, the proposed data augmentation method is potentially widely beneficial in various multiple-choice problems. In future, we will explore the feasibility of applying the proposed approach on other problems.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.