{'fraction_non_alphanumeric': 0.02519182767865397, 'fraction_numerical': 0.011971896089488768, 'mean_word_length': 4.831536388140162, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Closing the Reality Gap with Unsupervised Sim-to-Real Image Translation

Deep learning approaches have become the standard solution to many problems in computer vision and robotics, but obtaining sufficient training data in high enough quality is challenging, as human labor is error prone, time consuming, and expensive. Solutions based on simulation have become more popular in recent years, but the gap between simulation and reality is still a major issue. In this paper, we introduce a novel method for augmenting synthetic image data through unsupervised image-to-image translation by applying the style of real world images to simulated images with open source frameworks. The generated dataset is combined with conventional augmentation methods and is then applied to a neural network model running in real-time on autonomous soccer robots. Our evaluation shows a significant improvement compared to models trained on images generated entirely in simulation.


Introduction
In recent years, deep learning approaches became the standard solution to many problems in computer vision, such as classification [5], object detection [16], or semantic segmentation [18]. Efforts were made to reduce the computational complexity in order to deploy them to mobile devices [9]. These approaches usually require a vast amount of training data which can either be generated through human labor or be generated synthetically. Generating training data through human labor can result in datasets of high quality, but it is a cumbersome and expensive task. The CamVid dataset [4] contains detailed semantic labels and uses preprocessing to assist human labelers, but annotating a single frame takes 20 to 25 minutes. Multiple volunteers were tasked to perform the labeling, but only about 15 % of the volunteers delivered acceptable results. Similar problems exist in other datasets such as the PASCAL VOC challenge [6] or in the COCO dataset [14].
Recently, a trend can be seen to approaches that rely on simulated and synthetic data. The SYNTHIA dataset, for instance, consists of 213400 images and pixel-accurate semantic annotations as well as depth maps generated with the Unity framework [19]. Computer games can also be used to generate images that can then be labeled manually [17]. Unfortunately, data generated in a simulated environment often does not directly transfer to reality. This issue is referred to as the reality gap [12].
Hess et al. [8] introduced an environment to create annotated training data in a RoboCup Standard Platform League (SPL) setting and demonstrated the feasibility of performing a semantic segmentation on that data. A major challenge in the RoboCup SPL is the perception of the field in a diverse set of lighting conditions. Low quality cameras result in images with low contrasts and limited processing power usually requires using fast conventional computer vision approaches. Frameworks such as TensorFlow Lite [1] or CompiledNN [25] made utilizing neural networks in mobile and low-end devices more feasible.
In our work, we synthetically generate images with the tools provided by [8] and transform them with unsupervised image-to-image translation [10] and domain randomization [26] so that they can be used as training data for any kind of deep learning task. We use this dataset to train a semantic segmentation that is able to run in real-time on a NAO v6 robot. Our approach can generally be applied to any other domain where computing power is sparse and flexibility and reliability plays an important role. All required software dependencies are open source. Our evaluation shows that models trained with our method perform noticeably better than models trained with data directly generated from simulation as well as with generated data that is expanded with conventional augmentation techniques. In summary, our main contributions are: -We developed a method using publicly available state of the art image-toimage transformation frameworks and demonstrate that it can be used to generate high quality datasets that allow training highly performing models. -We introduce a multi-class semantic segmentation model architecture that is capable of running in real-time on a NAO v6 robot.
The remainder of this paper is organized as follows: After summarizing the related work in Sec. 2, we describe our data generation approach in Sec. 3. In Sec. 4, we present our semantic segmentation model architecture. Lastly, multiple models generated with different data augmentations are evaluated and discussed in Sec. 5 and Sec. 6 respectively. An overview of the proposed method is depicted in Fig. 1.

Related Work
There are two remarkable works in the area of closing the reality gap with imageto-image translation: Bousmalis et al. [3] use domain adaptation and domainadversarial neural networks to utilize synthetic training data in an end-to-end learning approach in order to learn robotic grasping. Bewley et al. [2] use imageto-image translation to transfer a vision-based driving policy from simulation to reality. Instead of explicit representations such as a semantic segmentation, their end-to-end approach uses more implicit representations. In contrast to our proposed method, both approaches have in common that they utilize the imageto-image translation in an end-to-end approach. Flexibility in the postprocessing plays an important role in RoboCup settings, which is not given with the current state of the art of end-to-end approaches. Using intermediate representations such as pixel-accurate labels allows a higher flexibility and versatility. A dataset using such low level representations can be used both for semantic segmentation and for high level classification and object detection. Deep learning approaches for robot vision in RoboCup were first applied in the humanoid league, where Schnekenburger et al. [20] use a segmentation to detect different field features such as line intersections and objects. All classes except for the detection of other robots performed satisfactory. Van Dijk et al. [27] propose a novel model architecture without any residual connections for a semantic segmentation, which was able to be executed in close to real-time on a typical smartphone CPU, but lacks the ability to properly detect complex features or multiple classes at once. In contrast to the humanoid league, the SPL, which uses the SoftBank NAO robots, is more constrained in terms of hardware. Nevertheless, Hess et al. [8] trained a simple classifier model to demonstrate the feasibility of using synthetic images in such a scenario. Szemenyei et al. [24] propose a novel, small semantic segmentation model that uses images generated with UERoboCup for pre-training the model and eventually tune it with real images. However, this approach does not focus on the more extreme conditions that games in environments with natural lighting require. Furthermore, Poppinga et al. [15] proposed a robot detection framework for which data obtained in a simulation was used to learn additional features that are hard to label manually, such as robot distances.
In this section, after briefly describing the background and the tools generating the simulated data, we give insights into the Multimodal Unsupervised Imageto-Image Translation (MUNIT) framework [10], which we use for the sim-to-real image translation, and describe the online data augmentation methods we used.

RoboCup and UERoboCup
While RoboCup provides a benchmark to quantitatively compare the progress over time, the community is small and lacks labeled training data for deep learning approaches. Efforts were made to create a community-driven database for labeled real images [7], but the data required for specific tasks and pixel-accurate labels are rarely available.
UERoboCup is an application based on Unreal Engine that allows generating game situations with multiple robots from the view of a specific robot [8] and is capable of generating pixel-accurate semantic annotation. To provide a more accurate representation of the environment, we added additional labels relevant to the SPL context, such as the penalty mark and the goal bar as well as an adapted appearance of the robots to that of the latest NAO robot generation. In addition, we increased the variation in the generated images to reflect reality more properly. This is achieved by a variable camera pitch instead of a fixed one and the definition of a skeleton for the previously static robot mesh, which allows dynamic robot poses. This procedure is referred to as domain randomization [26]. Furthermore, the robot skeleton can be used for a more detailed segmentation of individual robot limbs, allowing, for instance, a pose detection. We also export meta data such as the extrinsic camera parameters and the poses of robots in the standardized JSON format. Such information is difficult to annotate manually, therefore learning further characteristics, such as the distance to another robot, as shown in [15], mainly relies on synthetic data.
We generated a set of 10000 images and labels with UERoboCup. A generated image with the corresponding converted segmentation mask can be seen in the overview in Fig. 1.

Image Post Processing
UERoboCup only creates images of a plain RoboCup scene taking place in a white room. However, during an actual game, the background is cluttered with a wide range of different objects, such as people walking around. In order to make potential deep learning applications understand the concept of unwanted background clutter, we replace the background with structured images, similar to [26]. This can be considered another variation of domain randomization. We use images from the COCO test set [14]. Even though these images do not exactly represent how a scene would look like at a RoboCup event, we found that they are well-suited to help deep learning applications differentiate between relevant foreground and irrelevant background clutter.

MUNIT
MUNIT assumes images from two different domains x 1 ∈ X 1 and x 2 ∈ X 2 and, given samples drawn from the two marginal distributions p(x 1 ) and p(x 2 ), without access to the joint distribution p(x 1 , x 2 ), estimate the conditionals p(x 1 |x 2 ) and p(x 2 |x 1 ) with the image-to-image translation models p(x 1→2 |x 2 ), where x 1→2 is a sample resulting from translating x 1 to X 2 [10]. Due to the unsupervised nature of MUNIT, no explicit labeling has to be performed on any of the sample images. The learned mapping is multimodal, thus multiple different images with different styles from domain x 1 ∈ X 1 can be applied to the same image x 2 ∈ X 2 and each time a different image with the style from domain x 1 ∈ X 1 is computed.

Style Transfer
MUNIT requires a test set and a training set of images for the two classes X 1 and X 2 . For this application, the two classes are real images recorded by a robot's camera X real and simulated images created with UERoboCup X sim . For the real domain, we select images from previous RoboCup events and from ImageTagger [7], accumulating to an overall of 885 images in the training set and 155 images in the test set. We found that a large variance in the training images is essential for MUNIT to generate useful results. Since the images generated by UERoboCup are random, any subset with about the same size can be used. Note that in this subset, we already replaced the background with a random image from the COCO dataset. In the simulated domain, we used 1000 images for training and 200 for testing. We train on an NVidia Titan V with MUNIT default settings for 70000 epochs. Due to memory limitations, we slightly decreased the amount of the generator and discriminator filters. We noticed a convergence after 50000 epochs, with no further significant improvements from that point. To generate the processed images, we took three random style images from the set of real images and applied them to each image generated by UERoboCup, resulting in 30000 different images. An example can be seen in Fig. 2.

Data Augmentation
Data augmentation is a type of data regularization and helps to avoid overfitting. Additionally, data augmentation can be used to enhance the size and quality of datasets with warping or oversampling methods [21].
We used the following online data augmentations during training: Vertical image flipping, Gaussian noise, multiply, add (RGB and HSV), simplex noise, motion blur, contrast normalization, and simulated sun patches.
It is essential that the model learns to handle extreme environmental situations with patches of light and shadow. Since this is not captured with sufficient variance in our dataset, we introduce an additional domain randomization method during online data augmentation by simulating patches on the field that Fig. 2. After changing the background in the simulated image to a random image from the COCO dataset (left), we apply the image-to-image translation learned by MUNIT to it (middle). In addition to the offline augmentation obtained with the simto-real translation, we perform an online augmentation during training (right). Beside standard augmentations such as blur, we also introduce a simple simulation of sun light patches on the field. are illuminated by the sun. We implement this augmentation by generating multiple random polygons consisting of three to six points in the frame and multiply these areas with a random factor. A sample augmentation is shown in Fig. 2.

Semantic Segmentation
Designing models that are capable of being executed in real-time on platforms with limited resources is difficult due to computing constraints. In our application, the performance baseline for the design of the model is the NAO's camera frame rate of 30 fps. We allocate one CPU core for the processing of the frames of one camera, which means that the model must process each frame at the same rate. As inference framework, we use CompiledNN [25].
The model architecture is based on the U-Net architecture [18] and incorporates features from MobileNet [9]. The U-Net architecture consists of an encoder and a decoder with multiple residual connections between the encoder and the decoder. The reduced model from this work uses only two residual connections and downscales the image twice. MobileNet was designed to be deployed to mobile devices and is therefore optimized to run with limited computing power. This is mainly realized by replacing convolutions with separable convolutions [9]. For an additional acceleration, pooling layers are replaced with convolutions with a corresponding stride, as proposed in [23]. We use batch normalization for regularization [11] and LeakyReLU [28] as activation functions. The resulting model has 12909 learnable parameters and is shown in Tab. 1.
Our proposed model is capable of performing multi-class predictions (in our case ball, goal posts, lines, and robots). Due to limitations in CompiledNN, no softmax activation is applied to the last layer. This means that the model detects all mentioned classes independently from each other. Note that a binary cross entropy or something similar must be used as the loss function. Furthermore, it can be desirable to have multiple independent classifications outputs. For instance, if the model is not certain if the arm of a robot is a similarly looking goal Table 1. Our proposed segmentation model architecture. The scale refers to the tensor size relative to the input size. N is the amount of how often the layer in the row is repeated and F represents the amount of filters.

Layer
Scale N F Input 1 1 3 3 × 3-SConv2D, BN, LeakyReLU post, the likelihood of both outputs would be small with a softmax activation. By keeping the output layers independent from each other, the model can classify the arm of a robot both as part of a robot but also as a goal post. This can be validated in the postprocessing by applying additional domain knowledge and thus discarding false positives. Note that ideally, the model is able to capture this from context, but we found that increasing the model size more leads to an unacceptably low inference time. The maximum amount of features is limited by the complexity of the model. We achieved good results with predicting five and less classes, but it is to be expected that a larger amount of independent output predictions yields worse predictions, if the base model is not adapted.
We perform different experiments to find a working model architecture and look for a compromise between detection accuracy and inference time. The runtime was measured on a single CPU core of the NAO v6 (Intel Atom E3845@1.91 GHz [22]) using CompiledNN [25]. The results for different image resolutions are shown in Tab. 2. As the maximum feasible input resolution for real-time operation is 14 ms, the original input image is subsampled accordingly. The resulting aliasing should not affect the neural model as it should learn to ignore it. In order to evaluate the performance of the model and the synthetic dataset, we manually labeled 348 images, which we also used in the MUNIT augmentation resembling varying environment conditions with the five classes field, lines, robots, goal post, and ball. We evaluate the mean Average Precision (mAP) metric. In order to evaluate the effect of the different augmentation techniques, we use five different configurations of our dataset to train the previously described model: 1. no augmentation at all (only the raw images generated by UERoboCup are used, this is the baseline) 2. conventional augmentation without sun (all the augmentations described in Sec. 3.5 except for the sun patch augmentation) 3. conventional augmentation (including the sun patch augmentation) 4. only the image-to-image translation augmentation performed with MUNIT 5. the image-to-image translation combined with all previously described conventional augmentations We train all five models with a subset of 8000 images and 2000 images as test set with the same augmentations in a batch size of 128 using the Adam optimizer [13] with an initial learning rate of 0.1. We decay the learning rate by 0.5, if the loss on the validation set does not decrease for 10 epochs and we terminate training after 20 epochs without improvement, which was usually reached in less than 100 epochs.
The different precision-recall curves are visualized in Fig. 3. The mAPs for all models and all classes can be seen in Tab. 3. We report the precision-recall curves for the individual classes as well as micro-averaged for all classes. Note that the computed mAP represents the classification performance of each individual pixel over all test images and not the classification performance of the object instances.

Discussion
As expected, the model without any augmentation performs worst with an overall mAP of 0.4203, which shows that reality gap is an issue for plain images generated in UERoboCup. The ball and the goals are rarely detected with a slightly higher mAP for robots and lines. Since a quantitative comparison with related works [20,27,24] is not possible due to different metrics that do not capture the problem well, this model is used as baseline of what is possible with purely synthetic data generated in UERoboCup. Just by performing basic augmentation (all augmentations mentioned above except for the sun patch augmentation), the performance of the model increases drastically to an overall mAP of 0.914. Most classes receive a significant increase in mAP, particularly the robot, line, and ball classes. The goal classes' mAP only increase slightly. Adding the sun patch augmentation increases the overall mAP again slightly to 0.9165. The sun patches augmentation helps improving the line, goal, and robot classification, but results in a drop in the ball class, while the field prediction stays about the same. This demonstrates that the sun patch augmentation in fact helps. When considering the model trained solely with data augmented with MU-NIT, a clear rise in overall mAP to 0.9536 can be seen, with an improvement for all classes. Adding conventional augmentation to the MUNIT augmentation results in a slight overall mAP increase to 0.9647. While the ball and goal classes' mAPs drop again slightly, all other classes' performances increase. This shows that our proposed method yields the desired result. Particularly highly unbalanced classes benefit from the image-to-image translation augmentation.
The ball class is consistently the one with the lowest mAP, which is likely due to highly unbalanced training samples (with the ball being significantly smaller than any other objects). The same applies to the goal post class. The ball is one of the most challenging objects to detect in different lighting conditions since it is small, very close to the ground and throws shadows due to to its spherical shape. Lastly, the model operates on a small resolution which makes it impossible to detect the ball at large distances.
Despite its little size with only 12909 trainable parameters (opposed to 300000 parameters in [20]), our proposed model trained with our synthetically generated training data seems to perform better than the approaches proposed The segmentation model was applied to images recorded in extreme lighting conditions. Multiple samples can be seen in Fig. 4.

Conclusion
We proposed a method for segmenting an image in real-time at a reduced resolution into five different classes by utilizing a deep learning approach. We generated all training data synthetically and evaluated the results on a test set made of manually labeled real data.