{'fraction_non_alphanumeric': 0.031693537274355796, 'fraction_numerical': 0.013136741536906894, 'mean_word_length': 5.015057328360271, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MSeg: A Composite Dataset for Multi-domain Semantic Segmentation

We present MSeg, a composite dataset that unifies semantic segmentation datasets from different domains. A naive merge of the constituent datasets yields poor performance due to inconsistent taxonomies and annotation practices. We reconcile the taxonomies and bring the pixel-level annotations into alignment by relabeling more than 220,000 object masks in more than 80,000 images, requiring more than 1.34 years of collective annotator effort. The resulting composite dataset enables training a single semantic segmentation model that functions effectively across domains and generalizes to datasets that were not seen during training. We adopt zero-shot cross-dataset transfer as a benchmark to systematically evaluate a model's robustness and show that MSeg training yields substantially more robust models in comparison to training on individual datasets or naive mixing of datasets without the presented contributions. A model trained on MSeg ranks first on the WildDash-v1 leaderboard for robust semantic segmentation, with no exposure to WildDash data during training. We evaluate our models in the 2020 Robust Vision Challenge (RVC) as an extreme generalization experiment. MSeg training sets include only three of the seven datasets in the RVC; more importantly, the evaluation taxonomy of RVC is different and more detailed. Surprisingly, our model shows competitive performance and ranks second. To evaluate how close we are to the grand aim of robust, efficient, and complete scene understanding, we go beyond semantic segmentation by training instance segmentation and panoptic segmentation models using our dataset. Moreover, we also evaluate various engineering design decisions and metrics, including resolution and computational efficiency. Although our models are far from this grand aim, our comprehensive evaluation is crucial for progress. We share all the models and code with the community.


INTRODUCTION
When Papert first proposed computer vision as a summer project in 1966 [1], he described the primary objective as "...a system of programs which will divide a vidisector picture into regions such as likely objects, likely background areas and chaos." Five decades later, computer vision is a thriving engineering field, and the task described by Papert is known as semantic segmentation [2], [3], [4], [5], [6], [7].
Have we delivered on Papert's objective? A cursory examination of the literature would suggest that we have. Hundreds of papers are published every year that report ever-higher accuracy on semantic segmentation benchmarks such as Cityscapes [8], Mapillary [9], COCO [10], ADE20K [11], and others. Yet a simple exercise can show that the mission has not been accomplished. Take a camera and begin recording as you traverse a sequence of environments: for example, going about your house to pack some supplies, getting into the car, driving through your city to a forest on the outskirts, and going on a hike. Now perform semantic segmentation on the recorded video. Is there a model that will successfully perform this task?
A computer vision professional will likely resort to multiple models, each trained on a different dataset. Perhaps a model trained on the NYU dataset for the indoor portion [12], a model trained on Mapillary for the driving portion, and a model trained on ADE20K for the hike. Yet this is not a satisfactory state of affairs. It burdens practitioners with developing multiple models and implementing a controller that decides which model should be used at any given time. It also indicates that we haven't yet arrived at a satisfactory vision system: after all, an animal can traverse the same environments with a single visual apparatus that continues to perform its perceptual duties throughout.
A natural solution is to train a model on multiple datasets, hoping that the result will perform as well as the best dedicated model in any given environment. As has previously been observed, and confirmed in our experiments, the results are far from satisfactory. A key underlying issue is that different datasets have different taxonomies: that is, they have different definitions of what constitutes a 'category' or 'class' of visual entities. Taxonomic clashes and inconsistent annotation practices across datasets from different domains (e.g., indoor and outdoor, urban and natural, domain-specific and domain-agnostic) substantially reduce the accuracy of models trained on multiple datasets.
In this paper, we take steps towards addressing these issues. We present MSeg, a composite dataset that unifies semantic segmentation datasets from different domains: COCO [10], ADE20K [11], Mapillary [9], IDD [13], BDD [14], Cityscapes [8], and SUN RGB-D [15]. A naive merge of the taxonomies of the seven datasets would yield more than 300 classes, with substantial internal inconsistency in definitions. Instead, we reconcile the taxonomies, merging and splitting classes to arrive at a unified taxonomy with 194 categories. To bring the pixel-level annotations in conformance with the unified taxonomy, we conduct a largescale annotation via the Mechanical Turk platform and produce compatible annotations across datasets by relabeling object masks.
The resulting composite dataset enables training unified semantic segmentation models that come to a step closer to delivering on Papert's vision. MSeg training yields models that exhibit much better generalization to datasets that were not seen during This enables training models that perform consistently across domains and generalize better. Input images in this figure were taken from datasets that were not seen during training.
training. We adopt zero-shot cross-dataset transfer as a proxy for a model's expected performance in the "real world" [16]. We train models on MSeg and test on datasets that are disjoint from MSeg. In this mode, MSeg training is substantially more robust than training on individual datasets or training on multiple datasets without the reported taxonomic reconciliation.
Accurate semantic segmentation is typically not enough for a typical computer vision practitioner as it lacks any sense of an object. Instance segmentation extends semantic segmentation with finding semantically meaningful instances, and panoptic segmentation [17] combines instance segmentation with semantic segmentation. Similarly, we evaluate the capabilities of MSeg dataset for these problems as the component training datasets have labels for the instances. Moreover, the practical applications typically come with computational efficiency requirements. For this purpose, we further evaluate the computational efficiency of our models at various resolutions to understand the computational efficiency and accuracy trade-offs. Although the results of these experiments are still far from the ultimate requirements of the practitioners, it gives a complete understanding of the MSeg dataset and models. Moreover, we share all trained models, data, and source code with the community.

Cross-domain semantic segmentation
Annotation for semantic segmentation is an exceptionally costly process as pixel-level annotations are needed. Utilizing multiple or additional existing semantic segmentation datasets is a sensible direction to ease this cost and effectively increase the dataset size. Mixing segmentation datasets has primarily been done within a single domain and application, such as driving. Ros et al. [18] aggregated six driving datasets to create the MDRS3 meta-dataset, with approximately 30K images in total. Bevandic et al. [19] mix Mapillary Vistas, Cityscapes, the WildDash validation set, and ImageNet-1K-BB (a subset of ImageNet [20] for which bounding box annotations are available) for joint segmentation and outlier detection on WildDash [21]. On a smaller scale, [22], [23] mix Mapillary, Cityscapes, and the German Traffic Sign Detection Benchmark. In contrast to these works, we focus on semantic segmentation across multiple domains and resolve inconsistencies between datasets at a deeper level, including relabeling incompatible annotations.
Varma et al. [13] evaluate the transfer performance of semantic segmentation datasets for driving. They only use 16 common classes, without any dataset mixing. They observe that crossdataset transfer is significantly inferior to "self-training" (i.e., training on the target dataset). We have observed the same outcomes when models are trained on individual datasets, or when datasets are mixed naively.
An underlying issue that impedes progress on unified semantic segmentation is the incompatibility of dataset taxonomies. In contrast to the aforementioned attempts, we directly address this issue by deriving a consistent taxonomy that bridges datasets from multiple domains.

Domain adaptation and generalization
Training datasets are biased and deployment in the real world presents the trained models with data that is unlike what had been seen during training [26]. This is known as covariate shift [27] or selection bias [28], and can be tackled in the adaptation or the generalization setting. In adaptation, samples from the test distribution (deployment environment) are available during training, albeit without labels. In generalization, we expect models to generalize to previously unseen environments after being trained on data from multiple domains. An extensive collection of these methods have recently been summarized and benchmarked in various settings [29], [30], and we refer the interested reader to these studies. We summarize the most relevant ones and discuss their relationship with our work.
We operate in the generalization mode and aim to train robust models that perform well in new environments, with no data from the target domain available during training. Many domain generalization approaches are based on the assumption that learning features that are invariant to the training domain will facilitate generalization to new domains [31], [32]. Volpi et al. [33] use distributionally robust optimization by considering domain difference as noise in the data distribution space. Bilen and Vedaldi [34] propose to learn a unified representation and eliminate domainspecific scaling factors using instance normalization. Mancini et al. [32] modify batch normalization statistics to make features and activations domain-invariant. Recently, the tools from causality has been applied to the problem of domain generalization [35], [36]. The causal relationship between images and their labels are expected to generalize to novel domains whereas environment specific relationships are expected to be anti-causal.
The aforementioned domain generalization methods assume that the same classifier can be applied in all environments. This relies on compatible definitions of visual categories which our work enables. Our work is complementary and can facilitate future research on domain generalization by providing a compatible taxonomy and consistent annotations across semantic segmentation datasets from different domains.

Visual learning over diverse domains
In order to study the problem of learning from multiple datasets, the Visual Domain Decathlon [37] introduced a benchmark over ten image classification datasets. However, it allows training on all of them. More importantly, its purpose is not training a single classifier. Instead, they hope domains will assist each other by transferring inductive biases in a multi-task setting. Triantafillou et al. [38] proposed a meta-dataset for benchmarking few-shot classification algorithms.
For the problem of monocular depth estimation, Ranftl et al. [16] use multiple datasets and mix them via a multi-task learning framework. We are inspired by this work and aim to facilitate progress on dataset mixing and cross-dataset generalization in semantic segmentation. Unlike the work of Ranftl et al., which dealt with a geometric task (depth estimation), we are confronted with inconsistencies in semantic labeling across datasets, and make contributions towards resolving these.
A recent robust vision challenge [39] addresses robustness and generalization by learning from multiple datasets. The challenge is using our dataset but in a few-shot generalization setting instead of zero-shot generalization. Although our models are trained in the zero-shot regime, the resulting performance is competitive with few-shot models showing our approach is a promising direction to develop general and robust vision models.
Recently the problem of instance segmentation has gained interest. It segments an image into semantically meaningful instances [44], [45], [46], [47], [48]. However, it only consider things, not stuff [49]. Panoptic segmentation [17] is an emerging scene understanding task which jointly performs instance segmentation and semantic segmentation over both thing and stuff classes.
The vast majority of train datasets in our composite dataset include instance labels; hence, we extend our dataset to instance segmentation and panoptic segmentation. We show successful panoptic and instance segmentation results using our dataset, validating the usefulness of MSeg for these future directions.

THE MSEG DATASET
Our proposed dataset, MSeg, is a composite dataset as it is composed of multiple diverse datasets after a label unification process. We consider a large number of semantic segmentation datasets and select a subset to create MSeg. We list the semantic segmentation datasets used in MSeg in Table 1. The datasets that were not used, and reasons for not including them, are listed in the Table 2.
Our guiding principle for selecting a training/test dataset split is that large, modern datasets are most useful for training, whereas older and smaller datasets are good candidates for testing. We test zero-shot cross-dataset performance on the validation subsets of these datasets. Note that the data from the test datasets (including their training splits) is never used for training in MSeg. For validation, we use the validation subsets of the training datasets listed in Table 1. Next, we describe the dataset-specific choices we make.
We use the free, academic version of Mapillary Vistas [9]. In this we forego highly detailed classification of traffic signs, traffic lights, and lane markings in favor of broader access to MSeg.
For COCO [10], we use the taxonomy of COCO Panoptic as a starting point, rather than COCO Stuff [50]. The COCO Panoptic taxonomy merges some of the material-based classes of COCO Stuff into common categories that are more compatible with other We include only fully-manually labeled datasets in MSeg. ApolloScape uses a semi-automatic label generation pipeline. VIPER [57] Driving (Synthetic) 134,097 We do not use synthetic data. INTERIORNET [58] Indoor (Synthetic) 20,000,000 We do not use synthetic data. MATTERPORT3D [59] Indoor 194,400 Annotated in 3D, rather than in 2D. STANFORD 2D-3D-S [60] Indoor 70,496 Data is semantically annotated on 3D point clouds, rather than in 2D; the dataset includes only 13 object classes, which is few for an indoor environment. OPENIMAGES V5 [61], [62], [63] Everyday Objects 957,561 Provides only object segmentations (no segmentation of stuff categories). A2D2 [64] Driving (Germany) 40,000 Was not available at time of paper preparation. MINC OPENSURFACES [65] Materials Annotation categories are material-centric, instead of object-or stuffcentric. REPLICA DATASET [66] Indoor (Synthetic) N/A Real data is not available (images are synthetically rendered).
datasets. (E.g., floor-marble, floor-other, and floor-tile are merged into floor.) Naively combining the component datasets yields roughly 200K images with 316 semantic classes (after merging classes with synonymous names). We found that training on naively combined datasets yields low accuracy and poor generalization. We believe the main cause for this failure is inconsistency in the taxonomies and annotation practices in the different datasets. The following subsections explain these issues and our solution.

Taxonomy
In order to train a cross-domain semantic segmentation model, we need a unified taxonomy. In order to unify the training taxonomies, we need to perform merging and splitting operations. Merging and splitting of classes from component datasets have different drawbacks. Merging is easy and can be performed programmatically, with no additional labeling. The disadvantage is that labeling effort that was invested into the original dataset is sacrificed and the resulting taxonomy has coarser granularity. Splitting, on the other hand, is labor-intensive. To split a class from a component dataset, all masks with that class need to be relabeled. This provides finer granularity for the resulting taxonomy, but costs time and labor.
In order to make these decisions, we consider two primary objectives. First, as many classes should be preserved as possible. Merging classes can reduce the discriminative ability of the resulting models. Second, the taxonomy should be flat, rather than hierarchical, to maximize compatibility with standard training methods. In order to satisfy these objectives, we followed a sequence of decision rules, summarized in Figure 2, to determine split and merge operations on taxonomies of the component datasets. Although the choice of classes in any taxonomy is somewhat arbitrary (e.g. a taxonomy for a fashion dataset versus a wildlife preservation dataset will be highly-task dependent), our decision tree allows us to make objective choices.
Given a query category, if a category with a similar or related name is found in any one of the other datasets' taxonomies, we manually inspect the image masks. If we determine the category meaning to be equivalent, (e.g. Mapillary "On Rails" and COCO-1. A second version of the benchmark, WildDash-v2, was introduced after the completion of our experiments and publication of our CVPR 2020 paper. Panoptic "train"'), we combine class masks under a common name.
If no similar or related name is found in any other dataset's taxonomy, we must inspect a large number of images from each such dataset in order to determine if the category is present, but simply unannotated or annotated differently. If a category is truly unique to this dataset (i.e. present in no other dataset), we add this class to the unified taxonomy (e.g. IDD's "auto-rickshaw" or COCO-Panoptic's "surfboard" or "tie").
In the case that a similar category name exists, but was not equivalent (e.g. COCO-Panoptic "person" and Mapillary "person"), or a category is present in another dataset's images, a more challenging task remains. In such a scenario, we seek to answer the question, is there a category in any other dataset that is a strict superset or subset of query category? If superset/subsets were to be merged, current boundaries must keep unrelated classes separate. If the answer is no, (e.g. Mapillary "manhole" vs. COCO-Panoptic's "road, or IDD's "non-drivable fallback" and Cityscapes "terrain"), we mark masks as "unlabeled." Given there is no hierarchical relationship, one would need to re-draw new boundaries (an endeavor which we do not undertake). However, if there is a category in another dataset that is a strict superset or subset of the query category (e.g. ADE20K "glass" and COCO-Panoptic's "wine glass"), there is a possibility to "shatter" the superset category. Shattering the superset must be enforceable to an MTurk worker, given a typical field of view, without drawing a new boundary (e.g. ADE20K "swivelchair" and COCO-Panoptic "chair" is possible since boundaries are shared, and the field of view generally accommodates this, unlike "building" vs. "house"). If some shattered masks do not fall into any pre-existing subset, we add an "-other" category. However, if such shattering is impossible without drawing a new boundary, (e.g. ADE20K's "flower", "tree", "palm" vs. Cityscapes "vegetation"), we merge both into the superset. If such shattering is impossible because the classification is not enforceable to a Mechanical Turk worker given a typical field-of-view (e.g. COCO-Panoptic's "dining table" and "table-merged"), we merge both into the superset category.
Following this process, we condensed the 316 classes obtained by merging the component datasets into a unified taxonomy of 194 classes. The full list and class statistics are given in Figure 3 and further described and visualized in the appendix. Each of these classes is derived from classes in the component datasets. We also report the statistics of the number of relabeled masks in the appendix.
After this process, an MSeg category can have one of the following relationships to classes in a component dataset: (a) it can be in direct correspondence to a class in a component taxonomy, (b) it can be the result of merging a number of classes from a component taxonomy, (c) it can be the result of splitting a class in a component taxonomy (one-to-many mapping), or (d) it can be the union of classes which are split from different classes in the component taxonomy.

Relabeling Instances of Split Classes
We utilize Amazon Mechanical Turk (AMT) to relabel masks of classes that need to be split. We re-annotate only the datasets used for learning, leaving the evaluation datasets intact. Instead of recomputing boundaries, we formulate the problem as multiway classification and ask annotators to classify each mask into   finer-grained categories from the MSeg taxonomy. We include an example labeling screen, workflow and labeling validation process in the appendix In total, we split 31 classes and relabel 221,323 masks. We visualize some of these split operations in Figure 4 and provide additional details in the appendix. AMT workers sometimes submit inaccurate, random, or even adversarial decisions [67]. To ensure annotation quality, we embed 'sentinel' tasks within each batch of work [68], [69], [70], constituting at least 10% of each batch. These sentinels are tasks for which the ground truth is unambiguous and is manually annotated by the authors. We use the sentinels to automatically evaluate the reliability of each annotator so that we can direct work towards more reliable annotators. In order to guarantee acceptable accuracy, five workers annotate each batch, and the work is resubmitted until all submitted batches meet a 100% sentinel accuracy. Afterwards, the category is determined by majority vote; categories that do not meet these criteria are manually labeled inhouse by an expert annotator (one of the authors).

EXPERIMENTAL RESULTS
We quantitatively and qualitatively study MSeg for learning multidomain scene understanding models. We start with discussing the implementation details in Section 4.1. Then, we evaluate the performance of semantic segmentation models trained using MSeg data and the impact of our universal taxonomy in Section 4.2. Finally, we study the practical multi-domain scene understanding problem beyond semantic segmentation. We train multi-domain semantic, instance, and panoptic segmentation models using larger compute and evaluate them in terms of accuracy and efficiency. We share the source code, trained models, and the dataset for full reproducibility 2 .

Implementation details
Our semantic segmentation experiments largely follow the protocol introduced by Zhao et al. [71]. However, we used the HRNet-V2-W48 architecture [72], where W48 indicates the width of the high-resolution convolution. We use HRNet without Object Contextual Representations (OCR) and with synchronized BN. For instance segmentation, we use Mask R-CNN [73] with a ResNeXt-101-32x8d-FPN [74] backbone. For panoptic segmentation, we use Panoptic-FPN [75] with a ResNet-101 backbone. We use 1080p resolution in our experiments with the crop size of 713 × 713. When we further evaluate the impact of resolution, we use crop sizes of 473 × 473 for 480p and 593 × 593 for 720p.
We use SGD with momentum 0.9 and a weight decay of 10 −4 . We use a polynomial learning rate with power 0.9. Base learning rate is set to 10 −2 . For multi-scale inference, we use a multi-scale accumulation of probabilities, with scales of 0.5 to 1.75 with 0.25 increments.
When forming a minibatch of size m from multiple datasets, we evenly split the minibatch by the number of training datasets n, meaning each dataset will contribute m /n examples to each minibatch. Accordingly, there is no notion of "epoch" for the unified dataset during our training, but rather only total samples seen from each dataset. For example, in a single effectual "COCO epoch", Mapillary will complete more than 6 effectual epochs, as its dataset is less than 1 6 th the size of COCO. For all of our quantitative studies, we train until one million crops from 2. Trained models, data, and source code: https://github.com/mseg-dataset each dataset's images have been seen. We further train models until three million crops have been seen to share them with the community. We use a mini-batch size m = 35.
Image resolution is inconsistent across component datasets. For example, Mapillary contains many images of resolution ∼2000 × 4000, while most ADE20K images have resolution ∼300 × 400. Before training, we use 2× or 3× super-resolution [76] to first upsample the training datasets with lower resolution to a higher one (at least 1000p). At training time, we resize images from different datasets to a consistent resolution of 1080p. Before training, resizing images from different datasets to a canonical size is important: during training, a crop of a fixed size is fed to the network, and such crops could represent dramatically different relative portions of the image if images' sizes differ drastically. Specifically, in our experiments, we resize all images such that their shorter side is 1080 pixels (while preserving aspect ratios) and use a crop size of 713 × 713px. At test time, we resize the image to one of three different resolutions (360/720/1080 as the images' shorter side), perform inference, and then interpolate the prediction maps back to the original resolutions for evaluation. The resolution level (360/720/1080) is set per dataset.

Using the MSeg taxonomy on a held-out dataset
At inference time, at each pixel we obtain a vector of probabilities over the unified taxonomy's m u categories. These unified taxonomy probabilities must be allocated to test dataset taxonomy buckets. For example, we have three separate probabilities in our unified taxonomy for 'motorcyclist', 'bicyclist', and 'rider-other'. We sum these three together to compute a Cityscapes 'rider' probability. We implement this remapping from m u classes to m t classes as a linear mapping P from R mu to R mt . The weights P ij are binary 0/1 values and are fixed before training or evaluation; the weights are determined manually by inspecting label maps of the test datasets. P ij is set to 1 if unified taxonomy class j contributes to evaluation dataset class i, otherwise P ij = 0.

Zero-shot Quantitative Results
We evaluate zero-shot generalization only for semantic segmentation since our test datasets typically do not have instance labels. We use the MSeg training set to train a unified semantic segmentation model using 1 million crops. Table 3 lists the results of zero-shot transfer of the model to MSeg test datasets. Note that none of these datasets were seen by the model during training. For comparison, we list the performance of corresponding models that were trained on a single training dataset, among the training datasets that were used to make up MSeg. For reference, we also list the performance of 'oracle' models that were trained on the training splits of the test datasets. Note that WildDash does not have a training set, thus no 'oracle' performance is provided for it.
The results in Table 3 indicate that good performance on a particular test dataset sometimes is obtained by training on a training dataset that has compatible priors. For example, training on COCO yields good performance on VOC, and training on Mapillary yields good performance on KITTI. But no individual training dataset yields good performance across all test datasets. In contrast, the model trained on MSeg performs consistently across all datasets. This is evident in the aggregate performance, summarized by the harmonic mean across datasets. The harmonic mean mIoU achieved by the MSeg-trained model is 28% higher than the accuracy of the best individually-trained baseline (COCO).  When compared with oracle models, training on MSeg is competitive with oracle models on many datasets, validating the importance of learning from a diverse collection of datasets. One interesting failure case is ScanNet [55], as it is especially difficult in a zero-shot, cross-dataset test regime because its scenes are captured while the photographer is in motion; an oracle model trained on this data learns to account for the motion blur, yielding a 13.8 point improvement over MSeg.

WildDash benchmark
The WildDash benchmark [21] specifically evaluates the robustness of semantic segmentation models. Images mainly contain  road scenes with unusual and hazardous conditions (e.g., poor weather, noise, distortion). The benchmark is intended for testing the robustness of models trained on other datasets, and does not provide a training set of its own. A small set of 70 annotated images is provided for validation. The primary mode of evaluation is a leaderboard, with a testing server and a test set with hidden annotations. The main evaluation measure is Meta Average mIoU, which combines performance metrics associated with different hazards and per-frame IoU.
We submitted results from a model trained on MSeg to the WildDash test server, with multi-scale inference. Note that Wild-Dash is not among the MSeg training sets and the submitted model has never seen WildDash images during training. The results are reported in Table 5. Our model is ranked 1st on the leaderboard for the WildDash-v1 Benchmark. Remarkably, our model outperforms methods that were trained on multiple datasets and utilized the WildDash validation set during training. In comparison to the best prior model that (like ours) did not leverage WildDash data during training, our model improves accuracy by 9.3 percentage points: a 24% relative improvement. On ScanNet, our model can accurately parse cabinets, kitchen counters, and kitchen sinks, whereas COCO cannot because COCO counter masks are not labeled accurately. Because we relabeled COCO counter masks, we are able to effectively use them for learning. In comparison, ADE20K models are also blind to counters and Mapillary-trained models completely fail in ScanNet's indoor regime.

Zero-Shot Qualitative Results
On KITTI, the ADE20K and COCO-trained models cannot recognize the sidewalk along the road edge under difficult illumination conditions, whereas the MSeg and Mapillary models can. On WildDash, our model is the only one to correctly identify trucks on both sides of a road, while the other models confuse them for buildings. On another WildDash image, our MSeg model alone can parse the road surface under snowy conditions; a Mapillary-trained model predicts the snow-covered road surface as sky, and a COCO-trained model predicts it as wall. Table 4 lists the accuracy of trained models on the MSeg training datasets. We test on the validation sets and compute IoU on a subset of classes that are jointly present in the dataset and MSeg's taxonomy. Except for Cityscapes and BDD100K, results on validation sets of all training datasets are not directly comparable to the literature since the MSeg taxonomy involves merging multiple classes. As expected, individually-trained models generally demonstrate good accuracy when tested on the same dataset: a model trained on COCO performs well on COCO, etc. Remarkably, the MSeg model matches or outperforms the respective individually-trained models across almost all datasets. On a number of training datasets, the MSeg-trained model attains higher accuracy than the individually-trained model, sometimes by a significant margin. The aggregate performance of the MSeg model is summarized by the harmonic mean across datasets. It is 76% higher than the best individually-trained baseline (COCO). Table 6 reports a controlled evaluation of two of our contributions: the unified taxonomy (Section 3.1) and compatible relabeling (Section 3.2). The 'Naive merge' baseline is a model trained on a composite dataset that uses a naively merged taxonomy in which the classes are a union of all training classes, and each test class is only mapped to an universal class if they share the same name. The 'MSeg (w/o relabeling)' baseline uses the unified MSeg taxonomy, but does not use the manually-relabeled data for split classes (Section 3.2). The model trained on the presented composite dataset ('MSeg') achieves better performance than the baselines. Using MSeg's unified taxonomy gives a large boost over the naive taxonomy, and relabeling the data further helps.    each submission must be a single, unified model. The RVC creates an aggregate rank for methods based on their rank per dataset on 7 pre-defined evaluation datasets [78]. The competition winner, SwiftNet [77], was specifically trained on the 7 RVC test datasets (see Table 7); as a result, SwiftNet suffered from no domain gap, whereas our MSeg model saw only 3 of the 7 datasets during training. For models that are trained and evaluated on the same datasets, we find MSeg outperforms SwiftNet in all cases except Mapillary Vistas as seen in Table 7. For Mapillary, relative resolution of SwiftNet's taxonomy and our taxonomy are different. We merged 19 finegrained Mapillary classes into our universal classes, causing us to receive a score of zero when formulating a mapping backwards. In order to quantify the impact of this taxonomy difference, we re-compute the mIoU by ignoring these 19 classes in Table 9. As seen from the table, when a matching taxonomy is used, our mIoU score is 5% higher than SwiftNet's. We visualize this taxonomy difference qualitatively in Figure 6 for the WildDash-v2.

Practical Semantic Segmentation for Practitioners
Our main motivation behind MSeg is developing and sharing practical multi-domain models which practitioners can use without any additional training. In order to ease this adoption, we perform two crucial steps. Firstly, we go beyond the training duration of 1 million crops and obtain better models with longer training. Specifically, we train models until 3 million crops per dataset are seen. Secondly, we go beyond 1080p resolution, and train 480p and 720p resolution models. This is especially crucial for mobile applications where computation is scarcer. In order to understand the accuracy-computation trade-off, we analyze the accuracy and computational efficiency of these models. We tabulate these results in Table 8.
One interesting observation is the superior zero-shot generalization of the 480p model in the test datasets. On the other hand, the higher resolution is helpful for performance in training datasets. Although this is promising for practitioners as they could use the low resolution models and obtain the best of both worlds, it is somewhat puzzling. Although explaining this discrepancy is vastly beyond the scope of this paper, we state a conjecture which might help explain this phenomenon. One promising perspective would be the distributional distances, as they are strongly related with the out-of-distribution generalization [79]. It is reasonable to conjecture that the distribution of different training sets gets closer in a lower-resolution setting. As a supporting thought experiment, consider classifying an image into respective datasets (similar to [26]). It would be harder to classify which dataset the image is coming from at a lower resolution.
In order to quantify the computational efficiency, we provide an analysis of the feedforward inference runtime of our semantic segmentation models in Table 11. We average runtime over 100 single-scale forward passes. Realtime single-scale inference is possible on an NVIDIA RTX 2080 Ti desktop GPU. We only perform single-scale analysis as these experiments are performed on specially optimized single-scale inference code which we share with the community 3 .

Going Beyond Semantic Segmentation
Semantic segmentation problem has no sense of an object or instance. Instance and panoptic segmentation extend this problem to introduce the concept of objects. Instance segmentation segments an image into semantically meaningful instances. However, it only considers object categories, not "stuff'. On the other hand, panoptic segmentation jointly performs instance segmentation and semantic segmentation.
We train instance and panoptic segmentation models using MSeg since the vast majority of training datasets have instance labels. However, we only evaluate these models on the MSeg training datasets since our test datasets typically do not have instance labels. During our training and evaluation, we exclude BDD and SUN RGBD because they do not provide instance labels. We use validation sets of these datasets in our evaluation. To differentiate between "thing" and "stuff" classes in the universal taxonomy, we deem the class to be a "thing" if all classes mapped to that universal class are "thing" classes in their original datasets. This step is necessary since instance segmentation detects and segments "thing" objects, while panoptic segmentation will also segment "stuff" pixels.
We evaluate instance segmentation performance by reporting box average precision and mask average precision in Table 10    train and test distributions due to data augmentation following Touvron et al. [80]. We further evaluate the resulting models in the joint instance and semantic segmentation task called panoptic segmentation. We use panoptic quality as a metric and tabulate the results in Table 12. We also present qualitative results in Figure 7. Although all resolutions perform similarly in our quantitative analysis, high resolution models perform best. Moreover, the qualitative results in Figure 7 are promising.
Although answering the question of whether MSeg-relabeling improves instance and panoptic segmentation is not directly possible due to the lack of instance labels in many datasets we use, the results presented in this section suggests that practitioners can use these models in domain-free sense.

CONCLUSION
We presented a composite dataset for multi-domain scene understanding. To construct the composite dataset, we reconciled the taxonomies of seven semantic segmentation datasets. In cases where categories needed to be split, we performed large-scale mask relabeling via the Mechanical Turk platform. We showed that the resulting composite dataset enables training a unified semantic segmentation model that delivers consistently high performance across domains. The trained model generalizes to previously unseen datasets and was ranked first on the WildDash leaderboard for robust semantic segmentation, with no exposure to WildDash data during training. Moreover, we submitted the resulting models to the Robust Vision Challange (RVC) without any additional training as an extreme generalization experiment. Our model performed competitively and ranked second. MSeg training sets only include three out of the seven datasets in the RVC, making this competitive performance surprising and promising. In order to guide practical deployment of our models, we also evaluated the role of resolution in the trade-off between computational efficiency and accuracy. Finally, we trained instance and panoptic segmentation models to go beyond semantic segmentation. We see the presented work as a step towards broader deployment of robust computer vision systems and hope that it will support future work on zero-shot generalization. Code, data, and trained models are available at https://github.com/mseg-dataset. Vladlen Koltun is the Chief Scientist for Intelligent Systems at Intel. He directs the Intelligent Systems Lab, which conducts high-impact basic research in computer vision, machine learning, robotics, and related areas. He has mentored more than 50 PhD students, postdocs, research scientists, and PhD student interns, many of whom are now successful research leaders.

APPENDIX
In this appendix, we provide additional details and experimental results.

RELABELING INTERFACE AND WORKFLOW
In Figure 8, we provide a sample Amazon Mechanical Turk interface for three tasks: cabinet-merged mask splitting, curtain mask splitting, and animal mask splitting. We distribute human intelligence tasks (HITs) in batches of 100 masks at a time to each worker.

ADDITIONAL DETAILS ON MSEG
In Figure 2 of the main text, we visualized the mapping between the meta-training set taxonomies and the MSeg taxonomy for subset of the classes. We visualize the remaining classes in Figures  9 & 10. We also list the number of split and merge operations in Table 13

ADDITIONAL QUALITATIVE COMPARISON WITH SWIFTNET
In addition to the qualitative study on WildDash-v2 in the main paper, we qualitatively compare the segmentation performance of our method with SwiftNet on ADE20K in Figure 11.

PANOPTIC CLASS MAPPING
Resolving Stuff-Thing Inconsistencies. As discussed in the main text, each of our training datasets has its own set of 'stuff'  1  nightstand  277  unlabel  7120  pool table  2  stool  1  table  8842   water-other  2532   fountain  47  unlabeled  43  playingfield  1  river lake  1232  sea  81  swimming pool  101  terrain  1  wall  20  water other  985  waterfall  15  window  6   TABLE 14: We record the number of masks from each COCO Panoptic class that are re-labeled (classes g-z, listed alphabetically. and 'thing' classes and these inconsistencies must be resolved for unified training. This occurs because the delineation between 'stuff' and 'thing' classes can be ambiguous as times.
In Table 21, we show the stuff vs. thing assignments in MSeg. As also discussed in the main text, to differentiate between "thing" and "stuff" classes in the universal taxonomy, we deem the class to be a "thing" if all classes mapped to that universal class are "thing" classes in their original datasets. If a class is considered 'stuff' in even a single dataset, this pollutes the class, and must be considered 'stuff' for all datasets.

ADDITIONAL RE-LABELING FOR DENOISING
In our qualitative analysis of the datasets, we observed abundance of erroneous annotations for some specific classes in IDD [13]. We added such an example error in Figure 12 to visualize the issue. Hence, to solve these errors, we performed additional re-labelling for the IDD for rider, bicycle, and motorcycle classes.

Dataset
Original Relabeled         -merged  table  coffee table, table  dining table, table-  Qualitative results from SwiftNet (RVC 2020 champion) and our MSeg model (RVC 2020 runner-up without any re-training on RVC datasets) on images from the ADE20K RVC test dataset (test split). Once again taxonomies greatly influence predictions; We found that Mechanical Turk workers were unable to consistently separate skyscrapers, buildings, and house instances, therefore our taxonomy merges them and our model avoids confusion between the three. The mask above is labeled as class 8 ("rider"), which we relabel to class 10 ("bicycle")