{'fraction_non_alphanumeric': 0.030829276849537342, 'fraction_numerical': 0.012673770995044511, 'mean_word_length': 5.113672922252011, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Gated Path Selection Network for Semantic Segmentation

Semantic segmentation is a challenging task that needs to handle large scale variations, deformations and different viewpoints. In this paper, we develop a novel network named Gated Path Selection Network (GPSNet), which aims to learn adaptive receptive fields. In GPSNet, we first design a two-dimensional multi-scale network - SuperNet, which densely incorporates features from growing receptive fields. To dynamically select desirable semantic context, a gate prediction module is further introduced. In contrast to previous works that focus on optimizing sample positions on the regular grids, GPSNet can adaptively capture free form dense semantic contexts. The derived adaptive receptive fields are data-dependent, and are flexible that can model different object geometric transformations. On two representative semantic segmentation datasets, i.e., Cityscapes, and ADE20K, we show that the proposed approach consistently outperforms previous methods and achieves competitive performance without bells and whistles.


Introduction
Semantic segmentation refers to the problem of assigning semantic object category for each pixel. Recent stateof-the-art semantic segmentation approaches [17,2,3,4,6,33,28] are typically based on the Fully Convolutional Networks (FCNs) [17]. It benefits from the informative representations of object categories and semantic information learned by Convolutional Neural Networks (CNNs) [10,13]. CNNs are inherently limited by the design of structure, where the receptive field is restricted to constant regions [34]. Objects in semantic segmentation are in a large range of scales, deformations and different viewpoints, and the fixed field of view in CNNs is insufficient to deal with geometric variations.
Extensive efforts have been made to enlarge receptive fields to better understand the semantic scenes. Atrous convolution [2] incorporates larger context by dilating the con-volution kernel in a fixed manner. However, it lacks the ability to cope with multi-scale objects. To mitigate the problem, PSPNet [33] applies pyramid pooling module to aggregate information from different scales of feature maps. ASPP [3] and DenseASPP [28] are introduced to use a series of atrous convolution layers to learn features with multiple dilation rates. Nevertheless, those approaches ignore the same problem: different regions may correspond to objects with different scales and geometric variations, Therefore, receptive fields on regular grids are insufficient to tackle large appearance variations.
To capture rich semantic context, attention-based approaches [34,31,30,9] are proposed to adaptively aggregate short-and long-range features. To spotlight locally discriminative information, the recent work Deformable Convolutional Network (DCN) [8] shows that the adaptive sample positions can be acquired by predicting additional offsets. However, the learned receptive field is limited by the size of pre-defined convolution kernel which is difficult to capture long-range context. Moreover, DCN sparsely samples a fixed number of locations rather than considering all relevant locations within the contextual scope.
In this work, we introduce Gated Path Selection Network (GPSNet) to enhance the capability of CNNs to adaptively learn free form receptive fields for semantic segmentation. Different from previous approaches which focus on optimizing sample positions on the regular grids, GP-SNet densely captures semantic contexts which are adaptively weighted. The learned adaptive receptive fields are data-dependent, and are flexible to model various geometric deformations. Figure 1 gives an overview of the proposed method.
Specifically, to construct a set of message passing paths with various receptive fields and sample rates [28], we carefully design a two-dimentional multi-scale network architecture with multiple branches. We refer to the network as SuperNet. It stacks a series of bottlenecked branches which consist of different tuned dilation convolutions. To progressively produce multi-scale features, inspired by [12,28], dense connections are applied to feed the input from relative small receptive fields to the large receptive fields. Moreover, in order to aggregate information from different paths, a gate prediction module is introduced to predict soft masks to combine the semantic contexts. The gates not only enable dynamic selection of the effective pixel positions but also help regularize the receptive fields. Hence, different scales of objects can obtain desirable representations. Moreover, another notable advantage of GPSNet is its improved sample rates, which ensures that both the interior and exterior regions can be densely sampled to capture adequate semantic context.
The GPS module is light-weight as it adds a small amount of parameters and computation for multi-scale feature extraction. Meanwhile, it is model-agnostic which can be readily used in various ASPP-like structure and trained in an end-to-end manner.
We conduct experiments on two competitive semantic segmentation datasets, i.e., Cityscapes [7], and ADE20K [36]. The experiments demonstrate that our proposed GPSNet consistently improves the performance of previous state-of-the-art approaches.

Related Work
Semantic segmentation. Semantic segmentation is a fundamental problem in computer vision, which involves assigning a semantic category to each pixel. Recent progress in this problem has been largely driven by deep fully convolutional neural networks (FCNs) [17,35,29,3,33]. The pioneering work FCN [17] proposed to remove the fully connected layers in classification CNN networks [15,23], leading to a fully convolutional architecture for dense semantic segmentation. To facilitate dense prediction, the deconvolutional layer was also introduced in [17,19], which is a learnable upsampling operation. The following work Seg-Net [1] introduced an encoder and decoder network, where the decoder utilizes pooling indices in the encoding layers to upsample the feature map. UNet [21] adopted skip connections to combine shallow representations from the encoder and deep features from the encoder, which exploit low level feature for accurate semantic segmentation. To refine segment contours, CRF was applied as a post-processing procedure [3] or end-to-end integrated [35] into the network. Further, to develop real-time semantic segmentation networks for practical applications, Paszke et al. proposed a light weight neural network -ENet [20] by exploiting separable convolution and less channels. Zhao et al. [32] proposed ICNet to utilize image pyramid to optimize the network.
Contextual information modeling in semantic segmentation. Recent work has shown that contextual information is important for improving semantic segmentation accuracy. Deeplab [3] and Dilated Conv [29] proposed atrous convolutions to enlarge the network receptive field without sacrificing the resolution, which enables the network to harvest contextual information in a larger region for semantic segmentation. Moreover, an atrous spatial pyramid pooling (ASPP) module was developed to incorporate contextual information from multiple scales. ParseNet [16] proposed global average pooling layer which introduces global contextual information for semantic segmentation. Later on, Zhao et al. [33] proposed a Pyramid Pooling Module to aggregate contextual information from multi-scale regions. However, the above approaches can only aggregate contextual information from predefined or fixed regions, which still be limited for modeling large contextual variations in objects.
Deformable convolution or attention mechanism in semantic segmentation. Deformable and attention based mechanisms enable the network to dynamically select of the context for each pixel, which is adaptive to different pixels. Chen et al. [5] proposed to learn combination weights for combining multi-scale features. Deformable convolution layer is introduced in [37], which makes the convolution kernel adaptive to geometric variations of the object, extracting dynamic contextual information for image recognition. Wang et al. [26] proposed non-local operation that computes a weighted sum of features in the global map based on a attention mechanism. Further, Point-wise Spatial Attention Network [34] learns to aggregate information through a learned attention map dynamically adjusting contextual information aggregation. OCNet [30] proposed an object context network to learn an object context map by modeling pixel-pixel similarities which are further utilized to refine the representations of each pixel. CC-Net [14] harvested the contextual information on the crisscross path which can provide long-range contextual information to each pixel with improved efficiency.

Gated Path Selection Network
In this section, we present Gated Path Selection Network (GPSNet) for semantic segmentation in detail. GP-SNet estimates gate masks to weight contexts which are derived from various receptive fields (RFs) and sample rates (SRs). Specifically, in GPSNet we first carefully design a SuperNet with atrous convolution layers where the dilation rates varying in large range. Then a gate prediction module is introduced to determine the importance of samples from different feature maps and dynamically aggregate them.

SuperNet
ASPP. Atrous convolution is introduced to replace compact convolution in dense prediction. It is used to increase receptive field while maintaining the resolution of feature maps. However, atrous convolution fails to capture multiscale semantic context. ASPP is proposed to concatenate feature maps in multiple parallel atrous convolution layers with different dilation rates. One advantage of ASPP is to sample features at different scales to boost the performance. However, as the dilation rate increases, the atrous convolution layer gradually loses the power of modeling and fails to capture information effectively [4].
SuperNet. To further enhance the capability to learn effective feature representations, we propose an improved ASPP-like network structure with multiple entrances and exits to propagate information among atrous convolution layers. We modify the original ASPP structure in the following three aspects. Tuned Dilation. We first extend the ASPP atrous convolution layers to grid form. We double the atrous convolution layers with dilation rates {r 1 , r 2 , r 3 , r 4 } into {(r 1 , r 1 ), (r 2 , r 2 ), (r 3 , r 3 ), (r 4 , r 4 )}. To mitigate repeatedly sampling, the dilation rates are further tuned to improve the sample rate. As illustrated in Figure 1 Bottlenecked Branch. To alleviate the GPU memory usage and computing resource overhead, inspired by eA-SPP [24], we introduce bottlenecked branches in SuperNet to address the problem. Following [11], each of the branch starts with a Squeeze operation which is to reduce the channel of input features with a 1 × 1 convolution. Then two consecutive 3 × 3 atrous convolutions are used to extract features with different sample rates. Finally, an Excitation operation is applied at the exit of each branch, the features are produced to large channel features with a 1 × 1 convolution. All the convolutions are followed by InplaceABNsync [22].
Dense Connectivity. To facilitate information flow between different atrous convolution layers, we use dense connectivity to bridge parallel bottlenecked branches in Super-Net. The intermediate feature maps are aggregated by the results of one or two directions: 1) the output of a convolution from previous layer in the same branch (horizontal connection) and, if possible, 2) the result of a convolution from the previous branch (vertical connection). Consequently, the subsequent layers gather the information from early lay-ers with relative small receptive filed layers. In comparison with ASPP and DenseASPP, because of the dense connectivity pattern, we can acquire abundant features with more diverse and denser context.

Gate Prediction Module
In this subsection, we describe the gate prediction module in detail. The module is used to dynamically aggregate information in the SuperNet. In existing work, different features are merged through directly concatenation and summation. Actually, objects are in complex geometric transformations, augment features via simple concatenation or summation may be infeasible in dense prediction task. Therefore, we propose a gate prediction module to composite different scales of features. Figure 1 (b) depicts the process of the gate prediction module. It contains three operators: Projection, Comparison and Weighted Sum.
Projection. Each gate considers two input feature maps X v , X h with the shape of H × W × C, where X v is feature maps from the previous vertical layer and X h is the horizontal input features. The soft gate masks M v , M h with the size of H × W × 1 are predicted via a projection transformation F, where The transformation layer is defined with three consecutive operations: a 1 × 1 convolution, followed by a batch normalization (BN) and a rectified linear unit (ReLU).
Comparison. To adaptively adjust the pixel-wise receptive field and sample rates, the soft mask is used to integrate information from different scales of features. We first get the concatenation mask M c = [M v , M h ] and a comparison function C a is applied to get the soft mask M ∈ R H×W ×2 : The comparison function C a in our experiments consists of a 1 × 1 convolution layer, followed by a BN layer and a ReLU layer.
Weighted Sum. To aggregate features, we first split M into {M v , M h } along the channel dimension, and reweight the input feature {X v , X h } by element-wise multiplication. The adaptive features O ∈ R H×W ×C are obtained by summing the reweighted features.

Relation to other approaches
In this section, we compare our GPSNet with the most relevant approaches including ASPP, Dense Atrous Spatial Pyramid Pooling (DenseASPP), Deformable Convolution Network (DCN).  ASPP. ASPP [3] adopts atrous convolution layers to segment both small and large objects. It employs multiple parallel filters with different rates to exploit multi-scale features. The features extracted from each sample rate are further concatenated to produce the final result. It is different from GPSNet which applies soft gates to dynamically reweight the feature maps from separate branches. Moreover, by extending the parallel atrous convolution layers to grid form, GPSNet expands the receptive fields to large variations. As illustrated in Figure 2, the sample positions in GPSNet are linearly increased when more branches are added.
DenseASPP. In order to achieve large enough receptive field size, DenseASPP introduces a base cascade network which consists of atrous convolution layers. The final results are obtained for an input visiting from small receptive field to large receptive field sequentially. Differently, GP-SNet introduced SuperNet with multiple entrances and multiple exits, which is more flexible to get different scales of features. Specifically, we can feed the input in any entrances and reject it from different exits. Through gated modules, our method can generate adaptive receptive fields to tackle objects with large geometric deformations.

DCN.
To make the convolution kernel adaptive to geometric variations of the object, DCN predicts 2D offsets to augment spatial sampling positions. In contrast, GPSNet predicts soft gates to select the semantic context for objects with different shapes. It can get large receptive field, and enable densely position samplings in the effective semantic context.

Experimental Results
To demonstrate the effectiveness of our approach, we conducted extensive experiments on two representative semantic segmentation datasets, i.e., Cityscapes [7], ADE20K [36]. In addition, complete ablation studies are performed to analyze the components of GPSNet. To integrate global-aware information, we incorporate object context (OC) module [30] in our network. All of our experiments are conducted in PyTorch. The code to reproduce our results will be made public upon the acceptance of the paper.

Evaluation on Cityscapes
Dataset. Cityscapes is the dataset to understand urban scenes. It contains 30 common classes including road, person, car, etc. and only 19 of them are used for semantic segmentation evaluation. The dataset is comprised of 5,000 finely annotated images and 20,000 coarsely annotated images. The finely annotated 5,000 images are divided into 2,975, 500 and 1,525 images for training, validation, and testing.
Training Details. On the Cityscapes dataset, we train all models with the 2,975 finely annotated images. We set the mini-batch size as 8 with InplaceABNSync [22] to synchronize the mean and standard variation. The initial learning rate is set as 0.01 and weight decay as 0.0005. Following PSPNet [33], the original image is randomly cropped to produce 769 × 769 input. We train models with 40K iterations with 4×P40 GPUs. Following previous work [33], we employ the 'poly' learning rate policy, where the power is set to 0.9. We augment the dataset by scaling it with a factor in the rage of [0.5, 2], horizontally flipping.
Ablation Study. To investigate the effectiveness of the individual components of the proposed approach, i.e., Super-Net, Gate Module, Tuned Dilation and online hard example mining(OHEM) [27]. We integrate the components into two representative approaches, i.e., ASPP, OCNet. The ablation analysis is conducted on the Cityscapes validation set. Quantitative results are shown in Table 1. In Table 1, we show the mIoU 79.58 reported by OCNet, which is reproduced to be 78.70 in our experiment. All the components consistently improve the performance of both approaches.
• SuperNet. To further evaluate the effectiveness of Su-perNet, we compare the different OCNet trained with SuperNet and with the baseline OCNet. Validation accuracy in both settings is shown in Table 1. With Su-perNet, the validation accuracy is higher than the baseline model. This demonstrates SuperNet with different scales of receptive fields can help improve performance.
• Gate Prediction Module is used to adaptively select receptive fields. It can improve the performance as shown in Table 1 trained with OCNet. The SuperNet typically benefit from the gate module. The larger receptive field layers acquire information from the previous layers with relative small receptive fields. The gate module is able to not only control the size of the field of view but also increase the sample rate within the effective receptive field. This indicates that adaptive receptive fields are of importance in dense object prediction to deal with object transformations.
• Tuned Dilation. Quantitative improvements with the tuned dilation as shown in Table 1. By integrating the carefully tuned atrous convolution layers, the network tends to a gain higher sample rate which enables to densely capture the semantic context. The result further shows that tuned dilation is indeed effective for increasing sample rate to improve the performance.
• OHEM. To tackle with data imbalance and overfitting, we further conduct experiments to compare the models with OHEM and baseline in Table 1. following previous work [27], we set the threshold for selecting hard pixels as 0.7, and keep at least 10,0000 pixels within each mini-batch. The result shows that the OHEM built on our network can further boost the performance.

Evaluation on ADE20K
Dataset. The scene parsing dataset ADE20K contains 150 classes and diverse complex scenes with 1,038 imagelevel categories. It needs to parse both objects and stuff. The dataset is divided into 20,000, 2,000 and 3,000 for training, validation and testing. Results are evaluated with both mean of class-wise Intersection over Union (Mean IoU).
Training Details. On the ADE20K dataset, the base learning rate is set as 0.02 and with a weight decay 0.0001. The input image is resize to 480. The mini-batch size is 16 and we also apply InplaceABNSync to synchronize the mean and standard deviation across multiple GPUs. The models are trained with 200K iterations with 4×P40 GPUs. The learning rate policy and data augmentation are the same as the Cityscapes dataset.

Model Analysis
In this section, we analyze our approach in three aspects, i.e., receptive field, sample rate and parameter efficiency.

Method
BaseNet Mean IoU (%) PSPNet [33] ResNet269  The statistics of the size of receptive field, sample rate and the number of parameters of different methods are summarized in Table 4.
Parameter Efficiency. The results in Table 4 indicate that SuperNet utilizes parameters more effectively than alternative methods, i.e., ASPP and DenseASPP. This suggests that GPSNet is simple but effective to extract local context handle object with geometric variations

Adaptive Receptive Fields Visualization
As shown in Figure 4, we choose pixels from rider, road, car, pole and bus to visualize the path selection mechanism. The masks are relative with category, scales and appearance variance frequency. There are several conclusions as follows.
• For movable objects such as car and rider, the main features are captured with branch 1, in which the RF is the smallest.
• For textureless regions such as road, the features are captured with branch 4, in which the RF is the largest.
• To aggregate boundary features, the features from the convolution layer with dilation rate r 11 are estimated with large weights.
• For large objects like bus, the features from branch 4 are given larger weight than small objects.
• For tiny objects like pole, the local features from branch 1 are much more important than the features from branch 4.

Conclusion
In this paper, we have presented Gated Path Selection Network (GPSNet) to learn adaptive receptive fields and increase sample rates in semantic segmentation. A SuperNet is proposed to ensure maximum information flow in the network. It provides various paths to extract multi-scale representations. Dense connectivity in the network allows to feed the input from small receptive field to large receptive field. A gate prediction module is further introduced to estimate soft masks to dynamically select effective context positions and regularize the receptive fields. Besides, our method is simple, efficient and model-agnostic. It can be applied to various ASPP-like architectures. The proposed method has shown its effectiveness on two competitive semantic segmentation datasets, i.e., Cityscapes, ADE20K, and achieves new state-of-the-art results. Future research may focus on extending our results to other types of computer vision tasks, such as object detection and image generation.