{'fraction_non_alphanumeric': 0.04074360295380388, 'fraction_numerical': 0.012794092392237678, 'mean_word_length': 4.931499872676343, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fine-Grained Dynamic Head for Object Detection

The Feature Pyramid Network (FPN) presents a remarkable approach to alleviate the scale variance in object representation by performing instance-level assignments. Nevertheless, this strategy ignores the distinct characteristics of different sub-regions in an instance. To this end, we propose a fine-grained dynamic head to conditionally select a pixel-level combination of FPN features from different scales for each instance, which further releases the ability of multi-scale feature representation. Moreover, we design a spatial gate with the new activation function to reduce computational complexity dramatically through spatially sparse convolutions. Extensive experiments demonstrate the effectiveness and efficiency of the proposed method on several state-of-the-art detection benchmarks. Code is available at https://github.com/StevenGrove/DynamicHead.


Introduction
Locating and recognizing objects is a fundamental challenge in the computer vision domain. One of the difficulties comes from the scale variance among objects. In recent years, tremendous progress has been achieved on designing architectures to alleviate the scale variance in object representation. A remarkable approach is the pyramid feature representation, which is commonly adopted by several state-of-the-art object detectors [1][2][3][4][5].
Feature Pyramid Network (FPN) [6] is one of the most classic architectures to establish pyramid network for object representation. It assigns instances to different pyramid levels according to the object sizes. The allocated objects are then handled and represented in separate pyramid levels with corresponding feature resolutions. Recently, numerous methods have been developed for better pyramid representations, including human-designed architectures (e.g., PANet [7], FPG [8]) and searched connection patterns (e.g., NAS-FPN [9], Auto-FPN [10]). The above-mentioned works lay more emphasis on instance-level coarse-grained assignments and considering each instance as a whole indivisible region. However, this strategy ignores the distinct characteristic of the finegrained sub-regions in an instance, which is demonstrated to improve the semantic representation of objects [11][12][13][14][15][16][17][18]. Moreover, the conventional head [2,3,[19][20][21][22] for FPN encodes each instance in a single resolution stage only, which could ignore the small but representative regions, e.g., the hands of the person in Fig. 1(a).
In this paper, we propose a conceptually novel method for fine-grained object representation, called fine-grained dynamic head. Different from the conventional head for FPN, whose prediction only associates with the single-scale FPN feature, the proposed method dynamically allocates pixel-level sub-regions to different resolution stages and aggregates them for the finer representation, as presented in Fig. 1(b). To be more specific, motivated by the coarse-grained dynamic routing [23][24][25][26], we design a fine-grained dynamic routing space for the head, which consists of several fine-grained dynamic routers and three kinds of paths for each router. The fine-grained dynamic router is proposed to conditionally select the appropriate sub-regions for each path by using the data-dependent spatial gates. Meanwhile, these paths are made up of a set of pre-defined networks in different FPN scales and depths, which transform the features in the selected sub-regions. Different from the coarsegrained dynamic routing methods, our routing process is performed in pixel-level and thus achieves fine-grained object representation. Moreover, we propose a new activation function for the spatial gate to reduce computational complexity dramatically through spatially sparse convolutions [27]. Specifically, with the given resource budgets, more resources could be allocated to "hard" positive samples than "easy" negative samples.
Overall, the proposed dynamic head is fundamentally different from existing methods for head designs. Our approach exploits a new dimension: dynamic routing mechanism is utilized for fine-grained object representation with efficiency. The designed method can be easily instantiated on several FPN-based object detectors [2,3,5,20,21] for better performance. Moreover, extensive ablation studies have been conducted to elaborate on its superiority in both effectiveness and efficiency, which achieve consistent improvements with little computational overhead. For instance, with the proposed dynamic head, the FCOS [3] based on the ResNet-50 [28] backbone attains 2.3% mAP absolute gains with less computational cost on the COCO [29] dataset.

Method
In this section, we first propose a routing space for the fine-grained dynamic head. And then, the fine-grained dynamic routing process is elaborated. At last, we present the optimization process with resource budget constraints.

Fine-Grained Dynamic Routing Space
FPN-based detection networks [2,3,20,21] first extract multiple features from different levels through a backbone feature extractor. And then, semantically strong features in low-resolution and semantically weak features in high-resolution are combined through the top-bottom pathway and lateral connections. After obtaining features from the pyramid network, a head is adopted for each FPN scale, which consists of several convolution blocks and shares parameters across different scales. The head further refines the semantic and localization features for classification and box regression.
To release pixel-level feature representation ability, we propose a new fine-grained dynamic head to replace the original one, which is shown in Fig. 2. Specifically, for n-th stage, a space with D depths and three adjacent scales is designed for a fine-grained dynamic head, namely fine-grained dynamic routing space. In this routing space, the scaling factor between adjacent stages is restricted to 2. The ... ...   Figure 2: The diagram of the routing space for the fine-grained dynamic head. 'Stage n-1', 'Stage n' and 'Stage n+1' represent three adjacent FPN scales, respectively. The dynamic head (marked by the dashed rectangle) for each stage includes D fine-grained dynamic routers, where each router has up to three alternative paths. The router first aggregates the multiple input features by performing element-wise accumulation. And then, for each pixel-level location, the router dynamically selects subsequent paths, which is elaborated in 'Routing Process'. basic unit, called fine-grained dynamic router, is utilized to select subsequent paths for each pixel conditionally. In order to take advantages of distinct properties of the different scale features, we propose three kinds of paths with different scales for each router, which are elaborated in Sec. 2.2.3. Following common protocols [2,3,20,21], we adopt the ResNet [28] as the backbone. The feature pyramid network is configured like the original design [6] with simple up-sampling operations, which outputs augmented multi-scale features, i.e., {P3, P4, P5, P6, P7}. The channel number of each feature is fixed to 256.

Fine-Grained Dynamic Routing Process
Given the routing space with several individual nodes, we propose fine-grained dynamic routers for each router to aggregate multi-scale features by performing element-wise accumulation and choose pixel-level routing paths. This routing process of a dynamic router is briefly illustrated in Fig. 2.

Fine-Grained Dynamic Router
Given a node l, the accumulation of multiple input features is denoted as pixel-level locations and C channels. We define a set of paths according to the adjacent FPN scales, which is denoted as F = {f l k (·)|k ∈ {1, ..., K}}. To dynamically control the on-off state of each path, we set up a fine-grained dynamic router with K spatial gates, where the output of the spatial gate is defined as gating factor. The gating factor of the k-th spatial gate in i-th (i ∈ {1, ..., N }) location can be represented by where θ l k denotes the parameters of the partial network corresponding to the k-th spatial gate and shares parameters in each location. The conventional evolution or reinforcement learning based methods [4,9,10] adopt metaheuristic optimization algorithm or policy gradient to update the agent for discrete path selection, i.e., m k,l i ∈ {0, 1}. Different from those methods, motivated by [30], we relax the discrete space of the gating factor to a continuous one, i.e., m k,l i ∈ [0, 1]. Since the gating factor m k,l i can reflect the estimated probability of the path being enabled, we define the list of weighted output from each path as the router output. This router output is formulated as Eq. 2, where only paths with a positive gating factor are enabled. It is worth noting that, unlike many related methods [23-25, 31, 32], our router allows multiple paths to be enabled simultaneously for a single location. and 'GN' indicate the spatially sparse convolution [27] and the group normalization [35], respectively. Conditional on the input feature, the spatial gate dynamically enables pixel-level locations to be inferred by using spatially sparse convolutions, which is elaborated in the dashed rectangle. The 'Max Pooling', whose stride is one, is used to generate the spatial mask for 'SPConv' according to the size of the receptive field.
As shown in Fig. 3(c), we design a lightweight convolutional network to instantiate the spatial gate g l k (·). The input feature is embedded into one channel by using a layer of convolution. Since the output of spatial gate m k,l i is restricted to the range [0, 1], we propose a new gate activation function, denoted as δ(·), to perform the mapping.

Gate Activation Function
To achieve a good trade-off between effectiveness and efficiency, the dynamic router should allocate appropriate computational resources to each location by disabling paths with high complexity. Specifically, for the i-th location of the node l, the k-th path is disabled, when the gating factor m k,l i output by the gate activation function is zero. Therefore, the range of the gate activation function needs to contain zero. Meanwhile, in order to make the routing process learnable, the gate activation function needs to be differentiable.
For instance, [23,30,33] adopts a soft differentiable function (e.g., softmax) for training. In the inference phase, the path whose gating factor below a given threshold is disabled. On the contrary, [34] uses the hard non-differentiable function (e.g., hard sigmoid) in the forward process and the approximate smooth variant in the backward process. However, these strategies cause inconsistency between training and inference, resulting in performance degradation. Recently, a restricted tanh function [26], i.e., max(0, tanh(·)), is proposed to bridge this gap. When the input is negative, the output of this function is always 0, which allows no additional threshold to be required in the inference phase. When the input is positive, this function turns into a differentiable logistic regression function, so that no approximate variant is needed in the backward process.
However, the restricted tanh has a discontinuous singularity at zero, causing the gradient to change dramatically at that point. To alleviate this problem, we propose a more generic variant of the restricted tanh, which is formulated as Eq. 3 (the visualization is provided in the supplementary material).
Where τ is a hyperparameter to control the gradient at 0 + . In particular, Eq. 3 can be degraded to the original restricted tanh when τ = 0. When τ is set to a positive value, the gradient at 0 + will decrease to alleviate the discontinuity problem.

Routing Path
As presented in Fig 2, we provide three paths for each fine-grained dynamic router. The same network architecture except for the bilinear operations is adopted in 'Scale (up) Path' and 'Scale (down) Path', which is illustrated in Fig. 3(b). The mode of the bilinear operators is switched to up-sampling and down-sampling for 'Scale (up) Path' and 'Scale (down) Path' respectively, whose scaling factor is set to 2. The depth of head is critical to the performance, especially for one-stage detectors [2,3,20,21].
To improve the effectiveness of deep network for the head, we use a bottleneck module [28] with a residual connection for the 'depth' path, which is illustrated in Fig. 3(a). To release the efficiency of fine-grained dynamic routing, we adopt the spatially sparse convolutions [27] for each path, which is elaborated in Fig. 3(c). Specifically, the spatial mask M k,l is generated by performing the max-pooling operation on the gating factors m k,l , which is elaborated in Sec. 2.3. The positive values in the spatial mask are further quantified to one. To this end, the quantified spatial mask guides the spatially sparse convolution to only infer the locations with positive mask values, which reduces the computational complexity dramatically. All the spatially sparse convolutions take the kernel size of 3 × 3 and the same number of output channels as the input. In particular, to further reduce the computational cost, the convolutions in 'Scale' paths adopt depthwise convolutions [36]. Moreover, we use additional group normalizations [35] to alleviate the adverse effect of gating variations.

Resource Budget
Due to the limited computational resources for empirical usage, we encourage each dynamic router to disable as many paths as possible with a minor performance penalty. To achieve this, we introduce the budget constraint for efficient dynamic routing. Let C k,l denotes the computational complexity associated with the predefined k-th path in the node l.
The resource budget B l of the node l is formulated as Eq. 4, where Ω k,l i denotes the receptive field of i-th output location in the k-th path. As shown in Fig. 3(a), the gating factor m k,l only reflects the enabled output locations in the last layer. Since the receptive field of the stacked convolution network tends to increase with depth, more locations in the front layer need to be calculated than the last layer. For simplicity, we consider all the locations involved in the receptive field of locations with positive gating factors as enabled, which are calculated by a max-pooling layer, i.e., max j∈Ω k,l i (m k,l j ). Furthermore, this function can provide a regularization for the number of connected components. It encourages the connected components of the gating map m k,l to have a larger area and smaller number. This regularization could improve the continuity of memory access and reduces empirical latency [37].

Training Target
In the training phase, we select the appropriate loss functions according to the original configuration of the detection framework. Specifically, we denote the losses for classification and regression as L cls and L reg , respectively. For the FCOS [3] framework, we further calculate the loss for the centerness of bounding boxes, which is represented as L center . Moreover, we adopt the above-mentioned budget B l to constrain the computational cost. The corresponding loss function is shown as Eq. 5, which is normalized by the overall computational complexity of the head.
To achieve a good tradeoff between efficiency and effectiveness, we introduce a positive hyperparameter λ to control the expected resource consumption. Overall, the network parameters, as well as the fine-grained dynamic routers, can be optimized with a joint loss function L in a unified framework. The formulation is elaborated in Eq. 6. L = L cls + L reg + L center + λL budget .

Experiment
In this section, we first introduce the implementation details of the proposed fine-grained dynamic routing network. Then we conduct extensive ablation studies and visualizations on the COCO dataset [29] for the object detection task to reveal the effect of each component.

Implementation Detail
To verify the effectiveness of our proposed method, we conduct experiments on the FCOS [3] framework unless otherwise specified. The FCOS provides a simple architecture that can be used to reveal the properties of our method. In the default configuration of FCOS, it adopts a pair of 4convolution heads for classification and regression, respectively. Besides, the prediction of centerness shares the same head with the regression branch. For a fair comparison, we replace the original head with a fixed network. It has the same paths as our proposed dynamic head but without dynamic routers. Moreover, {Dn|n ∈ {2, 4, 6, 8}} represents the depth of the equipped head.
All the backbones are pre-trained on the ImageNet classification dataset [38]. Batch normalizations [39] in the backbone are frozen. In the training phase, input images are resized so that the shorter side is 800 pixels. All the training hyper-parameters are identical to the 1x schedule in the Detectron2 [40] framework. Specifically, we fix parameters of the first two stages in the backbone and then jointly finetune the rest network. All the experiments are trained on 8 GPUs with 2 images per GPU (effective mini-batch size of 16) for 90K iterations. The learning rate is initially set to 0.01 and then decreased by 10 at the 60K and 80K iterations. All the models are optimized by using Synchronized SGD [41] with a weight decay of 0.0001 and a momentum of 0.9.

Ablation Study
In this section, we conduct experiments on various design choices for our proposed network. For simplicity, all the reported results here are based on ResNet-50 [28] backbone and evaluated on COCO val set. The FLOPs is calculated when given a 1333 × 800 input image.

Dynamic vs Static
To demonstrate the effectiveness of the dynamic routing strategy, we give the comparison with the fixed architecture in Tab. 1. For fair comparisons, we align the computational complexity with these models by adjusting the coefficient λ for the budget constraints in Eq. 6. The results show that the dynamic strategy can not only reduce computational overhead but also improve performance by a large margin. For instance, our method obtains 2.3% absolute gains over the static baseline with lower average computational complexity. As shown in Fig. 4, this phenomenon may be attributed to the adaptive sparse combination of the multi-scale features to process each instance.

Fine-Grained vs Coarse-Grained
Different from most of the coarse-grained dynamic networks [23][24][25][26]31], our method performs routing in the pixel level. For comparison, we construct a coarse-grained dynamic network by inserting a global average pooling [42] operator between the 'Conv3 × 3' and the gate activation function in each spatial gate, as shown in Fig. 3 (c). The experiment results are provided in Tab. 1. We find that the upper bound of fine-grained dynamic routing is higher than that of coarse-grained dynamic routing in the same network architecture. Moreover, as the computational budget decreased, the performance of coarse-grained dynamic routing decreases dramatically, which reflects that most of the computational redundancy is in space. Specifically, the fine-grained dynamic head achieves 2.4% mAP absolute gains over the coarse-grained one with only 87% computational complexity.

Component Analysis
To reveal the properties of the proposed activation function for spatial gates, we further compare some widely-used activation function for soft routing, which is elaborated in Tab. 2. When adopting the Softmax as the activation function, the routing process is similar to the attention mechanism [43][44][45][46][47]. This means that the hard suppression of the background region is important for the detection task. Meanwhile, as shown in Tab. 3, we verify the effectiveness of the proposed 'Depth' path and 'Scale' path with ablation, respectively. The performance can be further improved by using both paths, which demonstrates that they are complementary and promoting each other.

Trade-off between Efficiency and Effectiveness
To achieve a good balance between efficiency and effectiveness, we give a comparison of varying the coefficient λ of budget constraints and the depth of equipped head, which is shown in Fig. 5. The baseline FCOS is scaled by the depth of the head. The redundancy in space enables the network to maintain high performance with little computational cost. For instance, when λ is set to 0.4, the   proposed network achieves similar performance with the fixed FCOS-D6 network, but only account for about 43% of the computational cost (including the backbone). In particular, without considering the backbone, its computational cost only accounts for about 19% of the FCOS-D6.

Application on State-of-the-art Detector
To further demonstrate the superiority of our proposed method, we replace the head of several state-of-the-art detectors with the proposed fine-grained dynamic head. All the detectors except 'EfficientDet-D1' [5] are trained with 1x schedule (i.e., 90k iterations), and batch normalizations [39] are freezed in the backbone. Following the original configuration, the 'EfficientDet-D1' adopts 282k iterations for training. In addition, all the detectors adopt single-scale training and single-scale testing.
The experiment results are presented in Tab. 4. Our method consistently outperforms the baseline with less computational consumption. For instance, when replacing the head of RetinaNet [2] with our fine-grained dynamic head, it obtains 2.4% absolute gains over the baseline.

Conclusion
In this paper, we propose a fine-grained dynamic head to conditionally select features from different FPN scales for each sub-region of an instance. Moreover, we design a spatial gate with the new activation function to reduce the computational complexity by using spatially sparse convolutions.
With the above improvements, the proposed fine-grained dynamic head can better utilize the multi-scale features of FPN with a lower computational cost. Extensive experiments demonstrate the effectiveness and efficiency of our method on several state-of-the-art detectors. Overall, our method exploits a new dimension for object detection, i.e., dynamic routing mechanism is utilized for finegrained object representation with efficiency. We hope that this dimension can provide insights into future works, and beyond.

Broader Impact
Object detection is a fundamental task in the computer vision domain, which has already been applied to a wide range of practical applications. For instance, face recognition, robotics and autonomous driving heavily rely on object detection. Our method provides a new dimension for object detection by utilizing the fine-grained dynamic routing mechanism to improve performance and maintain low computational cost. Compared with hand-crafted or searched methods, ours does not need much time for manual design or machine search. Besides, the design philosophy of our fine-grained dynamic head could be further extended to many other computer vision tasks, e.g., segmentation and video analysis. GenricRestrideTanH(x, τ) x τ=0 τ=0.5 τ=1.0 τ=1.5 τ=2.0 Figure 6: Visualization of the proposed gate activation function. As the coefficient τ increased, the gradient at 0+ will decrease to alleviate the discontinuity problem.