{'fraction_non_alphanumeric': 0.04549140608002532, 'fraction_numerical': 0.01764275401016634, 'mean_word_length': 4.251895709982342, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 6, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
From Open Set to Closed Set: Supervised Spatial Divide-and-Conquer for Object Counting

Visual counting, a task that aims to estimate the number of objects from an image/video, is an open-set problem by nature, i.e., the number of population can vary in [0, inf) in theory. However, collected data and labeled instances are limited in reality, which means that only a small closed set is observed. Existing methods typically model this task in a regression manner, while they are prone to suffer from an unseen scene with counts out of the scope of the closed set. In fact, counting has an interesting and exclusive property---spatially decomposable. A dense region can always be divided until sub-region counts are within the previously observed closed set. We therefore introduce the idea of spatial divide-and-conquer (S-DC) that transforms open-set counting into a closed-set problem. This idea is implemented by a novel Supervised Spatial Divide-and-Conquer Network (SS-DCNet). Thus, SS-DCNet can only learn from a closed set but generalize well to open-set scenarios via S-DC. SS-DCNet is also efficient. To avoid repeatedly computing sub-region convolutional features, S-DC is executed on the feature map instead of on the input image. We provide theoretical analyses as well as a controlled experiment on toy data, demonstrating why closed-set modeling makes sense. Extensive experiments show that SS-DCNet achieves the state-of-the-art performance. Code and models are available at: https://tinyurl.com/SS-DCNet.


INTRODUCTION
Counting is an open-set problem by nature as a count value can range from 0 to +∞ in theory. It is therefore typically modeled in a regression manner. Benefiting from the success of convolutional neural networks (CNNs), stateof-the-art deep counting networks often adopt a multi-branch architecture to enhance the feature robustness to dense regions [2], [4], [53]. However, the observed patterns in datasets are limited in practice, which means that networks can only learn from a closed set. Are these counting networks still able to generate accurate predictions when the number of objects is out of the scope of the closed set? According to Figure 2, local counts observed in the closed set exhibit a long-tailed distribution. Extremely dense patches are rare while sparse patches take up the majority. As what can be observed, increased local density leads to significantly increased relative mean absolute error (rMAE). Is it necessary to set the working range of CNN-based counter to the maximum count value observed, even with a majority of samples are sparse such that the counter works poorly in this range?
In fact, counting has an interesting and exclusive property-spatially decomposable. The above problem can  be largely alleviated with the idea of spatial divide-andconquer (S-DC). Suppose that a network has been trained to accurately predict a closed set of counts, say 0 ∼ 20. When facing an image with extremely dense objects, one can keep dividing the image into sub-images until all subregion counts are less than 20. The network can then count these sub-images and sum over all local counts to obtain the global image count. Figure 1 graphically depicts the idea of S-DC. A follow-up question is how to spatially divide the count. A naive way is to upsample the input image, divide it into sub-images and process sub-images with the same network. This approach, however, is likely to blur the image and lead to significantly increased computation cost and memory consumption. Inspired by fully convolutional  [53]. The orange curve denotes the relative mean absolute error (rMAE) of CSRNet [21] on local patches.
networks and RoI pooling [12], we show that it is feasible to achieve S-DC on feature maps, as conceptually illustrated in Figure 3. By decoding and upsampling the feature map, the later prediction layers can focus on the feature of local areas and predict sub-region counts accordingly.
To implement the above idea, we propose a simple yet effective Supervised Spatial Divide-and-Conquer Network (SS-DCNet). SS-DCNet learns from a closed set of count values but is able to generalize to open-set scenarios. Specifically, SS-DCNet adopts a VGG16 [39]-based encoder and an UNet [35]-like decoder to generate multi-resolution feature maps. All feature maps share the same counter. The counter can be designed by following the standard local count regression paradigm [25] or by discretizing continuous count values into a set of intervals as a classifier following [19], [23]. Furthermore, a division decision module is designed to decide which sub-region should be divided and to merge different levels of sub-region counts into the global image count.
We provide theoretical analyses to shed light on why the transition from the open set to the closed set makes sense for counting. We also show through a controlled toy experiment that, even given a closed training set, SS-DCNet effectively generalizes to the open test set. The effectiveness of SS-DCNet is further demonstrated on three crowd counting datasets: (ShanghaiTech [53], UCF_CC_50 [14] and UCF-QNRF [15]), a vehicle counting dataset (TRANCOS [13]), and a plant counting dataset (MTC [25]). Results show that SS-DCNet indicates a clear advantage over other competitors and sets the new state of the art. In addition, we remark that the closed set of SS-DCNet executes an implicit transfer in the output space, which is backed by state-of-the-art performance under the cross-domain evaluations. In particular, SS-DCNet even beats most state-of-the-art counting models that are trained directly on the target domain in the task from UCF-QNRF to ShanghaiTech Part_A.
Overall, the main contributions of this work are as follows.
• We propose to transform open-set counting into a closedset problem via S-DC. A theoretical analysis of why such a transformation makes sense is also presented; Figure 3. Spatial divisions on the input image (left) and the feature map (right). Spatially dividing the input image is straightforward. The image is upsampled and fed to the same network to infer counts of local areas. The orange dashed line is used to connect the local feature map, the local count and the sub-image. S-DC on the feature map avoids redundant computations and is achieved by upsampling, decoding and dividing the feature map of high resolution.
• We investigate the explicit supervision for S-DC, which leads to a novel SS-DCNet. SS-DCNet is applicable to both regression-based and classification-based counters and can produce visually clear spatial divisions; • We report state-of-the-art counting performance over 5 challenging datasets with remarkable relative improvements. We also show good transferablity of SS-DCNet via cross-dataset evaluations on crowd counting datasets. A preliminary conference version of this work appeared in [49] where S-DCNet, the first version of SS-DCNet, was developed. Here we extend [49] in the following aspects: i) we provide theoretical analyses why closed set modeling makes sense; ii) we further enhance S-DCNet at the methodology level by investigating further a regression-based closedset counter, by integrating a count-orientated upsampling operator and by improving the model training with explicit supervision of spatial divisions; iii) we provide more ablative studies and qualitative analyses to highlight the role of S-DC; and iv) we give an insight of SS-DCNet w.r.t. its good transferablity in the output space and report state-of-the-art performance under the cross-domain setup.

RELATED WORK
Current CNN-based counting approaches are mainly built upon the framework of local regression. According to their regression targets, they can be categorized into two categories: density map regression and local count regression. We first review these two regression paradigms. Since SS-DCNet works not only in regression counts but also in classification, some works that reformulate the regression problem are also discussed.

Density Map Regression
The concept of density map was introduced in [18]. The density map contains the spatial distribution of objects, thus can be smoothly regressed. Zhang et al. [52] may be the first to adopt a CNN to regress local density maps. Then almost all subsequent counting networks followed this idea.
Among them, a typical network architecture is multi-branch. MCNN [53] and Switching-CNN [2] used three columns of CNNs with varying receptive fields to depict objects of different scales. SANet [4] adopted Inception [44]-liked modules to integrate extra branches. CP-CNN [41] added two extra density-level prediction branches to combine global and local contextual information. ACSCP [37] inserted a child branch to match cross-scale consistency and an adversarial branch to attenuate the blurring effect of the density map. ic-CNN [34] incorporated two branches to generate high-quality density maps in a coarse-to-fine manner. IG-CNN [1] and D-ConvNet [38] drew inspirations from ensemble learning and trained a series of networks or regressors to tackle different scenes. DecideNet [22] attempted to selectively fuse the results of density map estimation and object detection for different scenes. Unlike multi-branch approaches, Idrees et al. [15] employed a composition loss and simultaneously solved several counting-related tasks to assist counting. CSRNet [21] benefited from dilated convolution which effectively expanded the receptive field to capture contextual information.
Existing deep counting networks aim to generate highquality density maps. However, density maps are actually in the open set as well. For a single point, different kernel sizes lead to different density values. When multiple objects exist and are close, density patterns are even much diverse. Since observed samples are limited, density maps are certainly in an open set. In addition, density maps do not have the physical property of spatial decomposition. We therefore cannot apply S-DC to density maps.

Local Count Regression
Local count regression directly predicts count values of local image patches. This idea first appeared in [7] where a multi-output regression model was used to regress regionwise local counts simultaneously. Authors of [10] and [25] introduced such an idea into deep counting. Local patches were first densely sampled in a sliding-window manner with overlaps, and a local count was then assigned to each patch by the network. Inferred redundant local counts were finally normalized and fused to the global count. Stahl et al. [43] regressed the counts for object proposals generated by Selective Search [47] and combined local counts using an inclusion-exclusion principle. Inspired by subitizing, the ability for a human to quickly counting a few objects at a glance, Chattopadhyay et al. [5] transferred their focus to the problem of counting objects in everyday scenes. The main challenge thus shifted to large intra-class variances rather than the occlusions and perspective distortions in crowded scenes.
While some methods above [5], [43] leverage the idea of spatial divisions, they still regress the open-set counts. Despite the fact that local region patterns are easier to be modelled than the whole image, the observed local patches are still limited. Since only finite local patterns (a closed set) can be observed, new scenes in reality have a high probability including objects out of the range (an open set). Moreover, dense regions with large count values are rare ( Figure 2) and the networks may suffer from sample imbalance. In this paper, we show that a counting network is able to learn from a closed set with a certain range of counts, say 0 ∼ 20, and then generalizes to an open set (including counts > 20) via S-DC.

Beyond Simple Regression
Regression is a natural approach to estimate continuous variables, such as age, depth, and counts. However, some works suggest that regression is encouraged to be reformulated as an ordinal regression problem or a classification problem, which often enhances performance and benefits optimization [6], [11], [20], [28], [23] for many vision tasks. Ordinal regression is usually implemented by modifying well-studied classification algorithms and has been applied to the problem of age estimation [28] and monocular depth prediction [11]. Li et al. [20] further showed that directly reformulating regression to classification was also a good choice. In counting, the idea of blockwise classification is also investigated [23]. All these attempts motivate us to devise a classification-based closed-set counter. In this work, in addition to the standard regression-based modeling as in [25], SS-DCNet also follows [20] and [23] to discretize local counts and classify count intervals. Indeed, we observe in experiments that classification with S-DC typically works better than direct regression.

Open-Set Problems in Computer Vision
Many vision tasks are open-set by nature, such as depth prediction [11], [20], age estimation [6], [28], object recognition [36], visual domain adaptation [31], etc. While the sense of the open set may be different, they generally suffer from poor generalization as object counting. However, we currently find that, only the learning target of counting, i.e., the count value, can be easily transformed into a closed set (via spatial division).

SUPERVISED SPATIAL DIVIDE-AND-CONQUER NETWORK
In this section, we describe how to construct a closed-set counter. We also explain our proposed SS-DCNet in detail.

Closed-Set Counter
In local count modeling, there are two ways to define a counter in the closed set [0, C max ], i.e., counting by regression [10], [25] and counting by classification [49], [23].
In practice, C max should be not greater than the maximum local count observed in the training set. It is clear that treating C max as the maximum prediction will cause a systematic error, but the error can be mitigated via S-DC, discussed in Section 6.
Regression-Based Counter (R-Counter): R-Counter directly regresses count values within the closed set. If predicted count values are greater than C max , the predictions will simply be truncated to C max .

Classification-Based Counter (C-Counter):
Instead of regressing open-set count values, C-Counter discretizes local counts and classifies count intervals as in [23]. Specifically, we define an interval partition of [0, +∞) as {0}, (0, C 1 ], (C 2 , C 3 ], ... , (C M −1 , C max ] and (C max , +∞). These M + 1 sub-intervals Table 1 The Configurations of Counter, Division decider and Upsampler. AvgP ool denotes Average Pooling. Convolutional layers are defined in the format: kernel size Conv, output channel, s stride. Each convolutional layer is followed by ReLU except the last layer. In particular, a Sigmoid function is attached at the end of division decider to generate soft division masks. A Spatial Softmax function is applied at the End of Upsampler, which constrains the sum of upsampling weights in each 2 × 2 adjacent regions to be 1 and ensures consistent local count values in the same image area after upsampling. The final output channel is 1 for R-Counter and class num for C-Counter Counter Division decider/Upsampler 2 × 2 AvgPool, s 2 2 × 2 AvgPool, s 2 1 × 1 Conv, 512, s 1 1 × 1 Conv, 512, s 1 1 × 1 Conv, 1/(class num.), s 1 1 × 1 Conv, 1, s 1 − Sigmoid/Spatial Softmax are labeled to the 0-th to the M -th classes, respectively. For example, if a count value falls into (C 2 , C 3 ], it is labeled as the 2-nd class. The median of each sub-interval can be adopted when recovering the count from the interval. Notice that, for the last sub-interval (C max , +∞], C max will be used as the count value if a region is classified into this interval. In what follows, we term the network as SS-DCNet (reg) when R-Counter is adopted, and SS-DCNet (cls) when C-Counter is used.

Single-Stage Spatial Divide-and-Conquer
As shown in Figure 4, SS-DCNet includes a VGG16 [39] feature encoder, an UNet [35]-like decoder, a closed-set counter, a division decider and an up-sampler. The structure of the counter, the division decider and the upsampler are shown in Table 1. Note that, the first average pooling layer in the counter has a stride of 2, so the final prediction has an output stride of 64.
The feature encoder removes fully-connected layers from the pre-trained VGG16. Suppose that the input patch is of size 64 × 64. Given the feature map F 0 (extracted from the Conv5 layer) with 1 32 resolution of the input image, the counter predicts the local count value C 0 conditioned on F 0 . Note that C 0 is the local count without S-DC, which is also the final output of previous approaches [5], [10], [25].
We execute the first-stage S-DC on the fused feature map F 1 . F 1 is divided and sent to the shared counter to produce the division count C 1 ∈ R 2×2 . Concretely, F 0 is upsampled by ×2 in an UNet-like manner to F 1 . Given F 1 , the counter fetches the local features that correspond to spatially divided sub-regions, and predicts the first-level division counts C 1 . Each of the 2 × 2 elements in C 1 denotes a sub-count of the corresponding 32 × 32 sub-region.
With local counts C 0 and C 1 , the next question is to decide where to divide. We learn such decisions with another network module, division decider, as shown in the right part of Figure 4. At the first stage of S-DC, the division decider generates a soft division mask W 1 of the same size as C 1 conditioned on F 1 such that for any w ∈ W 1 , w ∈ [0, 1]. w = 0 means no division is required at this position, and the value in C 0 is used. w = 1 implies that here the initial prediction should be replaced with the division count in C 1 . Since both W 1 and C 1 are 2 times larger than C 0 , C 0 is required to be upsampled by ×2 toĈ 0 . Notice that, since Algorithm 1: Multi-Stage S-DC Input: Image I and division time N Output: Image count C 1 Extract F0 from I; 2 Generate C0 given F0 with the closed-set counter; 3 Initialize DIV0 = C0; 4 for i ← 1 to N do 5 Decode Fi−1 to Fi; 6 Process Fi with the closed-set counter, upsampler and the division decider to obtain Ci, Ui and the division mask Wi; 7 Upsample DIVi−1 toDIVi−1 with Ui as per Eq. (3); 8 Update DIVi as per Eq. (4) ; 9 Integrate over DIVN to obtain the image count C; 10 return C C 0 denotes the local count of a 64 × 64 region, the sum of C 0 should equal to C 0 . The upsampling of C 0 is therefore a re-distribution operator that assigns C 0 to each sub-region. We compute the re-distribution map U 1 from the upsampler conditioned on F 1 , and the sum of U 1 equals to 1. We then upsample C 0 toĈ 0 bŷ where "⊗" denotes Kronecker product and 1 2×2 denotes a 2 × 2 matrix filled with 1. Finally, the first-stage division result DIV 1 takes the form where 1 denotes a matrix filled with 1 and is with the same size of W 1 , and "•" denotes the Hadamard product.

Multi-Stage Spatial Divide-and-Conquer
SS-DCNet can execute multi-stage S-DC by further decoding, dividing the feature map until reaching the output of the first convolutional block. In this sense, the maximum division time is 4 in VGG16 for example. Actually we show later in experiments that a two-stage division is sufficient to guarantee satisfactory performance. In multi-stage S-DC, is first upsampled as per and then merged according to in a recursive manner. Multi-stage SS-DCNet is summarized in Algorithm 1.

Loss Functions
Here we elaborate the loss functions used in an N -stage SS-DCNet.
Counter Loss: As mentioned in Section 3.1, both R-Counter and C-Counter can be chosen. We adopt 1 loss, denoted by L i R , i = 0, 1, 2, ...N , for each level of output C i when R-Counter is used, and cross-entropy loss, denoted by L i C , i = 0, 1, 2, ...N , when C-Counter is chosen. Note that, both ground-truth local counts and predicted counts are truncated A shared upsampler, a closed-set counter and a division decider receive divided feature maps, and respectively, generate upsampling map U i s, division counts C i s and division masks W i s, for i = 1, 2, .... After obtaining these results, C i−1 is upsampled with U i , then it is merged with C i by W i to the i-th division count DIV i shown in the right sub-figure. In particular, we upsample each count of low resolution into the corresponding 2 × 2 area of high resolution before merging with U i . "•" denotes the Hadamard product, and "⊗" denotes the Kronecker product. Note that, the 64 × 64 local patch is only used as an example for readers to understand the pipeline of SS-DCNet. Since SS-DCNet is a fully convolutional network, it can process an image of arbitrary sizes, say M × N , and return DIV 2 of size M 64 × N 64 . The configurations of the closed-set counter and the division decider are presented in Table 1.
to C max when R-Counter is adopted. The overall counter loss is

Merging Loss (Implicit Division Supervision):
We also adopt a 1 loss L m for the final division output DIV N . L m provides an implicit supervision signal for learning W i s [49].
Upsampling Loss: The upsampler predicts U i used to redistribute DIV i−1 to its ×2 resolution outputD IV i−1 . We compute the ground truth of U i according to the distribution of local counts as per where U gt i , C gt i and C gt i−1 denote the ground truth of U i , C i and C i−1 , respectively. '/' denotes element-wise division. We We can also generate explicit division supervision to supervise W i by comparing the ground-truth C gt i and C max . For example, if C gt 0 , the count value of 64 × 64 local region D, is larger than C max , one cannot guarantee that all quarters of D are underestimated by predicted C 0 , but at least the quarter of D with the maximum count value is underestimated and required to be replaced with C 1 . This means that, if C gt 0 > C max , a constraint that the maximum value of W 1 should approach 1 can be generated. The division loss of W 1 thus takes the form where 1{P } denotes the indicator function which outputs 1 when the condition P is true, and 0 otherwise. C gt 0 is the ground truth count value of C 0 , and 'max' is the operator that returns the maximum value. Following Eq. (6), the loss L i div , i = 1, 2, ...N , for W i can be deduced as where C gt i−1 [j, k] denotes the element of the j-th row and the k-th column of C g i−1 t, and W i [2j − 1 : 2j, 2k − 1 : 2k] the elements lying in the (2j−1)-th to 2j-th row, (2k−1)-th to 2kth column of W i . The overall division loss L div = N i=1 L i div . Division Consistency Loss: When R-Counter is used, we can further constrain the consistency between different C i s when C i s are in the range of the closed set [0, C max ]. For C 0 and C 1 , the division consistency loss is defined by where 'sum' is the operator that returns the sum of all elements. Following Eq. 8, the consistency loss between C i−1 and The overall division consistency loss L eq = N i=1 L i eq . Notice that, when C-Counter is adopted, the gradient of the consistency loss cannot be backpropagated because count values are discretized into count intervals and represented by class labels. We simply drop the consistency loss in this case.
As a summary, for SS-DCNet (reg) the final loss L reg is and for SS-DCNet (cls) the final loss L cls is

OPEN SET OR CLOSED SET? A THEORETICAL ANALYSIS
How does SS-DCNet benefit from transforming count values from open set to closed set? Here we attempt to provide some theoretical insights of the benefits of closed set transformed by S-DC.
Definition 1 (Spatial Division of an Image). Given an image I ∈ R H×W ×K , where H, W and K denote the height, width and channel dimensions, respectively, the spatial division of I leads to a group of sub-images {I i ∈ R Hi×Wi×K } i=1,2,...,M that satisfy: Definition 2 (Open Set and Closed Set). Given a positive number C max , for ∀ x ≥ 0 and x ∈ R, we can define a closed set In object counting, C max is the maximum count value observed in the training set.

Lemma 1.
Given an image I, let be the spatial dichotomy division operator such that Proof. Suppose that M is the number of divided sub-images after N divisions. i) For N = 1, according to the definition of , which means M = 4; ii) For N = t, assume M = 4 t , we have then when N = t + 1, for each I k ∈ t (I), so M = 4 × 4 t = 4 t+1 holds for N = t + 1.
Since both i) and ii) hold, by mathematical induction, we can deduce M = 4 N after N divisions.
According to Lemma 1, we know that, an image I will be divided into 4 N sub-images at most after N divisions. A subsequent question of interest is that, how many spatial divisions are required to transfer count values from S O to S C ? This leads to our following proposition.

Proposition 1 (Minimum Division Times).
Assume an image I with a count value C * > C max , C * ∈ S O , is divided by the operator, then at least log 4 we have Hence, M ≥ C * Cmax . With Lemma 1, we know M = 4 N . Hence, 4 N ≥ C * Cmax , i.e., N ≥ log 4 C * Cmax . Note that, since N is an integer, the minimum division times are log 4 C * Cmax . Proposition 1 suggests a lower bound when SS-DCNet can transform count values from S O to S C . This is a prerequisite that SS-DCNet can work. Is such a transformation effective? We further provide a theoretical analysis as follows.

Definition 3.
Let C (C > 0) be the ground-truth value, andĈ the inferred value. We define the relative error by r = C−Ĉ C and the absolute error by a = |C −Ĉ| = C × | r |.
By Definition 3, it is clear that the expectation of | r | varies as the ground truth C changes. We thus have is assumed to be continuous.
Before the presentation of our main results, we further need the conclusion of the following theorem. and By Definition 4, With generalized triangle inequality [17], we have By taking the expectation of both sides of Eq. (20), it amounts to According to Theorem 1, If Proposition 2 states that, by transforming the count value from S O to S C , SS-DCNet can achieve lower counting errors on the condition that the expectation of the relative errors on S C is smaller than that on S O . Could this condition be satisfied in practice? A plausible explanation can be given by leveraging the concept of domain adaptation [3], [26]. Considering that, S C is treated as the source domain, and S O the target domain. Note that two domains do not have intersection in the output space (a large domain shift). Since counting models are highly optimized with sufficient samples on S C , while no sample on S O is provided. The condition may be satisfied with a high probability according to the common sense in transfer learning-the model trained on the source domain significantly degrades on the target domain due to distribution mismatch. We will further verify this condition via experiments in Section 5.
It is worth noting that, although our theory is developed specifically for object counting, the theoretical results are generic and not limited to this task. As long as the learning target is spatially divisible as the count value (so far we only find counting satisfies the property of spatial divisibility), without loss of generality, the same conclusion can be deduced, as stated in Corollary 1.

Definition 5 (Spatial Divisibility).
Let P ∈ [0, +∞) be the learning target of a vision task defined on an Image I. {I i } i=1,2,...,M is the spatial division of I, and p i is the corresponding learning target of each

Corollary 1.
Let P be the learning target and is spatially divisible. Let o a denote the absolute error of P on S O , c a the absolute error of P on S C (after spatial divisions), f (x) = E P =x | r |, and P max a predefined positive number. Given a positive number P * , if P * > P max and f (P * ) > max 0≤x≤Pmax f (x), then

OPEN SET OR CLOSED SET? A TOY-EXAMPLE JUSTIFICATION
As aforementioned, counting is an open-set problem, while the model is learned in a closed set. Can a closed-set counting model really generalize to open-set scenarios? Here we show through a controlled toy experiment that, the answer is negative. In addition, in this experiment we illustrate that SS-DCNet indeed works better than that without S-DC, which supports our Proposition 2. Inspired by [18], we synthesize a cell counting dataset to explore the counting performance outside a closed training set.

Synthesized Cell Counting Dataset
We first generate 500 256 × 256 images with 64 × 64 subregions containing only 0 ∼ 10 cells to construct the training set (a closed set).
To generate an open testing set, we further synthesize 500 images with sub-region counts evenly distributed in the range of [0, 20].

Baselines and Protocols
We adopt three approaches for comparisons, they are: i) a density regression baseline CSRNet [21]; ii) a regression baseline with pretrained VGG16 as the backbone and the R-Counter used in SS-DCNet as the backend, without S-DC. Regarding the discretization of count intervals, we choose 0.5 as the step because cells may be partially presented in local patches. As a consequence, we have a partition of {0}, (0.0.5],(0.5, 1], ... ,(9.5, 10] and (10, +∞). All approaches are trained with standard stochastic gradient descent (SGD). The learning rate is initially set to 0.0001 and is decreased by ×10 when the training error stagnates.

Observations
According to Fig. 5 (b), it can be observed that both regression and classification baselines work well in the range of the closed set (0 ∼ 10), but the counting error increases rapidly when counts are larger than 10. This suggests a conventional counting model learned in a closed set cannot generalize to the open set. However, SS-DCNet can achieve accurate predictions even on the open set, which confirms the advantage of S-DC.  Fig. 5(c) and (b) report how these two metrics vary , respectively. We have the following discussions:

Relative mean absolute error (rMAE) is an estimation of f (x) as per Definition 4, and the mean absolute error (MAE) is
• As shown in Fig. 5 f (x) satisfies for the classification baseline when the patch count C 0 ≥ 12. According to Proposition 2, under the condition above, SS-DCNet (cls) will show lower M AE than the classification baseline without S-DC. When C 0 ≥ 16, the same conclusion can be drawn between SS-DCNet (reg) and its open-set regression baseline.
In general, our toy experiment verifies Proposition 2 to a certain extent. It is encouraged to transform count values from the open set to the closed set.

EXPERIMENTS ON REAL-WORLD DATASETS
Extensive experiments are further conducted to demonstrate the effectiveness of SS-DCNet on real-world datasets. We first describe some essential implementation details. After that, ablation studies are conducted on the ShanghaiTech Part_A [53] dataset to highlight the benefit of S-DC. We then compare SS-DCNet against current state-of-the-art methods on five public datasets. Finally, we also report cross-domain performance to verify the generalization ability of SS-DCNet.
Mean Absolute Error (M AE) and Root Mean Squared Error (M SE) are chosen to quantify the counting performance. They are defined as where Z denotes the number of images, C pre i denotes the predicted count of the i-th image, and C gt i denotes the corresponding ground-truth count. M AE measures the accuracy of counting, and M SE measures the stability. Lower M AE and M SE imply better counting performance.

Interval Partition for C-Counter
We generate ground-truth counts of local patches by integrating over the density maps. The counts are usually not integers, because objects can partly present in cropped local patches. We evaluate two different partition strategies. In the first partition, we choose 0.5 as the step and generate partitions as {0}, (0, 0.5], (0.5, 1], ..., (C max − 0.5, C max ] and (C max , +∞), where C max denotes the maximum count of the closed set. This partition is named as One-Linear Partition.
In the second partition, we further finely divide the subinterval (0, 0.5], because this interval contains a sudden change from no object to part of an object, and a large proportion of objects lie in this sub-interval. A small step of 0.05 is further used to divide the sub-interval (0, 0.5], i.e., Other intervals remain the same as One-Linear Partition. We call this partition Two-Linear Partition.

Data Preprocessing
We follow the same data augmentation used in [21], except for the UCF-QNRF dataset [15] where we adopt two data augmentation strategies. In particular, 9 sub-images of 1 4 resolution are cropped from the original image. The first 4 sub-images are from four corners, and the remaining 5 are randomly cropped. Random scaling and flipping are also executed. [32]. We train SS-DCNet using SGD. The encoder in SS-DCNet is directly adopted from convolutional layers of VGG16 [39] pretrained on ImageNet, and the other layers employ random Gaussian initialization with a standard deviation of 0.01. The learning rate is initially set to 0.001 and is decreased by ×10 when the training error stagnates. We keep training until convergence. For the ShanghaiTech, UCF_CC_50, TRANCOS and MTC datasets, the batch size is set to 1. For the UCF-QNRF dataset, the batch size is set to 16 following [15].

Is SS-DCNet Robust to C max ?
When reformulating the counting problem into classification, a critical issue is how to choose C max , which defines the closed set. Hence, it is important that SS-DCNet is robust to the choice of C max .
We conduct a statistical analysis on count values of local patches in the training set, and then set C max with the quantiles ranging from 100% to 80% (decreased by 5%). Two-stage SS-DCNet is evaluated. Another baseline of classification without S-DC is also used to explore whether counting can be simply modeled in a closed-set classification manner. To be specific, we reserve the VGG16 encoder and the C-Counter in this classification baseline.
Results are presented in Figure 6. It can be observed that the MAE of the classification baseline increases rapidly with decreased C max . This result is not surprising, because the model is constrained to be visible to count values not greater than C max . This suggests that counting cannot be simply transformed into closed-set classification. However, with the help of S-DC, SS-DCNet exhibits strong robustness to the changes of C max . It seems the systematic error brought by C max can somewhat be alleviated with S-DC. Regarding how to choose concrete C max , the maximum count of the training set seems not the best choice, while some small quantiles even deliver better performance. Perhaps a model is only able to count objects accurately within a certain degree of denseness. We also notice Two-Linear Partition is slightly better than One-Linear Partition, which indicates that the fine division to the (0, 0.5] sub-interval has a positive effect.
According to the above results, SS-DCNet is robust to C max in a wide range of values, and C max is generally encouraged to be set less than the maximum count value observed. In addition, there is no significant difference between two kinds of partitions. For simplicity, we set C max to be the 95% quantile and adopt Two-Linear Partition in the following experiments.

How Many Times to Divide?
SS-DCNet can apply S-DC up to 4 times, but how many times are sufficient? Here we evaluate SS-DCNet with different division stages. The maximum count value of 64 × 64 image patches in the test set is 136.50 and C max = 22. With Proposition 1, we know 2-time division is required at least. Quantitative results are listed in Table 2. It can be observed that when the division time N varies from 0 to 2, the counting error M AE significantly decreases for both SS-DCNet (reg) and SS-DCNet (cls). However, counting accuracy saturates when N continues increasing. In general, two-stage S-DC seems sufficient. We use this setup in the following experiments. Figure 6. The influence of Cmax to SS-DCNet on the ShanghaiTech Part_A dataset [53]. The numbers in the brackets denote quantiles of the training set, for example, 22 (95%) means the 95% quantile is 22. 'VGG16 Encoder' is the classification baseline without S-DC. 'One-Linear' and 'Two-Linear' are defined in Section 6.1.1. SS-DCNet (reg/cls) adopts two-stage S-DC. Table 2 Results of SS-DCNet with different S-DC stages. The best performance is in boldface.

The Effect of S-DC
To highlight the effect of S-DC, we compare SS-DCNet against several regression and classification baselines. These baselines adopt the same architecture of VGG16 encoder and the counter in SS-DCNet. classification is the result of C 0 adopting C-Counter without S-DC, and C max is set to be the 95% quantile (C max = 22). For regression baselines, we employ R-counter to obtain the prediction C 0 without S-DC. We create two regression baselines. open-set regression + S-DC is straightforward. We do not limit the output range, and it can vary from 0 to +∞. S c regression indicates that the output range is constrained within [0, C max ] (C max is also set to 22 for a fair comparison). Any large outputs will be clipped to C max . Results are shown in Table 3. We can see that counting by classification without S-DC suffers from the limitation of C max and performs even worse than S o regression. S c regression also suffers from the same problem. However, with S-DC, SS-DCNet (reg/cls) significantly reduces the counting error and outperforms both their regression/classification baseline by a large margin. It suggests that a counting model can learn from a closed set and generalize well to an open set via S-DC. We notice that SS-DCNet (cls) performs better than SS-DCNet (reg). It seems that reformulating counting in classification is more effective than in regression. One plausible reason is that the optimization is easier and less sensitive to sample imbalance in classification than in regression.   We further analyze the counting error of 64 × 64 local patches in detail. As shown in Figure 7, we observe that the direct prediction C 0 without S-DC performs worse than the S o regression baseline and CSRNet, which can be attributed to the limited C max of the C-Counter. After embedding S-DC, the counting errors of DIV 1 and DIV 2 significantly reduce and outperform open-set regression and CSRNet. Such a benefit is even much clear in dense patches with local counts greater than 100. It justifies our argument that, instead of regressing a large count value directly, it is more accurate to count dense patches through S-DC.

Choices of Loss Functions
Here we validate the effect of different loss functions used in SS-DCNet. Results are reported in Table 4. As analyzed in S-DCNet [49], L C provides supervision to C i s, and L m implicitly supervises the division weights W i s. With only L C and L m , S-DCNet (cls) can achieve good division results. However, S-DCNet (reg) cannot report competitive results as S-DCNet (cls) with L R and L m . After incorporating the upsampling loss L up in the SS-DCNet (reg/cls), MAE reduces by 3.2/0.5. Such an improvement can be attributed to the replacement of the average upsampling in S-DCNet with learned upsampling in SS-DCNet. L div provides explicit supervision for spatial division weight W i s. One can see that, L div can further improve the counting performance of SS-DCNet (reg/cls), and clear division results can be observed as shown in Fig. 8. Moreover, division consistency loss L eq is also effective for SS-DCNet (reg), with 1.1 improvement in MAE. Overall, SS-DCNet (reg/cls) shows a clear advantage Table 5 Configuration of SS-DCNet. max denotes the maximum count of local patches in the training set. Cmax is the maximum count of the closed set in SS-DCNet. Gaussian kernel is used to generate density maps from dotted annotations. Specially, since UCF_CC_50 adopts 5-fold cross-validation, max and Cmax are set adaptively for each fold.

Spatial Divide-and-Conquer Versus Spatial Attention
To highlight the difference between S-DC and spatial attention (SA), we remove the division decider, generate a 3-channel output conditioned on F 2 , then normalize it with softmax to obtain W att 0 , W att 1 and W att 2 . The final count is merged as W att 0 * upsample(C 0 ) + W att 1 * upsample(C 1 ) + W att 2 * C 2 . In SHTech PartA, SA achieves 64.1 M AE and 109.9 M SE, worse than SS-DCNet. As per the visualization of W att i in Fig. 8, we find SA only focuses on the highest resolution, and no effect of division is observed. Instead, SS-DCNet learns to divide local patches when local counts are greater than C max . In addition, SS-DCNet executes fusion recursively, while SA fuses the prediction in a single step.

Comparison with State of the Art
According to the ablation study, the final configurations of SS-DCNet are summarized in Table 5.

The ShanghaiTech Dataset
The ShanghaiTech crowd counting dataset [53] includes two parts: Part_A and Part_B. Part_A has 300 images for training and 182 for testing. This part represents highly congested scenes. Part_B contains 716 images in relatively sparse scenes, where 400 images are used for training and 316 for testing. Quantitative results are listed in Table 6. The improvements of SS-DCNet are two-fold. First, with the explicit supervision of S-DC, SS-DCNet (cls) performs better than our previous S-DCNet. Second, our method outperforms the previous stateof-the-art PGCNet [51] in Part_A and competitive results (6.6 MAE) as SPANet [9] (6.5 MAE) in Part_B, respectively. These results suggest SS-DCNet is able to adapt to both sparse and crowded scenes.    Table 8 Comparison with state-of-the-art approaches on the Test Set of UCF-QNRF [15] dataset. "r" denotes image resizing used in [27]. The best performance is in boldface.  Table 9 Comparison with state-of-the-art approaches on the test set of TRANCOS [13] dataset. The best performance is in boldface.

The UCF_CC_50 Dataset
UCF_CC_50 [14] is a tiny crowd counting dataset with 50 images in extremely crowded scenes. The number of people within an images varies from 96 to 4633. We follow the 5-fold cross-validation as in [14]. Results are shown in Table 7. Our method surpasses S-DCNet and the previous best method, PaDNet [45], with 12.2% and 3.4% relative improvements in MAE, respectively.

The UCF-QNRF Dataset
UCF-QNRF [15] is a relatively large crowd counting dataset with 1535 high-resolution images and 1.25 million head annotations. There are 1201 training images and 334 test images. It contains extremely congested scenes where the maximum count of an image can reach 12865. Some images in the UCF-QNRF dataset are too large, with the longer side equals to 10000, to process the whole image. There are two ways to solve this problem: i) cropping the original image into 224 × 224 sub-images following [15]; ii) resizing the original image to make the longer side no larger than 1920 as in [50], [27], then 9 sub-images of 1 4 resolution are cropped from the original image for data augmentation as described in Section 6.1.2. Results are reported in Table 8. We can make following observations: • For S-DCNet, it works better with strategy ii) than i). This means that resizing is a better choice than cropping. We think the reasons are two-fold. First, the receptive field of a CNN is limited, thus it cannot cover over-size images. Second, if cropping over-size images into 224 × 224 subimages, the surrounding pixels of sub-images, termed 'local visual context', are invisible to the CNN. However, the local visual context can provide support information to distinguish overlapped objects as demonstrated in [48], and CNNs tend to perform poorly when local context is lost. • With the explicit supervision of S-DC, SS-DCNet (cls) brings a significant improvement over S-DCNet by 15.8 in MAE, and SS-DCNet (reg) shows by 5.3 in MAE. • SS-DCNet (reg) reports competitive results against the current state-of-the-art BAY ESIAN + [27], while SS-DCNet (cls) outperforms BAY ESIAN + by 6.8 in MAE and 11 in MSE. • It is worth noting that, SS-DCNet only learns from a closed set with C max = 8.0, which is only 6% of the maximum count 131.5 according to Table 5. SS-DCNet, however, generalizes to large counts effectively and predicts accurate counts.

The TRANCOS Dataset
Aside from crowd counting, we also evaluate SS-DCNet on a vehicle counting dataset, TRANCOS [13], to demonstrate the generality of SS-DCNet. TRANCOS contains 1244 images of congested traffic scenes in various perspectives. It adopts the Grid Average Mean Absolute Error (GAME) [13] as the evaluation metric. GAM E(L) divides an image into 2 L × 2 L non-overlapping sub-regions and accumulates of the M AE over sub-regions. Larger L implies more accurate local predictions. In particular, GAM E(0) downgrades to M AE. The GAME is defined by where N denotes the number of images. C l pre and C l gt are the predicted and ground-truth count of the L-th subregion, respectively. Results are listed in Table 9. SS-DCNet surpasses other methods under all GAM E(L) metrics, and particularly, delivers a 33.1% relative improvement than SPN [8] on GAM E(3). This suggests SS-DCNet not only achieves accurate global predictions but also behaves well in local regions.

The MTC Dataset
We further evaluate our method on a plant counting dataset, i.e., the MTC dataset [25]. The MTC dataset contains 361 high-resolution images of maize tassels collected from 2010 to 2015 in the wild field. In contrast to pedestrians or vehicles that have similar physical sizes, maize tassels are with heterogeneous physical sizes and are self-changing over time. We believe that this dataset is suitable for justifying the robustness of SS-DCNet to object-size variations. We follow the same setting as in [25] and report quantitative results in Table 10. Although the previous best method, TasselNetv2 † [48], already exhibits accurate results, SS-DCNet still shows a substantial degree of improvement (26.4% on M AE and 29.8% on MSE). Qualitative results are shown in Fig 9. We can observe that SS-DCNet produces accurate predictions for various objects from sparse to dense scenes.

Cross Dataset Generalization
We further conduct cross-dataset experiments on the Shang-haiTech [53] (A and B) and UCF-QNRF (QNRF) [15] datasets to show the transfer ability of SS-DCNet. Quantitative results are shown in Table 11. The 'regression' and 'classification' methods are baselines for SS-DCNet (reg) and S-DCNet (cls), respectively, which adopt the VGG16 [39] as the feature encoder and R-Counter/C-Counter in SS-DCNet but do not apply S-DC. We can make following observations: • Consistent improvements in MAE are observed when comparing SS-DCNet (reg/cls) to its baselines. Especially in SS-DCNet (cls) vs. baseline cls, the M AE of B → QN RF shows a 40.89% relative improvement when S-DC is added. • Two types of SS-DCNet report superior or at least competitive results than other state-of-the-art methods under all cross-dataset tasks, which suggest SS-DCNet has strong transferring ability. • SS-DCNet (cls) transferred from the QNRF dataset reports even better results (61.8 MAE) than most state-of-the-arts methods (e.g., 68.2 MAE for CSRNet and 67.0 MAE for SANet) trained on the ShanghaiTech dataset. • All methods trained on the ShanghaiTech [53] dataset report worse cross-dataset results than trained directly on the target datasets. By contrast, all methods trained on the QNRF [15] dataset exhibits at least competitive transferring results against state-of-the-art methods trained on the target dataset. This may be attributed to the fact that the ShanghaiTech dataset is too small, with only 300 training samples in the Part_A and 400 in the Part_B, to train a robust model, while the QNRF dataset provides sufficient training samples. Overall, SS-DCNet demonstrates state-of-the-art results in all cross-dataset experiments. The good performance of SS-DCNet may be explained from its implicit transferring ability in the output space, which shares the same spirit with [46]. To justify this, we analyze the case of QN RF → A and visualize the distribution of count values with and without S-DC in Fig. 10. It can be observed that, the distribution of count values varies significantly between SHA and QNRF without S-DC, but the divergence of the distribution narrows down after count values are transformed into a closed set [0,8]. We further compute the JensenâȂŞShannon divergence [30] J s to quantify the divergence of the distribution, and find that J s = 0.0178 between SHA and QNRF without S-DC and J s = 0.0133 with S-DC. The smaller J s is, the smaller divergence between two distributions shows. This means the divergence of the output (count value) space reduces after closed-set transformation. We think this is the main reason why SS-DCNet reports remarkable performance in the task of QN RF → A. In short, SS-DCNet improves the cross-dataset generalization via the implicit adaptation of the output (count values) space.

CONCLUSION
Counting is an open-set problem in theory, but only a finite closed set can be observed in reality. This is particularly true because any dataset is always a sampling of the real world. Inspired by the decomposable property of counting, we Figure 9. Visualization of count map for SS-DCNet on various datasets. From left to right, for each visualization, the original image, the groundtruth count map, and the inferred count map, respectively. From top to down, left to right, the visualization of ShanghaiTech Part_A, Part_B [53], UCF_CC_50 [14], UCF-QNRF [15], TRANCOS [13], and MTC [25], respectively. The original image is zero-padded to be divisible by 64.  propose to transform the open-set counting into a closed-set problem, and implement this transformation with the idea of S-DC. We propose supervised S-DC in a deep counting network termed SS-DCNet. We provide a theoretical analysis showing why the transformation from the open set to closed set makes sense for counting. A toy experiment and extensive evaluations on standard benchmarks are also conducted to show that, even given a closed training set, SS-DCNet can effectively generalize to open-set scenarios. Furthermore, SS-DCNet shows its good generalization ability via state-ofthe-art cross-dataset performance. Many vision tasks are open-set by nature, depth estimation for example, while it is not immediately clear on how to tranform them into a closed set like counting. It would be interesting to explore how to transform other vision tasks into a closed set setting.