{'fraction_non_alphanumeric': 0.03657894736842105, 'fraction_numerical': 0.012763157894736843, 'mean_word_length': 4.905594405594406, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FusionCount: Efficient Crowd Counting via Multiscale Feature Fusion

State-of-the-art crowd counting models follow an encoder-decoder approach. Images are first processed by the encoder to extract features. Then, to account for perspective distortion, the highest-level feature map is fed to extra components to extract multiscale features, which are the input to the decoder to generate crowd densities. However, in these methods, features extracted at earlier stages during encoding are underutilised, and the multiscale modules can only capture a limited range of receptive fields, albeit with considerable computational cost. This paper proposes a novel crowd counting architecture (FusionCount), which exploits the adaptive fusion of a large majority of encoded features instead of relying on additional extraction components to obtain multiscale features. Thus, it can cover a more extensive scope of receptive field sizes and lower the computational cost. We also introduce a new channel reduction block, which can extract saliency information during decoding and further enhance the model's performance. Experiments on two benchmark databases demonstrate that our model achieves state-of-the-art results with reduced computational complexity.


INTRODUCTION
Crowd counting aims to automatically estimate the number of individuals present in a scene from an image or video. It can be applied in numerous areas, such as traffic control [1], biological studies [2], and recently, social distancing monitoring [3].
Over the years, models for crowd counting have evolved from using classical regression models, such as random forests [4] and Gaussian processes [5], to high-performing convolutional neural networks (CNNs) [6,7,8,9]. These deep nets usually adopt an encoderdecoder approach: First, an image is fed to the encoder to learn data representations (feature maps). The decoder then exploits the highest-level representation (the output from the encoder's last layer) to generate the density map, which is the distribution of the crowd. Since the convolutional and pooling blocks of VGG networks [10], where each layer exploits kernels of a fixed size, constitute most encoders, the size of receptive fields remains constant across the last encoded feature map. Thus, this representation can only handle images where crowds are of similar scales. However, people are usually depicted in various sizes because of the camera perspective, and as a result, the encoded feature should also have different receptive field sizes to model the scale variation. Multi-column structures [6,11] have been proposed to solve this problem. However, it has been recently shown that features from each column are almost identical, and training deep models of this type can be very unproductive [7]. Therefore, to solve this scale issue, state-of-the-art methods [8,12,13] employ a multiscale module that further processes the encoded representation and generates a feature map with different receptive field sizes. However, such a strategy ignores that those feature maps extracted by shallower encoding layers already provide information about different scales, and leveraging extra components makes the overall model more computationally expensive.
Hence, our contribution in this paper is a novel multiscale mechanism that addresses the scale issue by leveraging the majority of features generated from the encoder to avoid extra feature-extraction modules and keep the computational cost low. This design incorporates a comprehensive range of receptive field sizes (6 to 192), covering almost all possible scales a person can depict in a crowd image. Experiments on two benchmark databases (ShanghaiTech A & B [6]) demonstrate that our model can achieve state-of-the-art or comparable results with significantly fewer floating-point operations.

RELATED WORK
Early crowd counting approaches [14,15,16] are based on object detectors, while later works tend to avoid them because of their sensitivity to occlusion and the enormous efforts required to annotate bounding boxes. Some of these non-detection-based methods [4,5,17,18] treat crowd counting as a regression problem: they learn low-level feature representations first, from which the total count is then directly regressed. The training losses of these approaches depend only on the ground-truth count (a scalar) and do not consider crowd density distribution, hence suffering poor generalisation. Thus, these models have soon been superseded by algorithms [19,20,8,12] that instead predict crowd densities, and these density-based methods primarily rely on CNNs.
Since 3D spatial locations have to be projected onto a 2D space while an RGB crowd image is captured from the real world, people can be depicted in different sizes due to perspective distortion, and the variation in scales can severely affect density estimation. To solve this problem, in [21], extra geometric information is exploited to adapt their model to different scenes, but this information is not always provided. Therefore, later methods tend to learn the scales of crowds implicitly. For example, Hydra-CNN [20] divides an image into a pyramid of patches, each representing a different scale and fed to a separate encoder head. Then, all encoded features are concatenated without further processing and utilised to generate the density map. This approach neglects the fact that scales vary continuously across the whole image. CAN [8] is then proposed to address this issue. The whole model involves only one encoder, so a spatial pyramid pooling module [22] is leveraged to make it scale-aware. Then, features are averaged according to learnable weights to ensure that receptive field sizes of the fused feature changes smoothly. However, this architecture is less efficient since it does not exploit low-level features extracted during encoding. These representations, along with the final high-level feature map, can provide information about different scales because they have disparate receptive field sizes. Also, the multiscale module in [8] uses only four filter sizes, thereby covering a limite range of scales.

OUR MODEL
This section exhaustively describes our proposed crowd counting model, FusionCount. It extensively leverages representations learned during encoding to compute first-phase multiscale features, and its decoder further fuses these scale-aware features to generate the density map.

Encoding
Following common practices [7,8] in the field, VGG-16 [10] is employed as our model's encoder. The last max-pooling and fully connected layers are removed since they are accountable for class prediction. Therefore, the encoder comprises 13 convolutional layers and four max-pooling layers. Since the two feature maps extracted before the first max-pooling operation are not sufficiently informative, only features learned afterwards are preserved and fused. Thus, as shown in Fig. 1, the encoder outputs 15 feature maps in total, denoted by xj, j = 1, 2, · · · , 15. These feature maps are divided into four groups according to their heights and widths: x1 -x3, x4 -x7, x8 -x11, and x12 -x15.

Feature Fusion
Suppose xi, xi+1, · · · , x i+k is a group of feature maps of interest, where xi is generated by a max-pooling layer and has c channels, while the rest are are produced by convolutional layers and have c * channels. To assemble a scale-aware representation from them, we adopt a similar strategy as the one proposed in [8], and our method is illustrated in Fig. 2. Given that scales change continuously accross an image, the receptive field size of the output multiscale feature, which models the scales, should also be spatially continuous. To accomplish this aim, we fuse these features via weighted averaging, where weight maps are learned from each feature's spatial importance.
Firstly, since xi and xi+j (j ∈ {1, · · · , k}) may not have the same number of channels, we use point-wise convolution to expand xi, and denote the output by x * i . Notice that since xi is the output of a max-pooling layer and the derivation of x * i does not involve any integration of neighbouring spatial information, x * i is less influenced by background noise. With such an advantage, all other representations are compared against it. Namely, following [8], we propose a similar concept to contrast features: with j = 1, · · · , k. Since x * i and xi+j (j = 1, · · · , k) have different receptive field sizes, the contrast features cj incorporate disparities between any spatial location and its neighbouring pixels. Thus, they can facilitate learning the boundary of each person, thereby determining each receptive field size's relative importance. We then compute the weights ωj as follows: where σ denotes the sigmoid function. 1 Using these weights, ωi+1, · · · ω i+k , we then combine {xi, xi+1, · · · , x i+k } adaptively and concatenate the averaged feature map with xi. Finally, we add a bottleneck layer to reduce computation. This process can be expressed as: where f , Conv, "|" and " " denote the fused feature, the bottleneck layer, channel-wise concatenation, and element-wise product, respectively.

Decoding
As shown in Fig. 2, we denote the first-phase multiscale features generated from the four groups via (3) as f 1 , f 2 , f 3 and f 4 . These features are further fused in the reverse order (see Fig. 3). To combine f 4 and f 3 via addition, we first transform the shape of f 4 . Specifically, the number of channels of f 4 is lowered, while its spatial size is expanded. Traditionally, a single point-wise convolutional kernel is used to reduce the nubmer of channels. However, inspired by dilated convolution, which has been shown to have the ability to extract deeper salient information while maintaining the spatial resolution [7], we propose a new channel reduction module. Our two-stream module comprises a dilated convolution block and one bottleneck layer, and the two columns are connected via addition. 2 To match the spatial resolution of f 3 , we use bilinear interpolation to double the size of f 4 after channel redcution. Then these two features can be fused by summation. The new fused feature is then combined with f 2 following a similar pattern -we first modify its dimensions via the proposed channel reduction module and bilinear interpolation, and then add it to f 2 . This process is followed iteratively until all features are fused. Finally, we feed the final fused feature to the output layer to generate the estimation.

Datasets
We use the ShanghaiTech A & B datasets [6] for model evaluation and comparison. In ShanghaiTech A, there are 482 crowd im-

Experiment Settings
During recent years, loss functions based on probability theory, such as Bayes' theorem and Wasserstein distance, have been shown to help models achieve stronger generalisation capabilities and have thus gained increased popularity. We use the DM-Count loss [23] to supervise the training of FusionCount. An Adam optimiser [24] with a learning rate 1e-5 and batch size two is leveraged for optimisation. In order to make fair comparisons with other approaches, we use the default data splits. Given that some images have intractably large sizes, from each input image, two patches with a size of 384 × 512 are cropped and used for training. Our model and its training are implemented in the PyTorch [25] 1.10 framework, and the platform for training is a server with an NVIDIA RTX 3090 GPU and Ubuntu 20.04 LTS OS.

Results
Following prior works [7,8,23], the mean absolute error (MAE) and the root mean squared error (RMSE) of total counts are employed as evaluation metrics. FusionCount is compared with state-of-the-art models that have a similar computational complexity (quantified by the number of multiplications and additions involved in the inference on a 1080 × 1920 RGB image). In particular, CSRNet [7], CAN [8], BL [9] and DM-Count [23], all of which employ VGGs [10] as encoders, are encompassed for comparison. CSRNet uses dilated convolution in decoding to extract saliency. In addition to this characteristic, CAN also includes a spatial pyramid pooling block to generate multiscale features. BL and DM-Count have no innovations in terms of model architectures, and their contributions are particularly novel loss functions based on probability theory.  Fig. 4. The left column shows the original input images, and the right column illustrates the ground-truth and the predicted density maps, which are indicated by blue dots and orange shades, respectively. The example in the first row proves that our model can still work well in highly crowded cases. Although in this 600 × 900 image, there are over 1,600 people, our model still achieves an accurate prediction with a relatively small error (1.98%). Other rows demonstrate that our model is capable of effectively dealing with scale changes. In these images, crowds in the lower parts of the scene have larger scales, while those in the upper part of the scene have the smallest scales. Our model can make correct predictions for both cases with negligible errors (0.95%, 0.70% and 0.05%, respectively). Thus, all these four instances confirm our model's strong counting ability.

Effectiveness of Contrast Features
To confirm that the contrast features can boost the model's performance, we create a variant for our model, whose only difference from FusionCount is the feature fusion strategy. In this variant, weights are directly generated from encoded features xi+j instead of contrast features ci+j. We train this variant on ShanghaiTech B [6] with the same setting. The MAE and RMSE of this variant are 7.6 and 12.9, respectively, which are larger than those of FusionCount (6.9 and 11.8).

Ablation Study
This section proves the effectiveness of the channel reduction module proposed in Section 3.3. Experiments are conducted on Shang-haiTech B [6]. The results, which are shown in Table 2, demonstrate that both the point-wise convolutional layer and the two chained dilated convolutional layers are indispensable. Theoretically, they work in a complementary way in reducing the number of channels and extract saliency. Table 2. Effects of point-wise convolution and the number of dilated convolutional layers in the channel reduction module.

CONCLUSION
In this paper, we proposed a new crowd counting architecture, Fu-sionCount, which smartly utilises a large majority of features generated during the encoding process to handle perspective distortion. Unlike existing approaches, FusionCount avoids further extraction of multiscale features, thereby significantly reducing overall computation. To this end, We have also improved an existing multiscale fusion mechanism and devised a novel channel reduction block. Experiments on the ShanghaiTech databases demonstrated that Fusion-Count can outperform relevant state-of-the-art approaches of similar computational complexity. As part of our future work, we are working on accounting for any contextual information in the features fused at the decoding process. Such information can help to more effectively deal with scale changes, as the way the first-stage fusion of encoded features does.