{'fraction_non_alphanumeric': 0.03528285209192693, 'fraction_numerical': 0.01123305833824396, 'mean_word_length': 4.972943246810383, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CrowdMLP: Weakly-Supervised Crowd Counting via Multi-Granularity MLP

Existing state-of-the-art crowd counting algorithms rely excessively on location-level annotations, which are burdensome to acquire. When only count-level (weak) supervisory signals are available, it is arduous and error-prone to regress total counts due to the lack of explicit spatial constraints. To address this issue, a novel and efficient counter (referred to as CrowdMLP) is presented, which probes into modelling global dependencies of embeddings and regressing total counts by devising a multi-granularity MLP regressor. In specific, a locally-focused pre-trained frontend is cascaded to extract crude feature maps with intrinsic spatial cues, which prevent the model from collapsing into trivial outcomes. The crude embeddings, along with raw crowd scenes, are tokenized at different granularity levels. The multi-granularity MLP then proceeds to mix tokens at the dimensions of cardinality, channel, and spatial for mining global information. An effective proxy task, namely Split-Counting, is also proposed to evade the barrier of limited samples and the shortage of spatial hints in a self-supervised manner. Extensive experiments demonstrate that CrowdMLP significantly outperforms existing weakly-supervised counting algorithms and performs on par with state-of-the-art location-level supervised approaches.


Introduction
Object counting strives to estimate the total number of objects dispersing across still images [35] or dynamic video sequences [25]. It has increasingly drawn attention from computer vision community, thanks to its wide spread societal applications, e.g., social distance monitoring [28], traffic surveillance [48], counting in agriculture [24] and metropolis crowd management [26].
Early counting approaches either adopt body detection-based techniques [12] or recur to count mapping [5], which perform unsatisfactorily when encountering highly congested scenes. To evade the issue, Lempitsky et al. [13] introduces density maps, thereby casting the counting problem into that of predicting pixel-wise density values. Benefiting from the boom of CNNs, density map prediction has become the mainstream choice in the realm of crowd counting [49,15,18,10,44]; see Figure 1(a).
Albeit the ever-improving counting accuracy, it is notorious that training such pixel-to-pixel probability predictor demands plenty of location-level (strong) annotations. The prevailing datasets with such annotations are generated by manually placing dots at the centroids of people heads across the whole samples (e.g., 1.51 million dots are labelled for the JHU-CROWD++ dataset [32]), which is tedious and time-consuming. Therefore, a variety of recent approaches have been proposed to relieve the labelling burden by employing as few annotated samples as possible. For example, AL-AC [50] uses limited annotated images via active labelling strategy, whereas other works [23,34] design semi-supervised schemes to synthesize pseudo ground truths for supervising a larger host of unlabelled images; see Figure 1(b). More recently, Xu et al. [44] make further efforts to reduce the areas of pixel-wise labels by leveraging fixed partial region in each density map; see Figure 1(c). But even so, the protocol of using scarce location-level supervision has yet been dispensed with and the requirement of labour-intensive and memory-consuming dot maps persists.
If we rethink the necessity of location-wise spatial constraints, then training crowd counters with count-level annotations, i.e. the total number of objects of interest (see Figure 1(d)), has many merits. These include: i ) Training models to predict accurate 2D density maps or dot maps introduces a domain gap between training and inference phases since the models are evaluated solely on total counts, impeding the leaning of object-agnostic repeat patterns; ii ) 2D density maps are derived from ground-truth dot maps through a heuristically-defined procedure with inductive bias, which could confuse the models on what to learn; iii ) Density map requires up to 1,000 times more memory space compared to the global counts, e.g., 362 MB vs. 364 KB for the testing set in ShanghaiTech Part A [49]; and iv ) More essentially, collecting data with single ground-truth counts can be much easier in many real-world scenarios. For example, when the same scene is captured by multiple cameras, the manual count obtained on one view can serve as ground truths for all captured multi-view images. Furthermore, once an initial count is known for a scene with controlled access, the ground truths for future frames can be obtained easily by adding/subtracting numbers of people entering/leaving. Hence, count-level annotations hold great promises in building large-scale benchmark datasets for multifarious counting problems.
Recently, a resurge of multi-layer perceptrons (MLP) has lead to cuttingedge purely MLP-based paradigms for image classification [6,37], which benefit from intrinsic advantages of underlying fully-connected layers: more global receptive fields with less inductive bias than de facto CNNs component, simpler than self-attention layers in Transformer [38]. Inspired by these achievements, this paper presents a novel weakly-supervised CrowdMLP architecture, which exploits global-range receptive fields without considering the highly-costly selfattention mechanism and attains better performance than the transformer-based counterpart [16]. As shown in Figure 2, CrowdMLP is featured by the collaboration between a locally-focused CNN frontend and a devised global-oriented multi-granularity MLP regressor. Low-level representations with limited receptive fields are extracted by the pre-trained CNN frontend. These feature maps, along with raw input scenes, are then reshaped and linearly projected to construct a pyramid of token sequences at multiple levels of granularity. The holistic MLP regressor proceed to characterize these tokens and produce embeddings with global-range dependencies, which are fed into a high-level MLP head to directly predict a single and continuous count value. While training with count-level supervisory signals avoids inductive bias, the lack of location information often causes the performance to degenerate. An auxiliary counting-specific proxy task, namely Split-Counting, is designed to stem the degradation by randomly splitting the input into two complementary parts and enforcing the sum counts for the two parts to be close to the count for the whole (see Figure 2). This strategy implicitly enhance spatial awareness in a self-supervised manner, leading to accuracy gains.
In a nutshell, our contributions are summarized as follows: 1. A novel CrowdMLP is proposed to mitigate the burdensome location-level annotations by directly regressing total counts. By proposing multi-granularity MLP regressor with less inductive bias, global-range receptive fields at distinct semantics levels are exploited, while circumventing cumbrous self-attention module in Transformer. 2. To avoid performance loss caused by the lack of spatial hints in count-level supervision, a self-supervised proxy task is introduced to implicitly impose spatial cues and remedy the accuracy degradation. 3. Extensive experiments demonstrate that CrowdMLP offers massive performance gains over existing weakly-supervised approaches (i.e. CNN-and The overall architecture of CrowdMLP, which is featured by coarse-to-fine token streams, multi-granularity MLP, and self-supervised proxy task. Transformer-based counterparts) and competitive results as fully-supervised ones. They also shed light on the density-and spatial-awareness properties of CrowdMLP.

Counting with Fully-supervised Settings
Early approaches tackle the problem of crowd counting by detecting body parts [12,42,39,1 or regressing crowd numbers [5]. However, when handling highly congested scenes with dense crowd, severe occlusions, and clutter backgrounds, these traditional methods perform poorly. With the impressive capability of CNNs [31] and the introduction of density map [13], numerous CNN-based algorithms are proposed to predict density maps. Among these modern methods, MCNN [49] employs multiple columns with different kernel sizes to extract multi-scale features. Switch-CNN [1] uses an auxiliary classification sub-network to determine which branch should be selected for the regressor at each image patch. SANet [3] adopts inception module into the feature encoder and presents a local pattern consistency loss. CSRNet [15] stimulates a line of studies on enlarging the receptive fields via dilated convolutions. CAN [20] and SASNet [36] investigate the impacts of attention mechanisms on accuracy. Furthermore, BL [27], BM-Count [17], and P2PNet [35] propose to maximize confidence of head centres with only dot annotation.

Counting With Weakly-supervised Settings
For the sake of reducing the burden of annotations, a suite of models attach importance to weak supervisory signals. L2R [21] is presented to leverage abundantlyavailable crowd scenes from the Internet via an auxiliary sorting task with selfsupervised settings. Liu et al. [23] design a self-training strategy to train a robust feature extractor from unlabelled samples. Simultaneously, Sindagi et al. [34] incorporate a learning mechanism of Gaussian process to generate pseudo labels for unlabelled data. Considering how the model parameters evolve, GWTA-CCNN [29] updates 99.9% of the parameters in an auto-encoder manner using unlabelled crowd images, whereas remaining weights of the last two layers are fine-tuned with density map supervision. Xu et al. [44] use stochastically-selected partial sub-region in each density map to train the model. Although the above approaches make use of extra unlabelled images or regions, location-level dot or density maps supervisions are still decisive and vital for the final performance. To get rid of such dependency, Yang et al. [45] move towards implicitly modelling the hidden relationships in count numbers and predicting the crowd number, which is credited to the side sorting sub-networks and hand-craft soft labels. Nevertheless, the performance achieved is unsatisfactory since CNN-based regressor is incompetent to hunt for global receptive fields. The model is also too sophisticated, making it prone to overfitting. Different from their work, we probe into a model fed by count-level supervisory signals from the perspective of multi-granularity token sequences.

Cutting-edge Learning Paradigms
The past two years witness the focus of preeminent learning paradigms what aim to compete with de facto CNNs. Transformer-based ViT [7] opens a new avenue for treating image patches the similar way as word tokens and attains competitive results on image classification. Furthermore, the fabric-like models, i.e. DETR [4], shows the effectiveness of combining hybrid convolution and transformer. More recently, Tolstikhin et al. [37] trigger a line of research to exploit the intrinsic merits of MLPs. Albeit the splendid performance in image classification, the potentials for regression still have not been dived into. Moreover, multi-head attention mechanism [38] has become the standard component in Transformer and intuitively allows for attending to varying parts of the token sequence by passing the input token through the same self-attention unit several times in parallel. Inspired by the merits of multi-head design, we propose the multi-granularity MLP to enrich token embeddings. Considering the drastic scale shifts in the crowd counting, we further refine the plain multi-head method by injecting tokenizations with distinct patch sizes before individual MLP head, which remarkably differs from the multiple attention heads with strictly same inputs and multi-level operations in CNN-based models [33,9].

Methodology
As shown in Figure 2, the proposed count-level framework (CrowdMLP) is comprised of pre-trained CNN frontend, multi-granularity MLP, and self-supervised proxy task.

Coarse-to-fine Token Streams
As aforementioned, global-range receptive fields are pivotal for count-level regression due to the lack of local constraints. To this end, we propose an effective global-oriented MLP regressor which tokenizes the crowd images by breaking it up into a sequence of patches with lower density levels. Apart from the global dependencies, limited training samples and drastic scale variation are prone to impel the model to collapse into trivial outcomes. Hence, a tokenization structure is designed for marrying the advantages of locally-focused features (pre-trained CNNs) and global-range relationships (MLPs) in a hybrid fashion. In specific, given a square input I ∈ R H×H×3 , where H × H is image resolution, we first reshape I into a sequence of raw patches of size 16×16, denoted as R r . Meanwhile, following the wide-spread algorithms [15] for crowd counting, we incorporate first ten layers of a VGG-16 pre-trained on ImageNet as the frontend to gather crude and low-level features that embrace the implicit spatical hints and the property of translation invariance. The extracted low-level features, with the resolution of H 8 × H 8 , go through a 1×1 convolution layer to shorten channel width and suppress model complexity. The resulted feature maps are denoted as C I .
To remedy the lack of multi-scale cues in MLPs-based models and increase the cardinality of the model, C I are split with patch sizes of 16×16, 8×8 and 4×4, respectively, leading to three sets of feature sequences (F r ). Then, F r and R r are linearly projected to construct four token streams } at identical token dimension D, yielding rich coarse-to-fine tokens for further capturing embeddings with global receptive fields at diverse granularity. The overall process can be defined as follows: where S denotes split and reshape operations, whereas LP is linear projection. 2D Dropout (D r ) with drop rate of 0.2 is employed for raw token sequence to make embeddings robust against noise by stochastically dropping a small set of patches before each epoch.

Multi-granularity MLP Regressor
The MLP-based counter is built upon multiple fundamental transformations, denoted as T mlp , and each of them consists of two plain MLPs (token mixing and channel mixing). Each kind of mixing operator involves two fully-connected layers, non-linear ReLU activation function, and batch normalization BN . T mlp can be written as follows: where f 1 and f 2 indicate two plain fully-connected layers, w 0 and w 1 the corresponding learnable parameters, X and Y the input and output embeddings, and F m the intermediate embedding vector. While token-and channel-mixing communicate information along spatial and channel directions, the interaction across different granularities is not yet considered. This would damage the establishment of complementary embeddings at diverse semantic levels and lead to one-sided long-range dependencies. To attenuate this limitation and mine token embeddings with coarse-to-fine global receptive fields in depth, we present a simple yet effective multi-granularity MLP structure. To the best of our knowledge, this is the first attempt to address the issues of scale/density changes through global-oriented multi-granularity MLP. By imposing this structure on the four input token streams T p = {T 1 , T 2 , T 3 , T 4 }, our model allows for assembling distinct granularities of token sequences covering varying head scales. Four MLP-based heads H 1 , H 2 , H 3 , H 4 with depth of 1 are imposed on T p according to the Eq.(2), and generate a pool of embeddings with multi-level global receptive fields, denoted as E 1 , E 2 , E 3 , E 4 . Sequentially, these higher-level embeddings are concatenated as a holistic embedding vectorÊ involving adequate semantics.Ê is then fed into a higher-capable MLP encoder M top with depth of 3 to refine representations and to further mix feature hints at dimensions of cardinality, spatial, and channel. Finally, a light-weigh MLPbased count head C top is plugged at the top of counter to predict the total count P I ∈ R 1×1 as a continuous and uncontrolled value. We denote the concatenation as and the overall procedure is depicted as:

Self-supervised Proxy Split-Counting
Recently, self-supervised learning is in vogue in computer vision, which has lead to scale on a mass of annotation-free training data and capture robust semantic information. This rule-based learning paradigm provides supervisory signals by pre-designing pivotal surrogate tasks. Towards the goals of decoupling the feature spatial hints and regularizing the learning procedure, we propose a novel proxy task for counting-specific regression, namely Split-Counting. It is designed with the consideration of enforcing the model to capture global-range spatial cues and regress complementary counts accurately. Specifically, we randomly select a rectangular region of stochastic resolution from input I to form a positive decoupling (I P ) and the complement region is referred as a negative decoupling (I N ). The generation of pairwise decouplings can be formulated as: where I = I N ∪ I P and I N ∩ I P = ∅. M is randomly-generated mask maps with values of 1 or 0, whereas × represents pixel-wise multiplication. The pairwise decouplings (I N , I P ) are taken as the input of our Split-Counting. For simplicity, we denote three lines of regressions as P I = M 1 (θ, I), P P = M 2 (θ, I P ), and P N = M 3 (θ, I N ), where θ indicates the shared parameters of CrowdMLP and P are the predicted counting results. By doing so, the intra relationship hidden in a single count value is explicitly decoupled without resorting to extra modules, and the missing spatial constraint is revived. Another key benefit is to reduce regression difficulty by implicitly learning an ensemble (M ) of sub-regressors.

Analysis on Split-Counting
To theoretically demonstrate the impacts of Split-Counting on lowering empirical error of the entire model, we denote the ensemble regressor asM and three implicit sub-regressors as M 1 , M 2 and M 3 . Hence, we haveM = 1 2 (M 1 + (M 2 + M 3 )). Following the analysis in DNCL [47] and bias-variance decomposition theory [2], given the target crowd count Y , the prediction error expectation is decomposed into the combination of bias "Bias", variance "V ar" and covariance "Cov" as follows: As for the average regression error of each individual regressor, it is straightforward to show: Then, the variance between sub-individuals and the ensemble model can be transformed as the following equation: Finally, the regression error of the ensemble modelM (see equation (5)) can be reformulated as the summation of all individual errors and a negative term: This indicates that the quadratic error of the ensemble regressor is theoretically guaranteed to be less than or equal to the average of the main baseline and the proxy regressors. Hence, the proposed Split-Counting is analytically proven to play a positive role in lowering empirical error of the entire model.

Objective Function
In our framework, we utilize L 1 loss between the predicted count P I and groundtruth count C gt as the pivotal supervisory signal: L C = |P I − C gt |. In addition, we derive a self-supervised loss (L SS ) and an integral loss (L I ) from the proxy task as regularization constraints; see Figure 2. Given the outputs of the selfsupervised Split-Counting, P P and P N , they are defined as: The objective function of our CrowdMLP is formulated as L = L C + 1 2 (L SS +L I ).

Datasets
Four widely-used benchmark datasets are utilized to verify the proposed method. It has a total of 1.51 million annotations on centroids of people and the number of crowd dispersing across crowd scenes ranges from 0 to 25,791. Although NWPU [41] is also a popular large-scale dataset, it does not provide the ground truths for testing samples. Hence, the performance on large-scale dataset is evaluated using JUH-CROWD++ here.

Implementation Details
The CrowdMLP is optimized using Adam algorithm with the initial learning rate of 1e-5. The mini-batch size is set to 12 and the learning rate is degraded by a MultiStepLR scheduler. The internal token dimension D is fixed as 256 in our experiments. For all datasets, we first reshape the raw images to keep the max side of 1024 and the other of 768. During the training phase, we randomly crop a batch of patches with the fixed size of 256×256 from the reshaped crowd scenes in an online manner. Stochastic flipping and random lighting with the probability of 0.5 are chosen as data augmentation operations. At the inference time, we design a sliding window strategy with the size of 256×256 to infer a set of sub-counts and their summation is computed as the final count prediction. Two commonly-used metrics, Mean Absolute Error (MAE) and Mean Square Error (MSE), are employed for performance evaluation: where N represents the number of testing images, C gt i is the ground-truth total number of people across the i th image and the C i indicates the corresponding predicted count value.

Comparison with State-of-the-art
As shown in Table 1, on ShanghaiTech Part A, Part B, and JHU-CROWD++ datasets, CrowdMLP outperforms the previous state-of-the-art weakly-trained approach (TransCrowd-GAP) significantly, with 19.7%, 24.7%, and 13.3% MSE reductions, respectively. On UCF-QNRF, the improvement is subtle with 3.2% lower MAE but 1.1% higher MSE. As a powerful count-level regressor, CrowdMLP also substantially minimizes the performance gap between count-level and location-level supervised approaches. On ShanghaiTech Part A, it is slightly behind the best fully-supervised P2PNet (57.828 vs. 52.74 on MAE and 84.412 vs. 85.06 on MSE), but outperforms other approaches. For example, when compared to the up-to-date BL algorithm, CrowdMLP brings 17.1% and 4.51% MAE reduction on ShanghaiTech Part A and Part B, respectively. Furthermore, on the large-scale JHU-CROWD++ dataset, which demands more arduous distribution learning, CrowdMLP performs the best among all algorithms that have performances reported to-date. This observation supports our hypothesis that count-level labels provide ample cues for counting, even though they do not explicitly provide spatial information.

Ablation Study
Ablation studies are conducted on ShanghaiTech Part A dataset to thoroughly investigate the effects of each component. For simplicity, the presented CrowdMLP is denoted as the Baseline.
The Effects of Coarse-to-fine Token Streams. We first ablate the effects of structuring token sequences at multiple levels of granularity by removing each sub-encoder. Table 2 shows that removing individual encoders from the pyramid incurs performance loss, which confirms the importance of the devised multigranularity MLP.
Using Different Learning Paradigms. Next, we compare the regression capability of MLPs-based structure against CNNs and Transformer paradigms.   Table 2. Ablation study on multi-granularity structure. The first row removes the token sequence for raw image patches from the coarse-to-fine token streams, whereas each of the following 3 rows removes those for feature patches of corresponding sizes. Performance degradation is observed when any token sequence is removed.
MLPs transformations in the baseline with transformer in ViT that have the same depth and dimension. Table 3 shows that Crowd-CNN delivers a seriously degraded performance, which may be attributed to the fact that CNNs unit is incapable of modelling long-range receptive fields while focusing on locality [45]. The lower errors achieved by Crowd-Transformer and CrowdMLP verify the importance of globalrange receptive fields for count-level regressors. Albeit the acceptable accuracy, Crowd-Transformer suffers from increased model scale (36.43M vs. 26.63M) and high computational complexity. Our observation is consistent with the recent S 2 -MLPv2 [46] for image classification.
The Effect of Proxy Split-Counting. To evaluate the impacts of the proxy task, we also train a variant of CrowdMLP without the self-supervised Split-Counting on ShanghaiTech Part A. The comparison shows that Split-  Counting helps to bring 0.55% improvement on MAE and 0.32% improvement on MSE. This confirms our theoretical analysis that the proxy task is beneficial for implicitly decoupling spatial constraints and lowering the empirical error.

Visualization and Analysis
To discern learned features in depth under the count-level supervisions and investigate whether or not CrowdMLP, trained without any location signals, is aware of scale variation, density shift, and spatial semantics, we visualize the embeddings using T-SNE technique and present a typical inference example with drastic scale/density/semantics changes across the whole image. The clustering results in Figures 3 and 5 show that CrowdMLP learns discriminative features with underlying repetitive patterns for diverse scenes. Photos with similar properties appear to be grouped into same clusters. For example, on the leftmost (blue) cluster, we have street photos with large scale variation where crowd only show up at the bottom of the images, whereas on the rightmost (red) cluster, we have overhead views with uniformly distributed crowd. Simultaneously, Figure 4 exhibits the behaviour of sliding-window strategy with the size of 256×256 at the testing phase. It can be observed that, although lacking location-level prior, our weakly-supervised CrowdMLP can distinguish spatial semantics (the background and foreground people). For the common issues of scale variations and density shifts, our model seems to work pretty well and predict people number precisely under the variation trend of scale and density; see red and blue dotted regions in the Figure 4.

Conclusion
In this paper, a novel weakly-supervised CrowdMLP is presented to mitigate the burden of the labour-intensive location-level annotations and learn to directly regress crowd counts using only count-level labels. In specific, the multigranularity MLP regressor are creatively designed to capture global receptive fields at varying levels of granularity. Besides, a promising proxy task, namely split-counting, is introduced for regularizing the learning procedure of the crowd regressor, implicitly decoupling spatial constraints, and avoiding performance degradation. The split-counting also brings potentials for exploiting massive unlabelled or cross-domain data. Extensive experiments have demonstrated the effectiveness of the presented method. For future work, we plan to explore the potential of CrowdMLP for cross-domain counting and/or domain adaptation.  5. Embeddings cluster visualization of testing images in ShanghaiTech Part A using T-SNE technique. Each image is shown as a colour dot, where the colour represents the crowd number/densities. While the images are clustered primarily by scene features (see Figure 3), high-density images tend to group together (green cluster). The zoomed-in view depicts corresponding real scenes in green cluster.