{'fraction_non_alphanumeric': 0.03179703611337586, 'fraction_numerical': 0.011414320656083641, 'mean_word_length': 4.854014598540146, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Flow Base Bi-path Network for Cross-scene Video Crowd Understanding in Aerial View

Drones shooting can be applied in dynamic traffic monitoring, object detecting and tracking, and other vision tasks. The variability of the shooting location adds some intractable challenges to these missions, such as varying scale, unstable exposure, and scene migration. In this paper, we strive to tackle the above challenges and automatically understand the crowd from the visual data collected from drones. First, to alleviate the background noise generated in cross-scene testing, a double-stream crowd counting model is proposed, which extracts optical flow and frame difference information as an additional branch. Besides, to improve the model's generalization ability at different scales and time, we randomly combine a variety of data transformation methods to simulate some unseen environments. To tackle the crowd density estimation problem under extreme dark environments, we introduce synthetic data generated by game Grand Theft Auto V(GTAV). Experiment results show the effectiveness of the virtual data. Our method wins the challenge with a mean absolute error (MAE) of 12.70. Moreover, a comprehensive ablation study is conducted to explore each component's contribution.


Introduction
Video analysis [2,46] has become an increasingly important part of computer vision, which involves a wide range of fields, such as object detection and tracking [21,1], crowd segmentation [18], density estimation and localization [15,37], and group detection [31]. Among them, a new challenge that understands the crowd from a drone perspective has recently attracted lots of attention. Compared with the density estimation in the ground scenes, unmanned aerial vehicles (UAVs) have a broader surveillance scope. It can play a more crucial role in public safety and urban management. The scale of the datasets limited the development of crowd counting tasks over the past few years. However, it has been booming driven by the release of a series of large-scale datasets recently, such as UCF-QNRF [15], JHU-Crowd [29], NWPU-Crowd [32]. Unlike the counting task from the ground perspective, there are extra challenges in the UAV scenarios. In this paper, we work on density and crowd estimation for the DroneCrowd dataset [37]. As illustrated in Fig. 1, we found that the difficulties of UAV's task of counting people lie in the following three aspects: -Cross-scenarios. The training set and validation set of the DroneCrowd dataset are divided according to the scenario, which means that it is a crossscene crowd counting task. The test set scenarios are almost unknown to the model, which increases the difficulty of the task. -Scale variation. The flying height of drones is constantly changing during data collection, which significantly affects the scales of objects. The diversity of scale increases the difficulty of model fitting. -Night challenge. There are some extremely dark scenes in the test set, but there are no scenes with similar illumination conditions in the training set. These night scenarios have a significant impact on the test results.
The biggest problem of cross-scene attributes is that it has serious background noise in testing unknown scenes, which is impossible to be tackled directly by existing counting models. The effective measure is to reduce background noise by enhancing the model's ability to distinguish between background and moving objects. Note that the DroneCrowd dataset is composed of video sequences. We can extract the optical flow information of the video to achieve this purpose. Therefore, in our counting framework, an additional branch is designed to input optical flow information to enhance the feature extraction of moving objects. Besides, to improve the generalization ability of the model for the scale variables, multi-scale images are transferred to simulate the scenes at different heights. Finally, to alleviate the large test deviation caused by low illumination scenes, a variety of image brightness adjustment algorithms and synthetic data are used to simulate different illumination conditions in the training sequences. All three ideas contribute to the competitive performance, which we will explore in detail in section 4.
As a summary, we propose a bi-path framework based on RGB image and optical flow information to improve the model's ability to weaken the background noise for unknown scenes and exploit multi-scale transformation and multi-luminance adjustment algorithms to make data augmentation for reducing the adverse effects of scale and illumination. Besides, we generate synthetic data under dark scenes, which expands training data using low-cost methods and improves the performance of the module in extreme scenarios.

Related Work
Scale-aware Crowd Counting. With the development of crowd counting, it is found that scale change has the greatest negative impact among the influencing factors (occlusion, scale, and viewpoint, etc.). Many crowd counting algorithms focus on scale variability in recent years. Zhang et al. [47] propose a three-columns network with different kernels for scale perception in 2016. Onoro-Rubio et al. [25] introduce a Hydra CNN with three-columns, where each column is fed by a patch from the same image with a different scale. Wu et al. [38] develop a powerful multi-column scale-aware CNN with an adaptation module to fuse the sparse and congested columns. In the same year, Adaptively Fusing Predictions (AFP) [17] generates a density map by fusing the attention map and intermediary density map in each column. ic-CNN [27] generates a high-resolution density map via passing the feature and predict map from the low-resolution CNN to the high-resolution CNN. SDA-MCNN [40], a scale-distribution-aware neural network that resolves scale change by processing a crowd image with multiple Convolutional Neural Network columns and minimizing the per-scale loss. Hossain et al. [13] employs a scale-aware attention network, where each column is weighted with the output of a global scale attention network and local scale attention network. ACM-CNN [50], Adaptive Capacity Multi-scale convolutional neural networks assign different capacities to different portions of the input based on three modules: a coarse network, a fine network, and a smooth network. Except for multi-column scale-aware architecture, some works focus on the single-column scale-aware CNN, such as SANet [3], SaCNN [45]. To combine the multi-column and single-column scale-aware CNN, CSRNet [20], CAN [23] and FPNCC [4] develop a model containing multiple paths only in several part of the networks. In 2020, SCAN [41], SRN+PS [7], SRF-Net [5], and ASNet [16] also explore the scale-aware to improve the counting performance.
Cross-scene/domain Crowd Counting. Due to the laborious data annotation required of crowd datasets, cross-scene, and cross-domain crowd counting attract researchers' attention in recent years. In this task, the model is trained on a labeled dataset and then adapted to an unseen scene. Authors in [43] establish the earliest cross-scene dataset, which includes 103 scenes for training and the remaining five scenes for testing, a total of 3980 images. In [22], they propose a Fully Convolutional Neural Network(FCN) and a weighted adaptive human Gaussian model for person detection and then apply it to the new scene with few labeled data. DA-ELM [39], a counting model based on domain adaptationextreme learning machine, counts the people in a new scene with only a half of the training samples compared with counting without domain adaptation. In [14], they propose a one-shot learning approach for learning how to adapt to a target scene using one labeled example. In [28], they apply the MAML [8] to learn scene adaptive crowd counting with few-shot learning. Inspired by the synthetic data can automatically label as the source domain, Wang et al. [33] launch a large-scale synthetic dataset to pre-train a model and adapt it to real-world datasets by a fine-tune operation. Except fine-tuning, they also complete counting without any real-world labeled information by using the Cycle GAN [48] and SE Cycle GAN [33] to generate realistic images. Recently, several efforts have been made to follow them. DACC [9], a method for domain adaptation based on image translation and Gaussian-prior reconstruction, achieves a better performance on several mainstream datasets. At the same time, some works [10,12,24,35] extract domain invariant features based on adversarial learning. In [34], the authors propose a Neuron Linear Transformation (NLT) method that models domain differences and uses few labeled target data to train the domain shift parameters. Experimental results show that cross-domain crowd counting can alleviate the problem of data annotation to some extent.

Proposed Method
A perfect crowd counter should have stable and excellent performance in practical applications. However, as mentioned above, the actual application scenarios are complex and changeable, and the training data is limited. Therefore, in our proposed method, we promote the model's performance from the following two aspects: -Improve the feature extraction and representation ability of the crowd count network through better network structure design. Therefore, it boosts the cross-scene stability and accuracy of the model. -Try to mine the information contained in the limited training data. This includes using different kinds of data transform methods to manually create unseen verity scenario from existing data, and make use of the joint information between adjacent frames while reading continuous sequential data.
The remaining paragraphs follow these two aspects to introduce the work done by the proposed method.

Network Design
As shown in Figure 2, we design a bi-path flow-based crowd counting network to make use of inter-frame information together with the input RGB image. Each pathway consists of one powerful feature extractor and several convolutional layers as the decoder. The encoder is the the first three layers of the newly proposed ResNeSt [44] with weights pre-trained on the ImageNet [6] dataset. The crowd under the aerial view is small, and the building area is large. To better analyze the relationship between humans and background, we follow the idea of CSRNet [20], the decoder contains six dilated convolutions, which reduces the number of channels while parsing the semantic information of the input features. Atrous convolution layers enlarge the convolution kernel's reception field, which enriches spatial contextual information inside the features. The decoder outputs feature vectors of 1/8 size of the original input with 64 channels. The image branch and the flow branch are combined immediately after the decoders. Besides the dilated convolutions, we introduce spatial-wise attention module (SAM) and channel-wise attention module(CAM) from [11] to enhance large-range dependencies on the spatial and channel dimensions. The detailed network structure is given in Table 1. "ResNeSt50 Conv1 c64" means this layer has the same structure as the Conv1 layer of ResNeSt50, and its output is a 64 channel tensor. "k(3,3)-c512-s1-d2-R" represents convolutional layer with kernel size 3 × 3,output channel 512, stride 1, dilation rate 2, and followed by a ReLU activation layer.
The image stream accepts the three-channel RGB color image as input, while the flow stream also takes a three-channel input with the same size as the original image.
Generally speaking, the optical flow is a two-channel vector, and each channel stores the optical flow information of the image along the horizontal axis and vertical axis separately. We name this form of optical flow f x and f y . It can also be transformed into polar coordinates. Under the polar system, two transformed channels represent the polar radius and polar angle, respectively. Researchers Table 1. Network structure of the proposed method.
usually take these two channels as the first two channels in the HSV color space, and then convert them back to the RGB space for visualization, so we name them f h and f s here. The flow stream takes a three-channel vector as input, here we fill the third dimension of the input data with the frame difference vector of the two adjacent frames. Which means take frame t subtracts frame t+1 and get f sub . Different forms of optical flow contain different information, leading to distant results to the network's training. We have tried several combinations; the specific experimental results and analysis are given in Section 4.4. Figure 3 visualizes different forms of optical flow data. During the visualization and analysis process, we find that there may be a slight jitter in the process of data acquisition, which introduces a large area of noise to the optical flow information. Therefore, we add a simple threshold filtering in the actual use, which effectively suppresses part of the noise.

Data Argumentation
Apart from the network structure design, it is also significant to effectively use existing data.
The training data only contains limited scenes. By observing, analyzing, and summarizing the status of the existing data, we speculate the scenarios that may be encountered during the actual use. We manually augment the existing data of the scene according to the actual situation. During training, numerous transform methods were used to generate new data, improving the model's generalization ability. Instead of original data group (I, where subscript {C, F, G, S} represents crop, flip, gamma correction, scale change respectively. The following part of this subsection introduces all data transformation methods we use according to the characteristics of the data. Figure 4 gives examples of several transformations. Scene Change: Depending on the drone's shooting location, different scenes may vary considerably, whether it is the street or the park. Their style and direction will change. To cope with the scenario's variability, we make random transforms to each group of input data during the training process, which includes random crop into fixed size (576 × 576 here), random flip (up-down and left-right). Of course, those transformations perform the same processing within each group of data (I, F, D) used for training. We name those transformations T {C,F } (·).
Extreme Light Variation: All the training data we have are taken under good light conditions, but the data used in the test and the real world may not always have such good light conditions. There will be overexposure or underexposure. So before we send the data into the model, we carry out random gamma correction on the RGB image with a certain probability (here is 0.5). The range of Gamma values is from 0.4 to 2 to simulate excessive brightness and insufficient brightness. This transformation only applies on the RGB images, and we call it T G (·).
Scale Diversity: When the UAV collects data in different states, it may fly under a low or high altitude, seriously affecting the size of the crowd in the picture and ultimately affecting the model's analysis results. To further enhance the model's generalization ability, we force it to deal with images collected at various heights. We change the scale of the images before cropping, which means randomly enlarge or shrink the image, and then crop the new one for training. This transformation works on all three input data together. We name it T S (·).

Synthetic Data
The cost of collecting and labeling real scene data is very high. Moreover, in general, the real scene is limited, and the diversity is weak. The standard data enhancement method can only alleviate this problem to a certain extent. To better increase the diversity of training data, we refer to [33]'s method and generate some synthetic data to assist training. Figure 5 demonstrates some of the synthetic data that we generated.

Dataset
VisDrone crowd count dataset (DroneCrowd) [37] is a remote crowd counting dataset with manual labels and 1080p RGB images collected by drones. It consists of 112 sequences, of which 82 contains publicly reached point labels, and the rest of the sequences can only be tested by uploading the results to the official challenge [49] server. Therefore, we randomly divide the labeled 82 sequences into 75 for training and 7 for validation. Our model achieves MAE 12.7 in the test server and wins the challenge 1 . Because there is a limit on the number of test times, part of our following experimental results are based on the validation set.  [37]. ** MAE from official test server [36]. Data collected on 02:41:20, July 15, 2020, GMT.

Implementation Details
All experiments are build based on one NVIDIA TITAN RTX GPU and Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz under Ubuntu 18.04 LTS operating system and PyTorch [26] framework. The ResNeSt50 encoder is initialized with the weights pre-trained on the ImageNet dataset, all other convolutional layers are initialized with normally distributed weights and zero bias. The learning rate is set to 10 −5 .

Performance Measures
Researchers use MAE and mean squared error (MSE) to measure the differences between the predicted density map and the ground truth. The definition of MAE and MSE are given by following equations: where H and W are the height and width of the test density map, z i,j is the ground truth pixel value of location (i, j),ẑ i,j is the corresponding one of the predicted density map. The official challenge test server only provides MAE values. As for our experimental analysis, both MAE and MSE are calculated on the validation set. Table 2 shows MAE values on the test set of several methods from the challenge [49] and the official paper [37]. Our method outperforms all those methods and wins the challenge with MAE 12.70 1 .

Ablation Studies
Influence of Flow Stream: To verify whether the flow stream helps improving the network, we set up experiments to train the network without the flow stream. We use different methods to generate the optical flow; the first one is based on deep learning [30], while the second one is based on dense inverse search  [19]. We abbreviated them as PWC and DIS, respectively. We also take different combinations of [f x , f y , f h , f s , f sub ] into consideration. Experiment results are shown in Table 3. Due to time limitations, we only train 30 epochs under each setting and take the best result. It can be seen that the introduction of the flow stream improves the generalization ability of the model, and its type and composition will also affect the final result. Optical flow generated by DIS, with combination method [f s , f v , f sub ] achieves the best.
Influence of Gamma Correction: There are some extremely dark scenes in the test set, but there is no scene with such low light conditions in the training set. These night scenarios have a significant impact on the test results. The validation set can not verify the effectiveness of gamma correction. We submit results with/without gamma correction to the official test server. The test results are given in Table 4. It can be seen that random brightness transformation is performed on the input image during training to increase the diversity of the lighting conditions in the training data, which effectively improves the model's performance in the test set with low light environments.
Influence Influence of Synthetic Data: We use the synthetic data alone to train a new model to predict the night scene, and then use the original model to predict the daytime scene. Table 5 also shows the results of mixing the two models. It can be seen that the synthetic data has brought a significant improvement to the final result. Figure 6 shows the visualization of the predict density map under

Conclusion
This paper proposes a bi-path optical flow-based crowd counter network, together with several data argumentation methods specially designed for remote crowd count tasks. To be specific, two separate ResNeSt50 based encoder-decoder stream extract feature vectors from RGB images and 3-channel optical flow tensors. During the training phase, data argumentation operations like crop, random flip, gamma correction, scale variation are applied to create nonexistent scenes from the train set. Synthetic data are generated for better performance under extreme dark scenarios. With the combination of network design, data expansion and synthetic data, we finally win the official competition 1 . In addition, we designed sufficient experiments to verify and analyze the effectiveness of network structure design and different data argumentation methods. In the future, we will explore effective network design and data argumentation methods for remote crowd count task.