{'fraction_non_alphanumeric': 0.038778162911611784, 'fraction_numerical': 0.011611785095320624, 'mean_word_length': 5.214593430263866, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
E 2 TAD: An Energy-Efﬁcient Tracking-based Action Detector

Video action detection (spatio-temporal action localiza-tion) is usually the starting point for human-centric intelligent analysis of videos nowadays. It has high practi-cal impacts for many applications across robotics, security, healthcare, etc . The two-stage paradigm of Faster R-CNN in object detection inspires a standard paradigm of video action detection, i.e ., ﬁrstly generating person proposals and then classifying their actions. However, none of the existing solutions could provide ﬁne-grained action detection to the “who-when-where-what” level. This paper presents a tracking-based solution to accurately and efﬁciently localize predeﬁned key actions spatially (by predicting the associated target IDs and locations) and temporally (by predicting the time in exact frame indices). This solution won the ﬁrst place in the UAV-Video Track of 2021 Low-Power Computer Vision Challenge (LPCVC). The code is available at https://github.com/VITA-Group/21LPCV-UAV-Solution.


Introduction
The 2021 Low-Power Computer Vision Challenge (LPCVC) UAV-Video Track [24,25] requires contestants to spatio-temporally localize the key action of ball-catching from videos captured by drones. The evaluation metric takes both accuracy and efficiency into account. The solution is expected to precisely detect the key action temporally (by predicting frame indices) and spatially (by predicting associated persons' and balls' IDs), using as low energy consumption as possible. This challenge has three unique challenges: lack of training data, robustness, and efficiency.
Lack of Training Data: Unlike the ubiquitous "person" object found in benchmarks for detection, segmentation, pose estimation, or tracking tasks, the "ball" object could Efficiency: Detecting target persons/balls, extracting their ReID features, and localizing key action spatiotemporally are computationally intensive. Considering the limited computation power and memory capacity on Pi 3B+, running these modules on Pi 3B+ in real-time would be difficult. This paper presents our submitted solution, dubbed Energy-Efficient Tracking-based Action Detector (E 2 TAD). We present the following: • a tracking-based vision system to spatio-temporally localize key action from videos. It has three core components: ball-person detection, deep association, and action detection; • a harmonization-aware image composition module to generate synthetic but realistic ball with homogeneous color on pedestrian datasets; • two adaptive inference strategies and a cache-friendly pipeline to save energy cost; • a shape-texture debiased training and a domain-invariant adversarial training to improve the robustness of detection and ReID feature; • a morphology-based and learning-free action detector that only depends on bounding box trajectories to localize key actions spatio-temporally.
This solution won the 1st place in the UAV-Video Track of 2021 Low-Power Computer Vision Challenge (LPCVC).

Video Action Detection
Video action detection (a.k.a, spatio-temporal video action localization) requires localizing persons and recognizing their actions in space and time from video sequences.
STEP [72] is a progressive learning framework that consists of spatial refinement and temporal extension, which iteratively refines the coarse cuboid proposals toward action locations and incorporates longer-range temporal information to improve action classification. STAGE [56] learns relations between actors and objects by self-attention operations over a spatio-temporal graph representation of the video. LFB [64] introduces a long-term feature bank that stores long-term supportive memory extracted over the entire video. SlowFast [13] has a slow pathway operating at low frame rates to capture spatial semantic information, and a lightweight fast pathway operating at high frame rates to capture rapid motion better. Recently, Context-Aware RCNN [65] enlarges the resolution of small actor boxes by cropping and resizing instead of RoI-Pooling, also extracting scene context information aided by LFB. AIA [54] leverages an interaction aggregation structure and an asynchronous memory update algorithm to efficiently model very long-term interaction dynamics. ACAR-Net [43] learns to reason high-order relations (aka actorcontext-actor relations) with a actor-context feature bank that preserves spatial contexts. Collaborative memory [71] shares the long-range context among sampled clips in a computation lightweight and memory-efficient way.
Re-Identification: A standard closed-world Re-ID system has three main components: feature representation learning, deep metric learning, and rank optimization. Feature representation learning includes person-level global feature [81], part-level local features [52,73,79], and auxiliary information (e.g., semantic attributes, viewpoint, and domain) enhanced features [4,51,74]. Deep metric learning aims to design loss function to guide the feature representation learning. Verification loss [12,31,57], identity loss [81], and triplet loss [5,22,75] are the three widely-used loss function. Ranking optimization improves the retrieval performance in the testing stage.
Tracking: As the standard approach in multi-object tracking (MOT), Tracking-by-detection [1,62,63,77,78,86] has four stages: detection, feature extraction/motion prediction, affinity, and association. Given the raw frames, a typical workflow starts with an object detector that returns bounding boxes, a feature extractor and a motion predictor that give appearance and motion cues, an affinity calculator that computes the confidence of two objects belonging to the same target, and an object associator that assigns IDs to detected objects (if not excluded). MOT has two modes [8]: batch and online. Since exploiting global information often results in better tracking, batch tracking uses future information when assigning IDs for detected objects in a certain frame. In comparison, online tracking, often running in real-time speed, cannot fix past errors using future information.

Efficient Neural Network
Pruning: Pruning explores the redundancy in the model weights by removing the uncritical weights. It could be done in a one-shot way or an iterative way. One-shot methods prune weights at once based on some importance metric and require finetuning as accuracy loss compensation. Iterative methods prune weights during optimization following a progressively increasing sparsity rate. From a granularity perspective, ordered in decreased sparsity rate, pruning could be done element-wise, channel-wise, column-wise, filter-wise, or layer-wise. Channel-wise pruners include Network Slimming [41], AMC [19], and ADMM [76]. Filter-wise pruners include FPGM [20], magnitude-based 1 pruner [29], AGP [88], Network Trimming [23], and Taylor FO [42]. Layer-wise pruners include NetAdapt [70]. AutoCompress [38] supports combined filter and column pruning.
Quantization: Quantization refers to compressing models by reducing the number of bits required to represent weights or activations, which can reduce the computations and the inference time. From granularity perspective, quantization could be done in layer-wise, group-wise, channelwise (the standard method used for quantizing convolutional kernels), or sub-channel-wise. Deep neural net-works' major numerical representation bitwidth is 32-bit float (FP32). Many research works have demonstrated that weights and activations can be represented using 8-bit integers [27,85] without significant loss in accuracy. Lower bit-widths quantization, such as 4/2/1 bits [9,84], is an active field of research.
Dynamic Neural Networks: As opposed to static network's static computation graph and parameters, dynamic network, whose primary advantage is efficiency, can adapt its structures or parameters to the input during inference. Dynamic network allocates computation resources on-demand during inference, by selectively activating components (e.g., channels, layers, or stages) conditioned on the input [6,26,33,50,69]. For example, a dynamic network spends less computation on easy samples or less informative spatial areas/temporal locations of an input. For image [32,60] or video-related [30,59] tasks, sample-wise, spatial-wise, or temporal-wise adaptive inference could be conducted by formulating the recognition or detection task as a sequential decision problem and allowing early exiting during inference.
Algorithm and Hardware Codesign: Algorithm and hardware codesign enables deep neural network cross the performance wall when parallelism and data reuse are exhausted. Efficient data operation and model compression are the two widely approaches. Stochastic data operation(e.g., data mapping, matrix decomposition) increases hardware utilization and accelerates computation on hardware [7,11]. Compressed architecture (e.g., quantization and pruning) on accelerators ultra-highly speeds up inference time with negligible accuracy loss [21,28]. Ren et al. [48] proposed the first algorithm-hardware co-designing framework with a combination of weight pruning and quantization, to reduce performance overhead due to irregular sparsity. Wang et al. [61] extended Roofline model into deep learning area and incorporated computational complexity and run-time into models, which made it possible to analyze code performance for deep learning applications systematically.

System Design
The proposed Energy-Efficient Tracking-based Action Detector (E 2 TAD) has three core components: ball-person detection, deep association, and action detection.
Ball-Person Detection: Existing datasets are available for detecting pedestrians. However, there is no ball-related dataset in the literature. We adopts harmonization-aware  image compositing [37] to generate synthetic but realistic balls on pedestrian datasets, e.g., VisDrone [3], COCO ball-person subset, PANDA [58]. To address the efficiency, we design a lightweight model based on YOLOv3 and YOLOv5. Among the three branches that detect objects in multi-scales (e.g., small, medium, and large), the middle one is retained to detect medium-scale objects. Our proposed two YOLO variants are dubbed YOLO-MobileV1 and YOLO-MobileV2. Inherited from YOLOv5, YOLO-MobileV1 modifies PANet in YOLOv5 and cuts off one upsample module to directly output the medium-size feature map. As a descendant of YOLOv3, YOLO-MobileV2 replaces the convolution with depth-wise convolution and removes upsampling modules. The architecture details of YOLO-MobileV1 and YOLO-MobileV2 are shown in Fig. 1.
Deep Association: Shown in Fig. 2, this module tags the detected persons and balls with their corresponding identity labels. First, an identity-aware feature extractor (e.g., ResNet-18 trained on person ReID task) is adopted to obtain the feature embeddings of the detected persons/balls, given the bounding boxes together with the raw image. Second, Deep Association maintains the person tracklets and ball tracklets in a gallery. Each tracklet is a queue that stores ReID features of the latest K tracks for the corresponding person/ball. Last, using the ReID feature of newly detected persons/balls as queries and defining the cost of matching a query with a specific tracklet in the gallery as the distance between their ReID features, Deep Association formulates the identity tagging as a minimum cost assignment problem that can be solved in polynomial time by Hungarian algorithm. To reduce association errors, any distance larger than a predefined threshold is marked as ∞ in the cost matrix to rule out the possibility of assignment.
Video Action Detection: Free of learnable parameters, this module in Fig. 3 only depends on bounding box (bbox) trajectories to spatio-temporally localize key actions (e.g., catch and throw), thus extremely energy-efficient. A ball has two states: collision or non-collision. Collision happens when the center of one ball's bbox falls into one person's bbox. For each ball, the method plots its collision history with other persons as a gated signal spanning from 0 to T , where T is the total number of frames. The start and the end of any connected part in the gated signal signifies catch action and throw action, respectively.

Efficiency
Pruning: Pruning could be done in an unstructured or structured way. Unstructured pruning removes individual weights at the kernel level, while structured pruning removes groups of weight connections such as channels or layers. Unstructured pruning leads to higher sparsity but requires specific hardware support for acceleration, thus not applicable to Raspberry Pi.
Quantization: There are two types of quantization: post-training quantization and quantization-aware training (QAT) 1 . Unlike post-training quantization, QAT supports concurrent training and quantization. Consequently, such a simulation of quantization errors during training gives minimal accuracy loss if the representation bitwidth is converted from FP32 to INT8.
Adaptive Inference: Considering video's temporal coherency and key action's infrequent occurrences, we are motivated to propose two adaptive inference strategies in Fig. 4: Activity Region Cropping (ARC) and Collision Inspection (CI). Given the observation that persons and balls' locations not changing abruptly since videos are temporally coherent, ARC predicts the next frame's activity region (Î t+1 ) based on the current frame's bboxes' coordinates (B t ) and crops out the non-activity region after adding some residual space (∆x and ∆y), whose details are shown To improve the temporal locality of ReID feature extraction, crop records and box records are used since the layerwise computation could be fit in the L1 cache (32KB), and the weights after pruning and quantization could be fit in the L2 cache (512KB). To improve the spatial and temporal locality of video decoding, an image queue is used so that temporally consecutive frames could be decoded without interruption 2 . Finally, E 2 TAD's pipeline is turned from memory-access intensive to computation-intensive Since the computation cost is fixed for a given video, the latency could be significantly reduced due to less memory access.

Robustness
Detection: Although shape and texture are two critical and complementary cues for object recognition, CNNlearned features are biased towards either shape or texture, depending on the training dataset. For example, the proposed YOLO-MobileV1 and V2 frequently mistake homogeneous color patches for balls. To alleviate the bias towards texture, we propose a texture-shape debiased training strategy by compositing homogeneous color patches as hard negative samples on the training images. Moreover, we propose a geometry-aware circular anchor that takes the circle shape of balls into account to detect them better.  ReID Feature: The two major challenges to robust ReID feature learning are various camera views and occlusion. Therefore, we are motivated to propose the following two solutions.
(1) Occlusion-Aware Data Augmentation: Ballperson occlusion is mimicked by a circular binary mask overlayed on the person images. We also use pseudo masks predicted by a pretrained person segmentation model to extract the person foreground and overlay it on the target person image to simulate person-person occlusion. Both strategies are used as data augmentation to train the ReID model.
(2) Domain-Invariant Adversarial Feature Learning: Given training images X , the goal is to learn a ReID feature embedding that is discriminative with different identities while invariant under different camera views. The goal can be mathematically expressed as a minimax game be-low [66][67][68]82]: Here, the f e (X ) is the extracted ReID feature embedding from X . f id and f v predict identity labels and view labels from f e (X ), respectively. L tr and L ce stands for triplet loss and cross-entropy loss. Y id and Y v symbolizes identity labels and view labels. λ is a coefficient used to balance the two losses. More details are shown in Fig. 6.
Action: As the downstream task, action detection is vulnerable to errors propagated from the upstream tasks (e.g., detection and association). In detection errors, falsenegative cases may owe to occlusion, and false-positive cases could result from patches with homogeneous colors. Association errors arise from non-discriminative ReID features. Inspired by morphology-based denoising [44], we propose a heuristic approach in Fig. 3 to eliminate the preceding errors from the upstream tasks. As prior, θ is the minimum flying time for any ball. Given the collision history with different persons as a gated signal for each ball, we first dilate it by θ/2. Then, we vote in the connected parts and select the majority as the associated person holding the ball, so that the preceding detection and association errors are eliminated. Last, the dilated gated signal is eroded by θ/2. Note that each box entry in "Box Records" is ordered by its corresponding person/ball ID.

Identity Classifier
View Classifier

Feature Extractor
Feature Embeddings

ReID Images
Identity Labels View Labels Figure 6. Proposed adversarial training in minimax game to learn robust ReID feature. After training, the identity classifier and the view classifier are discarded.

LPCVC21
. Among the five sample videos, the two most challenging videos 7p3b and 5p5b are reserved for testing and dubbed as LPCVC21-test; the remaining three videos 5p4b, 5p2b, and 4p1b are used for training and dubbed as LPCVC21-train. The bounding boxes for persons are refined on the pseudo labels produced by a pretrained person detector and the bounding boxes for ball are manually labeled at every 15 frames. To increase the training set size, we mix PANDA [58], VisDrone pedestrian subset [3], and COCO ball-person subset [36] with LPCVC21-train to obtain a large-scale mixed dataset for ball-person detection, dubbed as MixDet-train. MixDet-test is the testing set of LPCV21, namely LPCVC21-test.
We adopt the COCO evaluation protocol's mAP 0.5 and mAP 0.5:0.95 to evaluate detection performance. Ablation Studies: 1. Harmonization-Aware Image Composition. Due to the lack of training data for the ball class, we propose a harmonization-aware image composition method to embed synthetic balls into images randomly selected from VisDrone and PANDA. Fig. 7 shows the two steps of harmonization-aware image composition. First, synthetic ball images are "copy-pasted" on background image guided by generated randomly-placed mask. Second, composited images are harmonized through Rain-Net [37] by taking the lighting and background into account. 2. Shape-Texture Debiased Training. To alleviate the ball-detection's bias towards texture, we add randomshaped patches with homogeneous texture, which serve as negative samples to detect balls. Debiased training images are showed in Fig. 8. 3. Geometry-Aware Anchor. Tra- Figure 8. Shape-texture debiased training samples. Random patches with homogeneous color are marked by red rectangles. Figure 9. Samples of geometry-aware anchor for "ball" class. The left shows traditional anchors and the right shows geometry-aware anchor.
ditional rectangular bounding box (R-box) is not suitable for ball detection especially for heavily-occluded ball. The background information included in R-box introduces unexpected noise for ReID feature extraction. Considering the geometry difference between persons and balls, we use traditional rectangular bounding box (R-box) to detection persons and propose circular bounding box (C-box) to detect balls. Fig. 9 shows samples of different anchors. We conduct ablation studies of the proposed three methods in Tab. 1. Image-composition significantly improves detection performance. In addition, debiased training and circular anchor significantly improve both precision and recall, which shows that our proposed methods can effectively reduce false negatives (FN) and false positives (FP) for ball detection.  For each query, we calculate the area under the Precision-Recall curve, known as average precision (AP). Then, the mean value of APs of all queries, i.e., mAP, is calculated. Table 3. Accuracy of different one-shot and iterative pruners under different sparsity rates.

Video Action Detection
Datasets and Evaluation Protocols: We evaluate the video action detection performance on the two reserved videos for object detection testing, namely 7p3b and 5p5b in LPCVC21-test.
The accuracy metric measure how well the action detector spatio-temporally predicts the ball-catching action. A True Positive (TP) predicts person and ball IDs correctly and the frame index within ±10 frames as a tolerance threshold. False positives and false negatives may arise from object detection and association errors. Results and Analyses: ReID↑ means robust ReID feature extractor improved by the proposed occlusion-aware data augmentation and domain-invariant adversarial feature learning. Det↑ means robust ball detector improved by the proposed texture-shape debiased training. Action↑ means robust action detector improved by the proposed morphology-based denoising to eliminate preceding association and detection errors. AdpInf↓ means the two adaptive inference strategies implemented with an Activity Region Cropper and a Collision Inspector. Pipeline↓ means the cache-friendly pipeline that trades memory for less data movement cost. Tab. 5 presents the ablation study results, from which we can draw a conclusion that incrementally adding robust ReID features, robust ball detections, robust action detections, adaptive inference, and the cache-friendly pipeline could consistently improve the accuracy-efficiency tradeoff. Performance on LPCVC's Dry-run and Test Set Tab. 6 gives the performance summary of the top 4 winners' solution on the dry-run and test set. Our proposed E 2 TAD has a minimal score loss of 0.096. The improved robustness of ball-person detector, ReID feature embedding, and action detector may account for it.

Conclusion
This paper is a tech report for our submitted solution to the UAV-Video Track of LPCVC-2021. Our proposed Energy-Efficient Tracking-based Action Detector (E 2 TAD) is a tracking-based vision system that can spatio-temporally localize key action from videos. E 2 TAD has three core components: ball-person detection, deep association, and action detection. First, we propose a harmonization-aware image composition module to generate synthetic but realistic balls. Second, we present two adaptive inference strategies, a cache-friendly pipeline, and some pruning and quantization strategies to address the energy-efficiency concern. Third, we put forward a shape-texture debiased training and a domain-invariant adversarial training to improve the robustness of ball detection and ReID feature extraction. Last, inspired by morphology-based denoising, we develop a learning-free action detector that only depends on bounding box trajectories to spatio-temporally localize key actions.