{'fraction_non_alphanumeric': 0.028231334912127634, 'fraction_numerical': 0.006107441106817898, 'mean_word_length': 5.112952380952381, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Generic Object Re-identification System for Short Videos

Short video applications like TikTok and Kwai have been a great hit recently. In order to meet the increasing demands and take full advantage of visual information in short videos, objects in each short video need to be located and analyzed as an upstream task. A question is thus raised -- how to improve the accuracy and robustness of object detection, tracking, and re-identification across tons of short videos with hundreds of categories and complicated visual effects (VFX). To this end, a system composed of a detection module, a tracking module and a generic object re-identification module, is proposed in this paper, which captures features of major objects from short videos. In particular, towards the high efficiency demands in practical short video application, a Temporal Information Fusion Network (TIFN) is proposed in the object detection module, which shows comparable accuracy and improved time efficiency to the state-of-the-art video object detector. Furthermore, in order to mitigate the fragmented issue of tracklets in short videos, a Cross-Layer Pointwise Siamese Network (CPSN) is proposed in the tracking module to enhance the robustness of the appearance model. Moreover, in order to evaluate the proposed system, two challenge datasets containing real-world short videos are built for video object trajectory extraction and generic object re-identification respectively. Overall, extensive experiments for each module and the whole system demonstrate the effectiveness and efficiency of our system.


INTRODUCTION
Short video has been one of the most important social media and entertaining fashion for people worldwide recently. As millions of short videos are uploaded to the platforms by users each day, requirement of generic object re-identification from a large amount of short videos increases rapidly.
However, building a generic object re-identification system for short videos poses a number of challenges due to characteristics of short videos as follows: • Great variety in pattern and object category: Most short videos in short video platforms are uploaded by users. Topics of video vary a lot according to interest of users, and numerous categories of objects may occur in the same video simultaneously. • Various visual effects (VFX): Visual effects are common when different video shots switching or emphasizing objects in short videos. These visual effects may cause color change of video frame, object distortion, etc. • Complicated motions and postures: Different from scenarios such as traffic monitoring, motion and posture of people in short videos are much more unpredictable. Moreover, frequent strenuous motion may also bring great burden to object detection and tracking. • Rapid changes in object appearance: Different from traffic scenarios in most person re-identification tasks, people dress differently in different short videos. Even more, some short videos are consist of multiple video clips (or shots), and appearance of the same object changes rapidly when the video shot switches, such as in dress switching videos.
As a result, short video has become one of the most challenging scenarios in recent years.
To extract the trajectories of objects in videos, there exist a series of studies on video object detection and video object tracking. However, most of the public benchmark datasets only focus on relatively few categories. For example, ImageNet VID [24], which is most frequently used for evaluation in video object detection, only contains 30 categories, most of which are animals and vehicles. By contrast, there are hundreds of various categories, complicated visual effects (VFX) and multiple video shots in short video scenario, which is opposite to the scenario that most recent approaches assumed.
In order to satisfy the mentioned challenge of generic object re-identification in short video scenarios, a novel system composed of a detection module, a tracking module, and a generic object reidentification module is proposed in this paper. In this framework, the trajectories of the major objects in the input query videos and their features are extracted by the three modules in the proposed system. As follow-up applications, the processed features can be applied in short video identical object retrieval tasks. The flowchart of the proposed system is illustrated in Figure 1.
In particular, in order to handle the video object detection task with hundreds of categories, we build our detection module based on an image object detector, and propose a Temporal Information Fusion Network (TIFN) to take full advantage of temporal information among continuous frames. Further, a Cross-Layer Pointwise Siamese Network (CPSN) is proposed in the tracking module to mitigate the fragmented issue of tracklets, enhancing the robustness to match the same target despite the variety in appearance of objects in short videos.
To evaluate the proposed system, two challenge datasets are built for short video object trajectory extraction and generic object re-identification respectively. Extensive experiments for proposed methods and the whole proposed system are conducted with both the proposed short video datasets and public benchmark datasets, and the results demonstrate the high effectiveness and efficiency of the proposed system.
The contributions of this paper are summarized as follows: • We design a generic object re-identification system for short videos. To the best of our knowledge, it is the first work to tackle the generic object re-identification problem under short video scenario, which is much more complicated and unpredictable in variety than regular benchmarks. • We propose TIFN, which takes full advantage of temporal information in videos to improve image object detector, and shows comparable accuracy and improved time efficiency to the state-of-the-art video object detector. • We propose CPSN, which makes full use of the local features in different scales, and excludes the disturb of the background and rapidly changing regions of foreground. This module can thus effectively reduce the fragmented issue of tracklets. • Two challenging datasets for short video object trajectory extraction and generic object re-identification are built, which have many potential applications in follow-up study of short video.
The rest of the paper is organized as follows. Section 2 discusses related work. The proposed framework and methods are detailedly described in Section 3. Proposed datasets introduction and experimental evaluations are shown in Section 4. Finally, Section 5 concludes this paper.

RELATED WORK
Building a system to handle generic object re-identification task among different short videos includes three primary sections: a video object detection module to locate objects, a video object tracking module to extract object tracklets in each shot, and a generic object re-identification module to retrieve tracklets of the same object in different shots to a complete trajectory. Therefore, the related work of the proposed can be concluded as follows.
Video Object Detection: Video object detection (VOD) focuses on temporal information between adjacent frames. Combining detection and tracking, [13] propose a multi-phase framework, including image object detection, bounding box tracking and temporal convolutional network to re-scoring tubelet, to obtain enhanced performance. [19] propose a scheduler network, which determines to detect or track at a certain frame. Without aid of tracking module, [37] improve the per-frame features by aggregation of nearby features along the motion paths obtained by optical flow. Innovatively, [28] jointly calibrate the features of objects on both pixel-level and instance-level by optical flow, capturing detailed motion and global motion features. Instead of the time-consuming optical flow, [18] use LSTM to extract temporal motion information. [31] propose a novel RNN architecture called the Spatial-Temporal Memory Network (STMM) to model both changing appearance and motion of objects over time. Considering the full-sequence level features, [30] devise Sequence Level Semantics Aggregation (SELSA) module to obtain more discriminative and robust features for video object detection. [6] take full consideration of both global and local information and enlarge range of accessible content of the key frame by introducing a Long Range Memory (LRM) module. However, all the methods mentioned above either associate temporal information in frames by optical flow and RNN, or formed as a complicated and time-consuming framework. Moreover, all of them need training dataset where bounding boxes in every frame are well-annotated, which is difficult for labelling when categories increase. In contrast, the Temporal Information Fusion Network (TIFN) we propose is based on fast one-stage detector without extra supervision, so it can be trained with dataset for image object detection.
Video Object Tracking: Most of the recent multiple object tracking methods only focus on tracking single category, such as pedestrians or vehicles. A standard tracking system can be divided into four stages: detection stage, feature extraction stage, affinity stage and association stage, while recent works pay more attention to the second and third stages. DeepSort [29] integrates the deep neural network to learn appearance information, and obtains improved performance compared with previously published approach SORT [4]. DAN [25] combines the feature extraction stage and affinity stage together to learn compact, yet comprehensive features in an end-to-end manner. Unlike most methods which use the off-the-shelf detector to detect region of interest, Tracktor [3] creates a new tracking paradigm, in which a single detector solve most of tracking problems. Unfortunately, most of these methods can only handle the pedestrian tracking problems under the traffic surveillance scenarios, which are not applicable to generic objects. In this paper, we propose a tracking system to adapt to generic object tracking for short videos.
Re-identification: Recently most re-identification (ReID) methods focus on person retrieval. A standard person re-identification system contains three main components: feature representation learning, deep metric learning and ranking optimization. Feature representation in ReID can be furtherly divided into global feature and local feature. Due to the complicated scenario in person ReID, local features such as partial region feature, human pose and landmark feature, are becoming principal components, rather than global features. [33] extract global feature which is jointly learned with local partial region features to enhance global feature learning. [34] employ human pose estimation and landmark detection in person re-identification. [26] target in learning discriminative partinformed features, and propose Part-based Convolutional Baseline (PCB) and refined part pooling (RPP), resulting in refined parts with enhanced within-part consistency. However, most methods mentioned above are not designed for short video scenarios. Due to the challenging characteristics of short videos described in Section 1, existing methods are difficult to be applied to generic object re-identification for short videos directly. Considering numerous categories, various VFX, multiple video shots and rapid appearance changes, we propose a novel system to handle generic object re-identification for short videos in this paper. To our knowledge, it is the first system to tackle generic object re-identification problem for short video, which is one of the most challenge task in recent years.

PROPOSED METHOD
As is shown in Figure 1, a generic object re-identification system for short videos can be formulated as follows. Given a sequence of video frames = { } =1 which contains objects and shots, the goal is to obtain the trajectories of the major objects {traj } ∈major( ) and use their features {feat } ∈major( ) to retrieve other videos { } with identical objects in gallery, where major( ) denotes the major objects of the query video. Shots in the query video are denoted as = { } =1 , where = { } * +length( )−1 = * denotes the th shot in video which started at = * .
Considering the complex scenario in short video and robustness of algorithm, we decompose the generic object re-identification for short videos problem into several sub-problems, and formulate them as an end-to-end system, which consists of three major modules: detection module, tracking module and re-identification module (colored in Figure 1). An algorithm summary follows the introduction of the three modules.

Detection Module
In order to extract feature embedding of each object in video accurately, all the candidate objects need to be localized at first. The aim of detection module is to detect bounding boxes = { } =1 of the objects in each frame .
To detect objects in video, most works utilize optical flow motion or on the basis of two-stage detector like Faster-RCNN [23], which are time consuming. Moreover, to train these models, dataset that objects are well-annotated in every frame is required. When category of object increases, the time cost of labelling all frames in video increases proportionally as well. In view of the great variety of categories in short video and the efficiency requirements in practical applications, we start from a fast one-stage detector YOLOv3-SPP [10,22] for single image detection, and take full advantage of temporal information in video to improve the accuracy by introducing the proposed Temporal Information Fusion Network (TIFN) without extra supervision.
Inspired by [6], the temporal information in video can be summarized as three parts: local semantic information, local localization information and global semantic information. Local semantic information is the appearance similarity of the same object between adjacent frames, and local localization information represents the continuity of spatial information (i.e., height, width and center location) of the identical object in continuous frames. Global semantic information, which is similar to the local one, means the appearance similarity of the same object in the entire video, but due to practical implementation and efficiency, we only consider frames before the frame at current * . Figure 2 is an illustration of the detection module, and Figure 3 shows the three time information fusion modules.
According to the candidate boxes predicted in each frame, the corresponding regions in the three feature maps before each detection head of three scales are cropped and saved with their spatial information, i.e., center position, width and height, denoted as {crop } , =1, =1 and {spat } , =1, =1 . For local semantic information, as illustrated in Figure 3 where Attn ls denotes the attention map of * obtained by local semantic information, i.e., the mean cross-correlation heatmap obtained, clip(·) denotes clipping value to [0, 1], xcorr(·) denotes calculating cross-correlation, and Φ(·) denotes the neural network for feature extraction in detector.
For local localization information, as illustrated in Figure 3(b), attention map is obtained simply by applying truncated 2-Dimension hanning window at the corresponding coordinates to the saved center position where Attn ll denotes the attention map of * obtained by local localization information and hanning(·) denotes applying truncated 2D hanning window with threshold .
For global semantic information, as illustrated in Figure 3(c), the key is to utilize high-quality frames in global time domain of the input video to improve detection of local low-quality frames. Since the appearance and position of objects in short video may change rapidly and with low-quality due to the frequent video shot switching or rapid motion, local semantic information and local location information are not sufficient enough. However, highquality frames in global with similar semantic information can be used to lead the attention to focus on objects in these low-quality frames.
To capture the global feature of objects, we build a global set with fixed size 1 which stores features of objects with high frequency. Similar to local semantic information, cropped features in each frame are obtained, and firstly added to a candidate pool with fixes size 2 ( 2 > 1 ). Features in the candidate pool are where Attn gs ( * ) denotes the attention map of * obtained by global semantic information. All the formulations mentioned above are with no extra parameters to be trained and can be done by processing each video frame only once. At last, 3 groups of candidate boxes are predicted by the feature maps applied with Equation (1)-(3) respectively, and proceeded to post-processing phase such as Non-Maximum Suppression (NMS) together to obtain the final predicted bounding boxes .

Tracking Module
After the detection stage, we could obtain a series of bounding boxes = { } =1 . The aim of the tracking module is to link each bounding box of the same object. So we have to extract the features of those bounding boxes, compute the affinity matrix between detection responses and tracklets, and assign the matched bounding boxes to the existing tracklet frame-by-frame. The paper in this part focus on the feature extraction stage. Most existing methods focus on the global features, i.e., image-level features. [16] argues that a measure in such a level may not be effective enough in light of the scarcity of examples under our scenario. Inspired by the paper, we could obtain a more fine-grained feature based on the local features.
In order to be consistent with the feature extractor in the following re-identification module, we first employ ResNet50 [11] to learn the feature map with spatial feature. Given the cropped feature map crop ∈ × 1 × 1 derived by the detected bounding box ( is ignored for convenience of description) and a single crop ∈ × 2 × 2 from existing tracklet, we try to obtain a pointwise response map. denotes the cell in feature map, i.e., the local feature of the target. For convenience of calculation, cosine similarity metric is used to measure the similarity of the local region between detection response and tracklet. Naturally, the pointwise similarity could be formulated as follows: where sim(·) denotes the similarity between different cells and denotes the pointwise response map. We can just use matrix multiplication to calculate the response map, and the amplitude of each cell of it represents the similarity of the part from the pair. So we could use the mean value of the top maximum response to approximate the original similarity of the pair. This score could exclude the disturb of the background and rapidly changing parts of foreground.
The operation above just happens on one stage. Under shortvideo scenarios, there always be scale changes, so the same target may occupy in different scales in different two frames. As a result, we use multi-scale feature of region in two frames to find the regions matched with highest response. Briefly, 1 , 2 , · · · , denote the feature map in different scales of the detection response, and similarly, 1 , 2 , · · · , denote the feature map in different scales of the tracklet. So we could integrate multiple layers in different scale and averaging the score mentioned above, and get the final similarity score.
where top (·) denotes the top values of cells in the matrix , and SIM(·) denotes the similarity between matrices, which is different from the similarity between cells in Equation (4).
For other parts such as motion prediction branch, bipartite matching, trajectory management, we just keep the same as DeepSort, due to its high efficiency.

Re-Identification Module
In re-identification module, tracklets of the th object from shots are grouped into a complete trajectory = {tr } =1 , and features of objects per frame in trajectory traj are sampled and averaged as the features of trajectory traj , denoted as feat . Then major trajectories {traj } ∈major( ) are selected by duration time and mean size, and features of major objects {feat } ∈major( ) are selected relatively. A similarity score is computed between each pair of major object features in query video and gallery video to retrieve videos with identical objects.
In retrieving objects of general categories for short videos, for person, we need to identify each instance since they are different in appearance. But for non-person object, such as animal or commodity, instances of same breed or same pattern are unrecognizable in appearance mostly. As a result, the re-identification module is divided to two branches: person branch and non-person object branch, dealing with each kind of object respectively.
In the person branch, we apply the person re-identification process with face detection and recognition, since in different short videos, the same person may dress in different clothes. As a result, a face detection and recognition model is introduced to support person re-identification. We use the PCB network [26] as the person ReID feature extractor and ArcFace [9] as the face feature extractor. The person similarity is decided by cosine similarity computed between each pair of features obtained by the two models together as follows , not detected (7) In the non-person object branch, since we regard unrecognizable objects of same breed or same pattern as same instance (unless they appear simultaneously), we formulate the non-person object re-identification as a fine-grained classification problem. The first four blocks of ResNet50 [11] trained by ImageNet dataset [8] with a average pooling layer followed are applied in non-person object branch as feature extractor. ImageNet dataset consists of 1000 finegrained categories, as a result of which, discriminative feature can be obtained.
Furthermore, since general re-identification architectures retain a fixed query set, tracklets appeared in the first frame in most cases, to retrieve identical objects in the gallery set, which is hard to satisfy the complicated variety of appearance and pose in short videos and is liable to cause mismatching, an updating mechanism is introduced to query set in both person branch and non-person object branch in re-identification module. Setting tracklets appeared in the first frame as query set similarly, when each tracklet remained in gallery set is accessed individually, similarities between query tracklets and it are first computed. Contrast to general formulation, the accessed tracklet is updated to the query set if the difference between mean similarity among matched tracklets and mean similarity among unmatched tracklets is larger than a threshold Δ. Therefore, the proposed updating mechanism can mitigate mismatching caused by complicated variety of appearance and pose in short videos.
In identical object retrieval among different videos, first the major objects are selected according to the length and the average size of predicted trajectories. Then the average features of each major object are calculated by re-identification module and used to match other features of major objects in the short video gallery according to cosine similarity metrics.

EXPERIMENTS
To demonstrate the performance of the proposed generic object re-identification system for short videos, two novel datasets for evaluation are proposed in this paper, which comprised of various short videos collected from short video platform. Besides, the proposed generic object re-identification system is compared with other approaches under public benchmark, such as ImageNet VID [24] and MOTChallenge [1], to indicate the generalization ability.

Proposed Datasets
To compare with existing methods and determine state-of-the-art approach under short-video scenarios, we propose two novel short video dataset for evaluation in identical object retrieval among different shots and different videos. Videos in the proposed datasets are both collected from real-world short video platform. The short video dataset for identical object retrieval among different shots, abbreviated as SVD-IOR, includes 400 videos in amount with a variety of characteristics, e.g., multiple categories, complex scenarios and frequent shot changes. Foreground objects are annotated every 10 frames in the form of bounding boxes with corresponding track ID in each video. As is shown in Figure 5, the lengths of most videos are distributed between 10 and 20 seconds. In total, it contains 232964 frames at various resolutions, with 1221 different identities and 66180 bounding boxes annotated. Comparisons among ImageNet VID, MOTChallenge and the proposed SVD-IOR, SVD-ReID are shown as Figure 6.
The short video dataset for generic object re-identification among different videos, abbreviated as SVD-ReID, contains 400 short videos. Objects of four categories, i.e., person, cat, dog, landmark, are annotated as same as which in SVD-IOR. Videos of each class contain 10 different object instances and each instance contains 10 different videos collected from corresponding user. Specially, track ID of each identical object among different videos is annotated as same value. The 10 videos of each object instance are split into 2 query videos and 8 gallery videos, and aggregated to build a query set and a gallery set in order to evaluate generic object re-identification among different videos.
To demonstrate the generalization ability of our method on public benchmarks, the proposed system is evaluated on benchmark dataset ImageNet VID and MOTChallenge respectively as well.

Implementation Details
For detection module during training, we follow the settings in [22] to train YOLOv3-SPP model with training set of OpenImage [15] and ImageNet VID, respectively evaluating on the proposed short video dataset and ImageNet VID validation set. During evaluation, the input videos are first split by shots with shot boundary detection approach based on color histogram [20] to ensure tracklets of objects are continuous in each shot. The input video frames are resized and padded to 608 × 608.
For tracking module, We use the training set of MOTChallenge to train, while using the validation set to evaluate. For the proposed CPSN, we rescale the image pair to 256 × 256 as inputs during training. With ResNet50 as the backbone, outputs of the second and forth stage are used to build the response map in Equation 5. It is worth noting that the calculation of the response map don't introduce any extra parameters. Circle loss and Adam optimizer are employed to optimize the proposed CPSN.
For re-identification module during training, we follow the settings in [26] and train the PCB network as the person re-identification model with Market-1501 [35], while using ResNet50 pretrained by ImageNet [8] as the non-person object re-identification model to extract fine-grained feature.

Evaluation of Detection Module
For video object detection under short video dataset, ablation experiments with SVD-IOR are applied to demonstrate the effectiveness of the proposed TIFN. Moreover, performance comparison between the proposed TIFN and the state-of-the-art video object detection models on ImageNet VID validation set are reported. Quantitative comparison results are summarized in Table 1 and Table 2. Table 1 describes the ablation experiment results of aggregation of local semantic information, local localization and global semantic information under short video dataset. As is shown in the table, the proposed TIFN obtains improvement of approximately 5% mAP rather than the base model YOLOv3-SPP, without extra supervision. The introduction of local semantic information contributes most, since the temporal information between adjacent frames is most significant in videos. The improvement brought by local localization and global semantic information indicate their effectiveness in dealing with the rapid change of appearance in short videos. Besides, comparison results of execution efficiency show that the proposed TIFN can still keep high efficiency after aggregating multiple temporal information.
As is shown in Table 2, TIFN performs comparable accuracy and improved time efficiency to the state-of-the-art approaches. Since most methods shown in Table 2 are formulated in much more complex structure with complicated post-processing, the proposed TIFN with a simple backbone is more suitable for real-world applications.

Evaluation of Tracking Module
For evalution of tracking module, the proposed CPSN is compared with numerous classic and recent object tracking methods both benchmark dataset MOTChallenge and the proposed short video dataset SVD-IOR. As is shown in Table 3, the proposed multi-object tracking method in this paper performs poor results on the public datasets, since it is not designed for the surveillance scenarios in MOTChallenge dataset. The better performance than jCC and FAMNet demonstrate the great generalization of the proposed CPSN to some extent.  On the contrary, Table 4 shows that our system achieves higher MOTA than Tracktor and Tracktor++v2 in short video dataset, which are the state-of-the-art video object tracking methods available in 2019. It indicates that our system has promising performance in video object tracking. Moreover, the proposed CPSN keep a higher efficiency than most of those methods.  Figure 8: Visualization of the re-identification results among different short videos. The first column denotes the query short videos, and the following columns denote query results in order of retrieve confidence.

Evaluation of Re-Identification Module
For identical object trajectory extraction among different video shot in single short videos, the detection and re-identification results under the proposed SVD-IOR dataset are visualized in Figure 7.
As illustrated in the odd rows, objects with rare categories and complicated motion can be still well detected. The corresponding re-identification results in the even rows shows that the identical object can keep same track-ID among different short video frames and shots, due to the robust detection results and the proposed system. For identical object retrieval among different videos, Table 5 indicates that the proposed system achieves promising performance evaluated on SVD-ReID dataset. Compared with the baseline model, which uses YOLOv3-SPP in detection module and disables the query set update mechanism in re-identification module, the proposed system performs better in all 4 categories. Due to the introduction of face detection and recognition, our system obtains high rank-1 accuracy in person re-identification among short videos. Moreover, landmark buildings are also well retrieved since each of them generally share a relatively consistent appearance in different videos. Cats and dogs in same breed but different ID are difficult to distinguish, as a result of which, these categories achieve a lower rank-1 accuracy. Re-identification results among different short videos are shown in Figure 8.

CONCLUSION
This paper makes two contributions to solving generic object reidentification problem for short videos. First, we propose a system composed of a detection module, a tracking module and a re-identification module, which formulate the challenging problem into three main sub-problems. In order to satisfy the high efficiency requested in practical application and get over the complicated variety of appearance of objects in short videos, we propose Temporal Information Fusion Network (TIFN) and Cross-layer Pointwise Siamese Network (CPSN) in detection module and tracking module respectively. Moreover, we propose two novel real-world short video datasets collected from short video platform for evaluating object trajectory extraction and generic object re-identification among different short videos. Quantitative experiments demonstrate the high effectiveness and efficiency of our system.