{'fraction_non_alphanumeric': 0.03815626291856139, 'fraction_numerical': 0.008309218685407193, 'mean_word_length': 4.741989081414669, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TGRMPT: A Head-Shoulder Aided Multi-Person Tracker and a New Large-Scale Dataset for Tour-Guide Robot

A service robot serving safely and politely needs to track the surrounding people robustly, especially for Tour-Guide Robot (TGR). However, existing multi-object tracking (MOT) or multi-person tracking (MPT) methods are not applicable to TGR for the following reasons: 1. lacking relevant large-scale datasets; 2. lacking applicable metrics to evaluate trackers. In this work, we target the visual perceptual tasks for TGR and present the TGRDB dataset, a novel large-scale multi-person tracking dataset containing roughly 5.6 hours of annotated videos and over 450 long-term trajectories. Besides, we propose a more applicable metric to evaluate trackers using our dataset. As part of our work, we present TGRMPT, a novel MPT system that incorporates information from head shoulder and whole body, and achieves state-of-the-art performance. We have released our codes and dataset in https://github.com/wenwenzju/TGRMPT.


I. INTRODUCTION
Service robots work in human-populated environments. Robust tracking surrounding people allow robots to serve and navigate safely and politely. In this work, we aim at the egocentric perceptual task, multi-person tracking, using a tour-guide robot (TGR). The TGR is a robot that can show people around a site such as museums, castles, and aquariums, and can introduce surroundings to people [1]. In this scenario, targets occlude with each other and step in or out the camera view, frequently. Moreover, humans in a group usually dress in the same clothes, making it a difficult task to identify each other. Despite the above challenges, individuals in a guided group must be tracked and assigned consistent IDs during the whole tour-guide service period.
However, there is a vast gap between the existing MOT or MPT datasets and the tour-guide scenario, making existing trackers perform unsatisfactorily. Some examples of such datasets are the well-known MOTChallenge [2] and KITTI [3], targeting applications in video surveillance and autonomous driving, respectively. Furthermore, in these datasets, a person will be assigned a new ID if he/she leaves the field of view or is occluded for a prolonged period and then reappears. This is very different from our scenario where consistent IDs are required during the whole tracking period. More recently, an egocentric dataset, JRDB, was presented in [4], which is more relevant to our work. JRDB is a multimodal dataset and was captured using a moving robot in a university campus, both indoors and outdoors. However, this dataset is still inapplicable in our scenario. On the one hand, JRDB was captured in daily life where few interactions All   between students and robot exist. As a result, cases that humans disappear then reappear again are few. On the other hand, people in JRDB dress casually and almost no one wears the same. To solve the above problems, we present the TGRDB dataset, a new large-scale dataset for TGR. To capture our dataset, we use a 180 • fisheye RGB camera mounted on a moving (sometimes standing) robot shown in Fig. 1. In an indoor tour-guide scenario, the robot shows 5 or 6 individuals around. At first, participants wear their own clothes and complete the first capturing round. Then they are required to change and put on the same clothes, as shown in Fig. 1. There are totally 5 capturing rounds for each group, including one casual-cloth round and 4 same-cloth rounds, and there are 18 groups in total. As a result, 90 video sequences are captured, totally of 5.6 hours long. We annotate each video frame with whole body and head shoulder bounding boxes, and unique IDs, resulting in over 3.2 million bounding boxes and 450 trajectories. We hope this dataset will drive the progress of research in service robotics, long-term multiperson tracking, and fine-grained [5] or clothes-inconsistency [6] person re-identification. Associated with this dataset, a new metric named TGRHOTA is proposed. It is a simplified but more practical version of HOTA [7]. TGRHOTA only considers matches between prediction and ground truth as true positives at the most previous frames when evaluating association score. We will describe it in Sec. III-B.
As part of our work, we present TGRMPT, a novel and practical multi-person tracker in the tour-guide scenario. As a person's head-shoulder contains discriminative information such as hairstyle [8], and involves fewer distractors than the whole body bounding box when he/she is partially occluded, we optimally integrate head-shoulder information into the tracker. Specifically, we adopt the state-of-the-art DeepSORT [9] framework and tailor it to our scenario. Both whole body and head shoulder detectors are trained using our dataset, as well as deep appearance embedding feature extractors. To integrate these two types of information, we simply concatenate the two embedding features extracted from the corresponding bounding boxes, resulting in stronger appearance descriptors. Extensive experiments verify the effectiveness of our proposal.
To summarize, our contributions are as follows: • We release the TGRDB dataset, a first large-scale dataset towards the applications of the tour-guide robot. This dataset not only benefits the domain of service robotics but can also drive the progress of domains related to multi-person tracking and person reidentification. • We propose a more practical metric, TGRHOTA, to evaluate trackers in the tour-guide scenario. Different from existing metrics, TGRHOTA punishes trackers when a new ID assignment occurs if the target has already been assigned an ID before. • We propose a novel head-shoulder aided multi-person tracker, named TGRMPT, that leverages best of both information containing in the whole body and head shoulder. Experiments show that our tracker achieves start-of-the-art results.

A. Tracking Methods
Most multi-object tracking algorithms consist of two critical components, object detection and data association, responsible for estimating the bounding boxes and obtaining the identities, respectively.
Detection in tracking: Benefiting from the development of deep learning, the detection algorithm has been significantly improved. In addition, the quality of detection directly affects the tracking results. Therefore, many researchers [10]- [14] are committed to improving the detection ability. POI [10] achieves the state-of-the-art tracking performance due to the high-performance detection and deep learningbased appearance features. STRN [11] presents a similarity learning framework between tracks and objects, which encodes various Spatial-Temporal relations. The trackingby-detection pipeline achieves leading performance, but the model complexity and computational cost are not satisfying. Centertrack [12] proposes to estimate detection box and offset by using the data of two adjacent frames, which improves the ability to recover missing or occluded objects. [13] leverages the transformer architecture, an attention-based querykey mechanism, to integrate the detection information of the previous frame to the detection process of the current frame. [14] leverages two adjacent frames information to improve detection under the transform framework and proposes dense pixel-level multi-scale queries that are mutually correlated within the transformer attention and produce abundant but less noisy tracks. In addition, many works directly use the off-shelf detection methods, e.g., two-stage [15] or one-stage object detectors [16], YOLO series detectors [17], anchorfree detectors [18].
Data association: As the core component of multi-object tracking, data association first computes the similarity between tracklets and detection boxes, then matches them according to the similarity. Many methods focus on data association to improve tracking performance. SORT [19] is a simple but effective tracking framework that employs Kalman filtering in image space and frame-by-frame data association using the Hungarian method with the association metric that measures bounding box overlap. To address occlusions, [9] put forward to adopt an independent REID model to extract appearance features from the detection boxes to enhance the association metric of SORT. To save computational time, [20]- [23] integrate the detecting and embedding models into a single network. To address the nonnegligible true object missing and fragmented trajectories that are caused by simply throwing away the objects with low detection scores, ByteTrack [24] propose to track by associating all detection boxes. To recover true objects, the association method in [24] utilizes similarities to filter out background detections with low scores. Many researchers consider splitting the whole tracking task into isolated subtasks, such as object detection, feature extraction, and data association, which may lead to local optima. To address this issue, the research in [25], as well as the subsequent works in [12], [26], [27], propose to use an end-to-end model to unify the three isolated subtasks.

B. Tracking Datasets
Our TGRDB is a multi-person tracking dataset and benchmark containing fine-grained and clothes-inconsistency targets in the tour-guide scenario. [2], [4], [5], [28], [29] are the most relevant datasets to TGRDB. MOT [2], [28] is a well-known multi-object tracking benchmark, and many methods are based on this. JRDB [4] is a novel multi-modal dataset collected from a mobile social JackRabbot. [29] is an image dataset for cloth-changing person REID. [5] released a fine-grained REID dataset containing targets with the same clothes. In addition, the automatic driving datasets [3], [30], [31] are also interrelated to our TGRDB, which annotate a large number of pedestrians bounding boxes. To the best of our knowledge, there is no tracking dataset containing finegrained and clothes-inconsistency targets on the same scale as our TGRDB in the tour-guide scenario.

C. Metrics
Within the last few years, new MOT metrics have been proposed enormously. To remedy the lack of generally applicable metrics in MOT, [32] introduces two novel metrics, the multi-object tracking precision (MOTP) and the multiobject tracking accuracy (MOTA), that intuitively express a tracker's overall strengths. [33] proposes a new precisionrecall measure of performance, IDF1, that treats errors of all types uniformly and emphasizes correct identification over sources of errors. MOTA and IDF1 overemphasize the importance of detection and association separately. To address this, [7] presents a novel MOT evaluation metric, higherorder tracking accuracy (HOTA), which is a unified metric that explicitly balances the effect of detection, association, and localization. Nonetheless, none of the above metrics is applicable in our scenario. Hence we propose a new metric, TGRHOTA, for fair comparison of trackers in TGR.

III. DATASET AND METRIC A. Dataset
Our TGRDB dataset was collected with a 180 • fisheye RGB camera on-board of a tour-guide robot shown in Fig. 1. Equipped with a 360 • 2D LiDAR, our TGR can autonomously navigate and show people around. There is a pan-tilt with two degrees of freedom mounted at the neck, so the robot can rotate its head to see what it's interested in. There are 4 exhibit points where the robot stays and introduces exhibits to participants. Pictures are captured at these points respectively. .
We collect our dataset in an indoor tour-guide scenario. From the start point, the robot navigates to the first preset exhibit point, stays for a while and introduces the surrounding exhibits to participants, then moves to the next exhibit point, as shown in Fig. 2. Repeat the above steps until it returns to the start point. There are totally four exhibit points and the robot stays for around 22 seconds at each point. During the introduction period, the head of the robot alternatively looks forward or rotates randomly. Participants are required to walk freely and behave naturally. They need to follow the robot when it moves from one exhibit point to another. During this period targets are out of the field of camera view.
We divide all the participants into 18 groups, in each there are 5 or 6 individuals. Each group is required to finish 5 tourguide rounds, including 1 casual-cloth round and 4 samecloth rounds as shown in Fig. 1.
Each video frame is annotated with both whole-body and head-shoulder bounding boxes, as well as unique IDs. For one target, his/her ID is consistent no matter which cloth he/she dresses in. As a result, totally 90 video sequences are captured at 25 fps. The average duration of each video is 3.76 minutes. We divide the dataset into train and test sub-datasets. Table I shows more details of our dataset.

B. Metric
An applicable metric is significant when comparing different trackers. Existing metrics, such as HOTA [7], MOTA [32] and IDF1 [33], treat all matched prediction (pr) and ground-truth (gt) pairs as true positives (TPs), which may be reasonable in video surveillance or autonomous driving. However, they are not applicable in TGR for the requirement of consistent IDs during the whole tour-guide period. In retrospect, HOTA [7] counts TPAs (True Positive Associations), FNAs (False Negative Associations), and FPAs (False Positive Associations) for each TP and defines association score as follows: where all matched pr and gt pairs are considered as TP of interest. In other words, as shown in Fig. 3, TP of interest is selected in matches within all dashed areas. However, in TGR, we only consider TPs in the green area, i.e., matches at most previous frames. Formally, we define the set of TPs in TGR as follows: where t is the time index, and ≡ denotes that two TPs have the same prID and gtID, while ≡ denotes two TPs have different prID and gtID. Thus, the association score in TGR can be written as: (3) The detection score, DetA, and the calculation of the final HOTA score, is the same as in [7], except that we use AssA instead of AssA. Fig. 4 shows an example where HOTA gives the two trackers the same scores which is counterproductive for our TGR scenario where tracker B performs much better than tracker A. Treating all TPs equally even if they have been assigned another ID at a previous frame, HOTA is not applicable in our scenario. On the contrary, our TGRHOTA gives much more reasonable scores.

IV. TGRMPT: HEAD-SHOULDER AIDED MULTI-PERSON TRACKING
Fig. 5 depicts our proposed TGRMPT tracker. By sequentially applied whole body (wb)/head shoulder (hs) detectors and wb/hs feature extractors, deep appearance signatures describing global whole-body and local head-shoulder are generated. Concatenating these two types of features results in the final strong descriptor that contains both global and local information. The followed Hungarian algorithm produces the final robust tracking result.

A. Detector
Detection is the core component of the existing MOT or MPT systems. In consider of system speed, we deploy YOLOv5s [34] as our detector. We fine-tune this network on our dataset using wb and hs annotations to form wb and hs detectors, respectively. L }, and we discard those hs detections that are not matched to any wb detections.

B. Appearance Descriptor
Appearance descriptors are used to measure the similarities between detections at current frame and history trajectories. They offer signification, sometimes the only, cue to re-identify the target when he/she was missed for a period of time and reappears. To obtain strong and discriminative appearance descriptors, we leverage ResNet18 [35] as our feature extracting network and train it using our TGRDB dataset described above. We choose ResNet18 due to the trade-off between speed and performance. To integrate global and local information, wb and hs feature extracting networks are trained individually using wb and hs re-identification dataset, respectively. For detection pair (d wb i , d hs j ) ∈ D match t , the corresponding appearance features are we denote its appearance feature as

C. Tracking
We adopt DeepSORT [9] framework to perform tracking. Given a set of appearance descriptors, F f used t , we associate them to trajectories at previous frames. To do so, we keep the latest P descriptors for each trajectory and leverage cosine similarity to measure the distance between trajectories and detections. Specifically, for one detection f ∈ F f used t and one trajectory T = {f 1 , ..., f P }, a set of cosine similarities, S(f, T ) = {s 1 , ..., s P } are calculated, and we average them to produce the final distance, D(f, T ) = P i=0 (1 − s i )/P . We also explore the performance of the minimum similarity values, D(f, T ) = min(1 − s i ), in experiments. After computing the pairwise distance between current detections and history trajectories, we obtain Q × (K + L) cost matrix where Q is the number of trajectories, i.e., the number of tracked targets. Hungarian algorithm is employed to get the final association results and trajectories are updated using the corresponding associated detections. Any association that has a distance greater than the preset threshold τ is treated as unmatched.
The origin DeepSORT is designed for video surveillance and is not well fitted to our tour-guide scenario. In the data association process of DeepSORT, matching cascade is introduced to give higher priority to trajectories that miss targets for less time. Meanwhile, a so-called gating mechanism is applied and only detections near the predicted locations of trajectories are considered. These two tricks may cause more track fragments and worsen the tracking performance in our scenario as frequent occlusion and long-term missing exist in our dataset. Consequently, we abandon these two tricks in our tracker. Another trick to handle challenges in TGR is to assign a relatively great value to the age threshold, α. When a trajectory is missed, i.e., no detections associated with it, for consecutive α frames, it will be deleted. We explore different values of α in our ablation experiments and conclude that α = ∞, i.e., no trajectories are deleted during tracking, gives the best performance.

V. EXPERIMENTS
To evaluate on our TGRDB dataset, the widely used metrics, HOTA [7], MOTA [32] and IDF1 [33] are adopted.  Fig. 5: The pipeline of our proposed TGRMPT framework. Integrating both whole body (wb) and head shoulder (hs) information, our method produces strong and discriminative appearance feature descriptors. Followed by Hungarian algorithm, robust tracking result is obtained.
Besides, we report results evaluated using the proposed TGRHOTA metric as well. For a more comprehensive analysis, we split the test dataset into casual-cloth and same-cloth sub-datasets, and conduct experiments respectively. But we don't split the training dataset.
A. Ablation Studies 1) Hyper-Parameters: Different values of hyperparameters, α and τ , may impact the final performance dramatically. We fix the distance threshold τ and set it to 0.5 at first, and change the age threshold α to explore the impact. In this stage, we employ the whole body branch only. The results are shown in Table II. Intuitively, α impacts more on data association. We can see this from AssA and IDF1 scores. In the TGR scenario, it is required that people's IDs remain consistent even after long-term target missing. Thus the greater α should give the better performance. This is verified by the results in Table II, where the infinite α gives the best performance. This means that during the period of tour-guide, no tracks will be deleted, which gives chance to the tracker to find back the target even if he/she has disappeared for a long time. We then fix α = ∞ and change the distance threshold τ to explore the impact. In data association, there are two ways, mean or min as described in IV-C, to compute the distance between detection and trajectory. We show their results 1 in Fig. 6, where we conclude that the averaged distance value gives much better performance. The reason is apparent: the reserved P galleries in each trajectory may contain noisy 1 For simplicity, only results on same-cloth sub-dataset are shown as casual-cloth sub-dataset has similar results. samples and the minimum distance value suffers much from these noises.
The above experiments on hyper-parameters indicate that α = ∞ and τ = 0.85 give the best performance. We will fix them in the following experiments and explore other aspects of our method.
2) Addition of Head-Shoulder: Head-shoulder contains non-negligible features, such as haircut, glasses, complexion, etc., which can provide discriminative cues for person reidentification. We conduct comparative experiments on the employment of: 1) wb, the whole body only; 2) hs, the head shoulder only; 3) wb+hs, both whole body and head shoulder. The results are shown in Table III. Undoubtedly, the method that integrates both wb and hs information gives the best performance. To dig deeper into how head shoulder helps improve the overall performance, we show precision and recall scores of association in Fig. 7. Compared to the wb-only method, for casual-cloth sub-dataset, the percentage promotion of (a) Casual cloth (b) Same cloth Fig. 7: Precision and recall scores of data association. The integration of head shoulder contributes equally to precision and recall scores in casual-cloth sub-dataset while contributing more to precision score in same-cloth sub-dataset. .
precision and recall after involving head shoulder is 2.3% and 2.9%, respectively. Thus, the head shoulder contributes equally to precision and recall. Nevertheless, for same-cloth sub-dataset, the respective promotion is 5.9% and 1.4%. Head shoulder contributes much more to precision. In other words, with the help of head shoulder, the tracker does better in identifying different people even if they wear the same. This is due to the discriminative details contained in head shoulder which are untraceable in whole body.
3) New Metric: We show scores of our proposed TGRHOTA in Table IV. We can see that TGRHOTA scores are higher than HOTA scores. This is a similar situation as tracker B shown in Fig. 4. That is to say, once a target was assigned an ID, our tracker can keep it consistent most of the time.

4) FPS:
On a single thread, our method runs at 16fps endto-end, using an Nvidia 2080Ti GPU card. Parallel running of wb and hs detectors may speed up the method.

B. Comparison with State-of-the-Art
We compare our proposal with two newest state-of-theart methods, CenterTrack [12] and ByteTrack [24]. Results are shown in Table V. We train CenterTrack and ByteTrack using our dataset and fine-tune some hyper-parameters. Our method achieves the best performance. We show comparison results for reference only, as we have carefully tailored DeepSORT to fit the tour-guide scenario, especially for data association. Designed for video surveillance, it is foreseeable that CenterTrack and ByteTrack perform poorly without any scenario-oriented modifications.

VI. CONCLUSION
We release the TGRDB dataset, the first large-scale dataset for the applications of TGR. It is captured using a TGR in an indoor tour-guide scenario. We annotate the dataset with whole-body and head-shoulder bounding boxes, as well as unique IDs for each participant. Frequently occlusion, longterm missing and fined-grained targets (dress the same) are the main challenges in this dataset. We believe that TGRDB will help future research in service robotics, long-term multiperson tracking, and find-grained or clothes-inconsistency person re-identification. Along with the dataset, we propose a more practical metric, TGRHOTA, to evaluate trackers in the tour-guide scenario. Different from existing metrics, TGRHOTA only considers TPs at most previous frames, which is more applicable in TGR. As part of our work, we propose TGRMPT, a novel head-shoulder aided multi-person tracking system that leverages best of discriminative cues contained in head shoulder which are untraceable in whole body. Extensive experiments have confirmed the significant advantages of our proposal.