{'fraction_non_alphanumeric': 0.03730938051495543, 'fraction_numerical': 0.009774197927508032, 'mean_word_length': 4.8932, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adaptive Affinity for Associations in Multi-Target Multi-Camera Tracking

Data associations in multi-target multi-camera tracking (MTMCT) usually estimate affinity directly from re-identification (re-ID) feature distances. However, we argue that it might not be the best choice given the difference in matching scopes between re-ID and MTMCT problems. Re-ID systems focus on global matching, which retrieves targets from all cameras and all times. In contrast, data association in tracking is a local matching problem, since its candidates only come from neighboring locations and time frames. In this paper, we design experiments to verify such misfit between global re-ID feature distances and local matching in tracking, and propose a simple yet effective approach to adapt affinity estimations to corresponding matching scopes in MTMCT. Instead of trying to deal with all appearance changes, we tailor the affinity metric to specialize in ones that might emerge during data associations. To this end, we introduce a new data sampling scheme with temporal windows originally used for data associations in tracking. Minimizing the mismatch, the adaptive affinity module brings significant improvements over global re-ID distance, and produces competitive performance on CityFlow and DukeMTMC datasets.


I. INTRODUCTION
M ULTI-target multi-camera tracking (MTMCT) aims to formulate trajectories for different identities across multiple cameras. It plays a vital role in many applications including smart city analysis and autonomous driving [1], [2].
MTMCT extends the multiple object tracking (MOT) problem, which only focuses on a single camera and does not need to align cross camera identity. In fact, MTMCT can be divided into two steps: first, in single camera tracking (SCT), trajectories are linked within each camera; second, in multiple camera tracking (MCT), within-camera trajectories are associated across cameras. Most existing works on MOT and MTMCT follow the tracking-by-detection paradigm [3], where data association is arguably the most defining part (object detection is separately studied in other works [4]- [6]). In data association, targets of the same identity are linked into trajectories based on cost matrices using graph optimization. Such cost matrices are generated by affinity or similarity estimation, and their quality greatly influences the Yunzhong  data association performance [7], [8]. For affinity estimation, re-identification (re-ID) [9] feature distance is widely chosen in tracking systems [2], [10]- [12], since it can distinguish target identities across multiple cameras.
However, in this work, we find that directly using the re-ID feature distance as affinity may not be the best choice for MTMCT. As shown in Fig. 1, re-ID is a global matching problem, where the system tries to retrieve targets of the same identity from anywhere any time. Thus, re-ID systems have to deal with all possible appearance changes (e.g., occlusions, pose changes) at the same time. On the other hand, tracking aims to formulate continuous trajectories, where data associations are usually conducted in an hierarchical (first SCT and then MCT) and iterative (iteratively adding new candidates to existing trajectories) manner with smaller matching scopes (one frame or one time-batch at a time). As such, associations in tracking can be regarded as local matching problems, and the appearance changes that the system has to deal with are limited to the problem sizes considered in associations (matching scopes). The difference in problem sizes (global matching for re-ID versus local matching for tracking) violates the common belief that affinity estimations should be tailored for the corresponding matching scopes.
This phenomenon (different problem sizes in re-ID features and MTMCT data associations) is previously overlooked in MTMCT problems, and is less pronounced in MOT problems. In fact, in MOT (not to be confused with SCT in MTMCT), targets for re-ID feature learning only come from a single camera within a limited time period, and contain a similar level of appearance changes as the tracking system might encounter during data associations. By contrast, in MTMCT, such re-ID training data might come from different cameras over a long period of time (for them to travel to different camera locations), and contains much larger appearance changes. During data association, such drastic appearance changes are unlikely to all appear at the same time in either SCT or MCT, since the association problem sizes are usually limited to neighboring time frames and cameras.
In this paper, we first design experiments to verify this mismatch between global re-ID features for affinity estimation and local matching in MTMCT data association. To this end, we skip the detection and graph optimization algorithms, and directly compare the estimated affinity (similarity/dissimilarity) between the ground truth bounding boxes. Experiments show that directly using the same re-ID feature distance for both SCT and MCT can result in higher ratios of false positives, suggesting that the global affinity scores might be too tolerant and not discriminative enough for the local matching.  Fig. 1: Differences in matching scopes between re-ID and MTMCT. Re-ID (red dotted box) aims to retrieve from all locations and all times, thus requiring re-ID features to deal with all potential appearance changes at the same time. In contrast, MTMCT usually associates candidates in an hierarchical manner, by first tracking within single camera (black dotted box) and then across multiple cameras (blue dotted box). With limited problem sizes (matching scopes), associations in tracking only have to deal with a limited number of appearance changes at a time. Given the difference between matching scopes, we argue that directly using re-ID feature distance as affinity might not be the best choice for MTMCT data association.
To address this mismatch, we propose a simple-yet-effective adaptive affinity module for data associations in MTMCT that tailors affinity estimations to the corresponding matching scopes. In order to benefit from the strong identification ability of re-ID features while tailoring them to fit the local matching problem, we learn metric networks on top of the re-ID features. To fully exploit existing Siamese networks [13], we propose new data sampling mechanisms for metric leaning. To produce affinity estimations that best fit the smaller matching scopes in tracking, we train the metric networks with data pairs that might appear in the corresponding association problems. To this end, during training, we select data pairs with the same temporal windows adopted in the data association step. Specifically, we learn an intra-camera metric for associations in SCT and an inter-camera metric for associations in MCT. For intracamera metric, temporal windows select data pairs within the same camera; and for inter-camera metric, data pairs from all cameras are allowed, and temporal windows automatically select those in neighboring cameras. We clarify that this work does not make any architecture-wise contribution or design any new trackers. Its key contributions are 1) identifying and exploring the mismatch between global re-ID feature distances and local matching in MTMCT data associations and 2) a new data sampling scheme for learning adaptive affinity metrics to bridge the mismatch.
We show that the proposed adaptive affinity can effectively improve tracking accuracy on two MTMCT datasets, including a vehicle dataset, CityFlow [2], and a pedestrian dataset, DukeMTMC [1]. It can also be applied and on top of multiple re-ID features, such as IDE [14], PCB [15] and the triplet feature [16]. With a competitive tracker [12], we report the state-of-the-art accuracy on the DukeMTMC dataset.
II. RELATED WORK Multi-object tracking. Multi-object tracking (MOT) [8], [17], [18] tracks multiple targets within each scenario. The MOT challenge and its datasets witnessed the bloom of the modern MOT system [8], [17]. Most MOT systems follow the tracking-by-detection paradigm [3]. Since the detection part is also studied in other field [5], [6], many researchers focus on data association methods for MOT systems. For affinity estimation, existing works adopt convolutional neural network (CNN) feature distance [19], [20] or Siamese network scores [21], [22]. For optimization algorithms, there are both online and offline ones. Online tracking methods have a very small association problem size as they only consider the current frame and thus have minimal computation cost [23], [24]. As targets move continuously, their trajectories can still be formulated iteratively even with per-frame matching in online systems. The offline methods, on the other hand, consider multiple frames inside a temporal window during association. The increase in problem size can improve tracking performance at the cost of higher computation complexity. They usually formulate the problem as batch optimization, such as shortest path [25], [26], bipartite graph [27], [28], and pairwise terms [29], [30]. To reduce computation complexity, some employ a hierarchical approach [31], [32], or temporal sliding windows [24], [33].
Multi-target multi-camera tracking. Multi-camera monitoring receive wide attention from researchers [34]- [40]. Specifically, multi-target multi-camera tracking (MTMCT) tracks targets across cameras [12], [41]- [45], and assumes no overlapping field-of-view across cameras. Similar to MOT systems, MTMCT systems also follow the tracking-by-detection paradigm. For affinity estimation, most existing works also directly adopt the CNN feature distance trained from re-ID systems [2], [12], [45]. With that said, in contrast to MOT problems, targets in MTMCT systems can appear in more than one camera/scenario, leading to a potential misfit between the affinity and the association problem sizes. Speaking of association algorithms, given the huge problem size, existing works on MTMCT usually solve it in a hierarchical manner.
Specifically, first, single camera tracking formulates within camera trajectories. Second, cross camera tracking link the trajectories across cameras. Temporal windows are usually adopted to further restrict the problem size, so as to maintain a manageable problem size. In order to solve it in an online manner, Yoon et al. [43] formulate the problem as trackhypothesis trees and solve it via multiple hypothesis tracking algorithms. On the other hand, offline methods [1], [12], [41], [44] employ batch optimization techniques for higher accuracy, which is similar to MOT trackers. For example, in [42], Maksai et al. propose a global optimization method via a non-Markovian problem formulation. Tesfaye et al. provide a quadratic optimization formulation with constrained dominant sets clustering techniques [41]. Vehicle MTMCT is also studied. Tang et al. [46] use multiple cues to accommodate the similar appearance, heavy occlusion, and large viewing angle variation in vehicle tracking.
Re-identification. The affinity or similarity for data association in tracking are also studied [47]- [51]. With that said, most recent tracking systems adopts re-identification (re-ID) distance for similarity estimation. Re-ID systems focus on retrieving all targets of the same identity across cameras. CNN based methods achieve very high accuracy in pedestrian re-ID [9], [15], [52], [53]. Ye et al. [53] provide a detailed investigation to state-of-the-art re-ID methods and their application. Multiple loss functions are proposed towards training better re-ID models, such as the contrastive loss [54] and triplet loss [55]- [57]. Hermans et al. investigate training techniques and propose hard negative mining [16] for triplet loss. Zhong et al. propose random erasing as a data augmentation method to enrich the database [58]. Wang et al. investigates deep hidden attributes for further performance increases. Vehicle re-ID, on the other hand, also attracts much attention [2], [59], [60]. Compared to the pedestrian counterpart, vehicle re-ID exhibits additional challenges as the targets might look very similar. Video re-ID use tracklet feature to represent the video containing the target [61]. Spatial-temporal cues also help re-ID [62]- [65], but using them do not change the global matching nature of re-ID.
Departing from existing works, this paper studies the intrinsic dissimilarities between MTMCT and re-ID. Instead of network architectures or tracker designs, we propose a new training data sampling method, which adapts global re-ID features to affinity metrics that suit local matching in MTMCT data associations.

III. AFFINITY AND ASSOCIATION
In this section, we first introduce affinity estimations and data associations in MTMCT. Then, we design experiments to verify the mismatch in problem sizes between re-ID feature distances and MTMCT data associations.

A. Affinity Estimation
Similar to many previous works [2], [12], [44], [45], we first calculate the affinity a i,j from re-ID feature distances, where f i and f j denote the re-ID feature for target i and target j, respectively. dist (·, ·) denotes the distance function, where we choose the Euclidean distance. thres denotes the threshold for assuming the data pair as of the same identity. We denote data pairs with different identity as negative, and data pairs of the same identity as positive. Following Ristani et al. [1], we choose thres = µn+µp 2 , where µ p and µ n denote the average feature distance of positive and negative data pairs, respectively. We calculate µ p and µ n from all possible data pairs following the global retrieval task of re-ID. In this manner, positive data pairs should have positive affinity scores, and negative data pairs should have negative affinity scores.

B. Data Association
Similar to previous works [1], [2], [12], we conduct the data association for MTMCT in a hierarchical and iterative manner. Hierarchical means that the detection bounding boxes are first connected into within camera trajectories in the single camera tracking (SCT) step, and then these within camera trajectories are linked across multiple cameras in the multiple camera tracking (MCT) step. Iterative means that the data association problem sizes are limited to the size of temporal windows. Such temporal windows include detection bounding boxes within a single camera for SCT, and within camera trajectories for MCT.
With a limited association problem size, we then conduct the graph optimization problem for the association. Specifically, we create a graph with the targets as nodes, and their affinity estimations (Eq. 1) as weighted edges. For all targets within a temporal window in either SCT or MCT, we optimize the following problem, where x i,j ∈ {−1, 1} is an indicator for whether target i and target j are of the same identity. Given perfect affinity estimations (positive affinity scores for all data pairs of the same identity and negative scores otherwise), maximizing Eq. 2 should return perfect association results. On the other hand, with affinity estimations of undesired quality, the graph optimization also struggles, clearly showing the importance of a good affinity estimation.

C. Verifying the Mismatch
The re-ID feature distance specializes in the global matching problem in re-ID, which deals with all possible appearance changes at the same time. On the other hand, data associations in MTMCT are usually hierarchical and iterative, which limits the matching scopes and reduces the number of possible appearance changes each time. In this paper, we argue that the global re-ID feature distance as affinity is not the best choice for local matching in MTMCT, and design experiments to verify this phenomenon.
Preliminary experiment design. To evaluate the affinity estimation, we skip the detection step and the graph optimization step. We consider the ground truth bounding boxes in the validation partition, which are not accessible during the re-ID feature learning. We use this preliminary experiment setting in Fig. 2, Table I We also report the overall success (True = TP + TN) and failure (False = FP + FN) rates.
Results. We compare the affinity score distribution from different matching scopes in re-ID, MCT, and SCT. From re-ID to MCT to SCT, the matching scopes become smaller and smaller (re-ID considers targets from all cameras and all times; MCT considers targets across multiple cameras within a longer temporal window; SCT only considers targets within the same camera inside a shorter temporal window).
In Fig. 2, we show normalized distributions of positive pairs and negative pairs. Going from re-ID to MCT to SCT, as the problem sizes decrease, the distributions of affinity scores in Eq. 1 move towards the right-hand side, indicating higher affinity values on average and more false positives. The global re-ID distance based affinity score (Eq. 1) do a great job in separating the positive and negative pairs in re-ID problems, but are not as effective in the MCT and SCT problems (cannot effectively distinguish the positive and negative pairs in those scenarios). For example, for SCT, within each camera, re-ID features of different identities are still relatively similar [66], leading to false positives. For MCT, since the temporal sliding windows only include targets from neighboring cameras, the re-ID features also have limited diversities, and can lead to high false positive rates. This is further proven by the statistics of the positive and negative pairs in different matching problems in Table I. Going from re-ID to MCT to SCT, as the matching scopes decrease, using the same global re-ID distance based affinity score (Eq. 1) leads to a higher failure rate in differentiating the positive and negative pairs. Specifically, we witness stable false negatives and a lot more false positives in data associations for MTMCT (SCT and MCT).
From Fig. 2 and Table I, we verify that directly using re-ID feature distance as affinity might not be the best choice for data associations in MTMCT. We find the global re-ID distance failed to do as good a job in differentiating the positive and negative pairs in SCT and MCT as in re-ID. This is not a problem of re-ID features, since the re-ID problem is essentially different from MTMCT in terms of matching scopes, which requires the feature to be robust and can deal with all appearance changes at once. Even if we adopt re-ID features with higher performance, their higher overall robustness does not necessarily translate into being more discriminative in data associations in MTMCT (see Section V-E and Table VI for more details). Instead, what we need is affinity estimations more suitable for the data association problems in MTMCT.
The mismatch and its non-significance in MOT problems. It is noteworthy that the problem scale mismatch between re-ID features and MTMCT data associations is less pronounced in multiple object tracking (MOT) problems, where the target only appears in a single camera. Re-ID features learned on MOT data only learns to deal with limited appearance changes within a single camera and a short time period, which are of similar levels to that in the MOT data association problem. In this case, the problem scales between re-ID feature learning and MOT data association are already very similar, and there does not exist a significant mismatch between the matching scopes.

IV. METHOD
In this section, in order to tailor affinity estimations for corresponding data association problem sizes, we present a simple-yet-effective adaptive affinity module for data associations in MTMCT. The key contribution lies in the proposed data sampling schemes for learning such adaptive affinity. To highlight the benefit of our data sampling scheme, we adopt existing metric learning techniques, which also makes the overall method easy-to-apply.

A. Intra-Camera Metric and Inter-Camera Metric
Learned from all cameras, the re-ID features possess strong identification ability. However, as mentioned in Section III-C, directly adopting global re-ID feature distance might not be the best choice for local matching problems in MTMCT. To benefit from the strong ability of re-ID features while tailoring them to fit the smaller matching scopes in tracking, we learn metric networks on top of re-ID features for affinity estimation.
Specifically, we learn an intra-camera metric for associations in SCT and an inter-camera metric for associations in MCT, respectively (Fig. 3). Different form re-ID feature distance that focuses on large matching scopes (all data pairs), the proposed adaptive affinity focuses on smaller matching scopes in MCT or SCT (data pairs within temporal windows).  Matching scopes in tracking are usually smaller (Section III-B). Global metric from re-ID might have limited performance due to its slack decision boundary, and potentially introduces more false positives (Section III-C). Adaptive affinities can learn tighter decision boundaries, and should be a better fit for local matching in tracking. The proposed adaptive affinity module learns (C) an intra-camera metric for SCT and (D) an inter-camera metric for MCT. Similar to their corresponding matching scopes, the former is learned from neighboring frames in the same camera, and the latter is learned on tracklet pairs across neighboring cameras. In this manner, we make sure that the affinity estimations are tailored for the association matching scopes.
To adapt re-ID features to smaller matching scopes, we train the metric network with data samples that might appear in corresponding association problems in tracking. To this end, we sample data pairs with temporal windows from SCT and MCT data associations and train intra-camera metric and intercamera metric for SCT and MCT, respectively.
During training, we select positive/negative pairs with a 1 : 1 ratio for data balance, and feed the absolute difference vector f = |f i − f j | into the metric network as input.
Intra-camera metric. For data associations in SCT, we train an intra-camera metric to provide similarity estimation between data pairs. In training, we sample data pairs within a small temporal duration of τ S within each camera.
Inter-camera metric. For data association in MCT, we train an inter-camera metric to provide similarity estimation between single camera trajectories. Specifically, for positive data pairs, within the τ M -sized window, we choose from targets of the same identity but from different cameras; for negative data pairs, within the temporal sampling window τ M , we choose randomly from all cameras.
Sampling window lengths are critical hyper-parameters in the proposed adaptive affinity module, and are set differently 1) for different datasets and 2) for SCT and MCT in the same dataset. To achieve the best fit, during training, we set single camera sampling window length τ S and multi-camera sampling window length τ M to the same as that of temporal sliding windows adopted in SCT and MCT data associations, respectively. In this manner, we can prepare the Siamese metric networks with the exact amount of appearance changes they might encounter in data associations. In practice, we set the data sampling windows lengths as the temporal sliding window lengths, whereas the latter is usually set as the average traveling time inside a single camera (for SCT) or across multiple cameras (for MCT) [2], [12]. In Section V, we further study the influence of different sampling window lengths.

B. Siamese Metric Network
For the proposed adaptive affinity module, we replace re-ID feature distances with affinity scores estimated by a Siamese metric network [13]. Given the absolute difference between the data pairs, the network learns a binary classifier with 3 hidden layers. The metric network outputs a 2-dim probability distribution p ij = p − ij , p + ij , where p − ij and p + ij encode the possibility of the input pair being of different identities or the same identity, respectively. The affinity score for the proposed metric is computed by, This affinity value should be positive if the data pair belongs to the same identity, and negative if otherwise. During training, the re-ID feature extractor is fixed, and only the metric network is updated with a binary crossentropy loss. During testing, neural network classifiers can easily get over confident [67] with the prediction, with then turns the affinity to essentially either −1 or 1 in most scenarios. However, such overconfidence can cause trouble for the graph optimization algorithm, as all positives and negatives are treated equally. For this reason, we exert a temperature scaling factor of 0.1 onto the softmax layer, so as to prevent overconfident outputs.

C. Discussion
Preliminary experiments on the effectiveness of adaptive affinity. We show that the proposed adaptive appearance module can better address the mismatch between affinity and association. In Fig. 4 (which follows the same experiment design as Fig. 2 and Table I. See Section III-C for more details), we show the matching errors during SCT and MCT under a similar setting as Fig. 2 and Table I. Using the proposed inter and intra camera metrics, when compared to the global re-ID distances, we can reduce the false positives significantly while maintaining a similar level of false negatives.  Fig. 4: Matching error comparison between the global re-ID metric, "global metric" that learns the Siamese metric with global sampling, and the proposed adaptive appearance metric (intra-camera metric and inter-camera metric). We report false positives and false negatives on DukeMTMC validation set.
Main contributors to performance increases. In addition, we show that the improvements come from the focus on the problem size (using temporal sampling window) rather than applying an additional metric network for similarity estimation. To this end, we create "global metric", one that adopts the same Siamese network and training mechanism as the proposed adaptive affinity metrics, but without the temporal sampling window. For the global metric, we adopt global data sampling, i.e., setting the temporal window length to infinity and allowing data from all cameras for both positive pairs and negative pairs. Between "global metric" and "re-ID distance", we witness no significant differences in either false positives or false negatives, indicating the Siamese metric network architecture makes little difference. As such, the drop in false positives should come from the usage of temporal data sampling, which to a certain extent validates that similarity metrics should follow guidance from matching scopes.
Extreme cases. First, under extremely low frame rates, unless they are returning, each target will only appear in one camera once. In this case, SCT is no longer needed and intracamera metric will not be useful. However, since the trajectory continuity still holds, the locality in MCT associations will not be influenced. Thus, the inter-camera metric is still useful. Second, the scenario can be in open topology, i.e., targets travel to all cameras at the same probability. This time, the inter-camera metric will fall back to the global metric. However, the SCT associations are still local, and thus the intra-camera metric remains useful.

V. EXPERIMENT A. Datasets and Evaluation Protocol
CityFlow [2] is a vehicle tracking dataset over an entire city. Specifically, we use the AI-City-2019 challenge MTMCT (track-1) and evaluate on its online test set. CityFlow has a relatively low frame rate (10fps), severe occlusion, and fastmoving vehicles from 40 cameras, spanning over 2km.
DukeMTMC [1] is a pedestrian tracking dataset that includes 1080p 60fps videos from 8 cameras on a school campus. Due to some reasons, the 35-minute test partition is no longer available on the MOTchallenge website [17]. We report some results tested on the test partition online and the others on the selected validation partition. Specifically, we use the first 40 minutes of the training set to train the re-ID feature extractors and affinity Siamese metrics, and the remaining 10 minutes as the validation set. Evaluation protocol. For MTMCT, following [1], we use IDF1, IDP, and IDR as evaluation metrics. IDF1 is the ratio of correctly identified detections over the average number of ground-truths and computed detections. IDP (IDR) is the fraction of computed detections (ground truth detections) that are correctly identified. As IDF1 considers both false negatives (considered in IDR) and false positives (considered in IDP), we use it as the main evaluation criterion. For MOT, we adopt MOTA (multiple object tracking accuracy) as the main criterion following the CLEAR metric [68]. We also report MT (mostly tracked), ML (mostly lost), and IDs (ID switches). For re-ID, we adopt the rank-1 accuracy and mean average precision (mAP) [9] evaluation protocol.
Variants and notations. As shown in Table II, "re-ID distance" uses the Euclidean distance between feature pairs as in Eq. 1 for affinity. The other variants adopt the Siamese metric score Eq. 3 for similarity estimation. Similar to experiments in Fig. 4, "global metric" uses the same Siamese metric network but samples training data pairs randomly from all cameras and all times. "adaptive affinity" is the proposed full system.

B. Implementation Details
Object detector performance greatly influences the overall tracking-by-detection system [69]. As such, to minimize the influence of different detectors and examine the potential performance increase from our adaptive affinity module, we directly adopt the provided detection results by the datasets: SSD [6] for CityFlow, and OpenPose [70] for DukeMTMC.
Re-ID features. On DukeMTMC, we inspect three globally learned re-ID features, namely, the ID-Discriminative Embedding (IDE) [9], the triplet feature [16], and the Part-based Convolutional Baseline (PCB) [15], all of which use ResNet-50 [71] pre-trained on ImageNet [72] as the backbone. In the following experiments on DukeMTMC, we use the IDE feature in our tracker unless otherwise specified.
On CityFlow, we use a DenseNet-121 [73] based re-ID feature with both softmax and triplet loss.
MTMCT tracker. We adopt DeepCC [12] as our MTMCT tracker, which adopts hierarchical and iterative associations. For hierarchical association, detection bounding boxes are first grouped into tracklets, then into single camera trajectories in SCT. Lastly, in MCT, single camera trajectories are linked across cameras. For iterative association, temporal sliding windows (either within camera or across cameras) are adopted. On DukeMTMC, each tracklet has 40 frames. The temporal Methods with * include additional data in training, thus yielding higher performance. Method with † escapes from the tracking-by-detection paradigm and also adopts video object tracking as reference. The proposed adaptive affinity method yields substantial accuracy increase over global re-ID distance and global metric as affinity estimations. sliding window lengths for SCT and MCT are 600 frames and 2, 400 frames, respectively. On CityFlow, we set the tracklet length to 10 frames. Temporal sliding windows for SCT and MCT are set to 150 frames and 500 frames, respectively. µ p and µ n are calculated from the training set in both datasets. Siamese metric learning. We train the Siamese metric network with a learning rate of 1 × 10 −3 for the 40 epochs. We adopt the Cosine learning rate scheduler for its fast convergence [74]. A cross-entropy loss and a batch size of 64 are adopted in training. Sampling window lengths are set the same as corresponding matching windows lengths in either SCT or MCT. On DukeMTMC, we set τ S = 600 and τ M = 2, 400. On CityFlow, we set τ S = 150 and τ M = 500.

C. Evaluation of the Tracker
As shown in Table III and Table V, using global re-ID feature distances as affinities, we achieve competitive performance on both CityFlow and DukeMTMC dataset. On CityFlow, using the provided SSD [6] detection results and only the provided data for training, "re-ID distance" achieves 56.6% MCT IDF1, lagging behind the top-performing teams in the AI-City-2019 challenge that either adopt private detector or use additional data in re-ID feature learning. On DukeMTMC, using the provided OpenPose [70] detection results [12], "re-ID distance" achieves 91.3% and 87.4% for SCT and MCT IDF1 on test (easy); and 83.7% and 75.4% for SCT and MCT IDF1 on test (hard), outperforming the previous methods.

D. Evaluation of the Adaptive Affinity Module
Improvements over the global re-ID distance. We first compare the proposed adaptive affinity module against traditional global re-ID feature distance. Results on CityFlow and DukeMTMC are shown in Table III, Table V, and Table VI. Going from "re-ID distance" to "adaptive affinity", we witness consistent and non-trivial improvements on the two datasets. On CityFlow, the proposed adaptive affinity improves MCT IDF1 by +6.4%. On DukeMTMC test (hard), our method excels the re-ID distance baseline by +2.1% for SCT and +6.9% for MCT in terms of IDF1. On the DukeMTMC validation set, "adaptive affinity" improves the MCT IDF1 results by +2.4%, +3.6%, and +2.3% using IDE, triplet and PCB features, respectively. Such improvements are coherent with our preliminary experiments on matching errors in Fig. 4, where the intra-camera metric for SCT association and intercamera metric for MCT association exhibit lower false positive ratios while maintaining similar false negative ratios. Overall, the MTMCT experiments and results demonstrate the effectiveness of the proposed adaptive affinity.
In the MOT scenarios, however, we find the adaptive affinity of limited use in our preliminary experiments. We use MOT17 [17] as the training set and MOT15 [17] as the testing set (we use the training partition that is publicly available) and JDE [69] as the tracker. As shown in Table IV, we find the adaptive affinity does not bring significant improvements over the global re-ID distance baseline. In terms of MOTA, adaptive affinity brings a -0.1% overall decrease; in terms of IDF1, adaptive affinity brings a +0.6% overall increase. Coherent with our analysis in Section IV-C, targets in MOT scenarios only appear within a single camera and a short time period, which makes the re-ID features learned in MOT scenarios already suitable to the problem scale in MOT data associations. In comparison, for MTMCT scenarios, adaptive affinity brings performance boosts because it can bridge the mismatch global re-ID features (all cameras at all time) and MTMCT data associations (single camera or multiple cameras but within a short time period). Since the mismatch is less pronounced in MOT scenarios, the proposed adaptive affinity is less effective.
For multiview scenarios (multiple cameras with overlapping fields-of-view focused on the same scenario), when jointly considering multiview information, the problem scales of re-ID feature learning and data association are similar. On Campus and Shelf datasets [85], since the targets have large appearance disparities and jointly considering multiple cameras well addresses the occlusion issue, existing methods achieve very high tracking results (e.g., MOTA and IDF1 both around 98% from [86]). Due to the same reasons, we do not report quantitative results on these two datasets, since we believe adaptive affinity would not bring significant improvements. Moreover, like single-view MOT scenarios, adaptive affinity is also less effective in this scenario, as the problem scales are similar.
Comparison with the state-of-the-art methods. The proposed "adaptive affinity" further improves performance for our tracker and achieves competitive performance on  CityFlow while reaching new state-of-the-art on DukeMTMC. On CityFlow, our tracker with adaptive affinity module reaches top-6 using the provided SSD detector. Note that we cannot achieve as competitive results as some of the challenge participants. This is because the test participants focus on building a competitive tracker and adopt various techniques, e.g., using more training data, modeling of vehicle motion and road network in that city, cropping road-side vehicles with prior knowledge, and including single-object tracking results for cross-reference. On the other hand, this work focuses on improving the affinity metric to fit the data association in MTMCT, and is parallel to the mentioned works on building a stronger tracker. On DukeMTMC, using the provided OpenPose [70] detector, on test (easy), we obtain 92.5% and 88.6% IDF1 on SCT and MCT, respectively. These numbers are +0.7% and +1.8% higher than previous state-of-the-art [84]. On DukeMTMC test (hard), our IDF1 scores are 85.8% and 82.3% on SCT and MCT, which translate into a tie and a +1.0% improvement, respectively. Main contributors to performance increases. There are two possible sources of the improvements in adaptive affinity: the Siamese metric network, and temporal data sampling. Siamese metric network learns affinity using neural network layers, which possibly has an edge over the Euclidean distance for affinity. Temporal data sampling, on the other hand, focuses on the mismatch between affinity and the association problem size, where we believe exist a mismatch. To verify the source of improvements and our mismatch arguments, we examine the tracking performances from the "global metric" variant, which samples training data globally from all cameras at all times. On CityFlow and DukeMTMC, global metric cannot bring constant and significant performance increases. For example, global metric brings a mere +0.4% MCT IDF1 improvement on CityFlow dataset, but fails to improve SCT IDF1 on DukeMTMC test sets. On DukeMTMC validation, global metric sometimes gives performance increases for SCT or MCT, but also sometimes leads to performance drops. Overall, the performance differences between global Siamese metric and global re-ID distance are not constant and significant. which agrees with the finding in our preliminary experiments in Fig. 4. This leaves the temporal data sampling the source of performance increase, which is also supported by the tracking performance increase. On CityFlow, adaptive affinity increases performance by +5.9% over global metric. On DukeMTMC validation, adaptive affinity also brings consistent performance improvements over global metric, e.g., +2.0% SCT IDF1 and +2.2% MCT IDF1 using IDE features. In summary, these results verify the temporal data sampling as the source of improvements, further validating the mismatch between global affinities and the local matching nature in SCT and MCT.

E. Variants and Ablation Study
Necessity of intra and inter camera metric. In Table III  and Table VI, we replace the intra and inter camera metric with the global metric, and find both are necessary. When replacing intra-camera metric with the global metric, IDF1 drops by -4.5%, -2.0%, and -1.3% on CityFlow, DukeMTMC SCT, and DukeMTMC MCT (both with IDE features), respectively. As SCT and MCT are conducted in an orderly fashion, changing intra-camera metric for SCT can also lead to performance differences for MCT. A similar but smaller accuracy drop can be observed when the inter-camera metric is replaced with the global metric. Specifically, this leads to a -1.8% IDF1 drop on CityFlow, and -0.1% and -0.7% for SCT and MCT on DukeMTMC (using IDE as re-ID features). Actually, when changing the inter-camera metric for MCT, the slight performance drops in SCT are due to that some targets returning to the same camera after a long-time departure (much longer than considered problem size in SCT), and is only considered in MCT. The removal of the intra-camera metric causes a larger accuracy drop. This is because the problem size differences between MCT and re-ID is smaller compared to that of SCT and re-ID. Overall, these results show that both the intra-camera and inter-camera metrics are necessary components in our system. intra-camera metric sampling window length (frames) MCT IDF1 results re-ID distance global metric adaptive affinity Fig. 5: Influence of data sampling window lengths for intra and inter camera metrics on DukeMTMC validation set (using IDE features). Red dots highlight the corresponding temporal sliding window lengths for data associations in SCT and MCT, respectively. When data sampling window lengths are set to the same as temporal sliding window lengths, a best match between affinity estimation and data association is achieved, providing highest results.
Temporal sampling window lengths. In Fig. 5, we evaluate the impact of different data sampling window lengths. Too short or too long a temporal window will affect SCT and MCT results. Too short a sampling window may significantly reduce the choices of training pairs, and the Siamese metric network cannot learn to deal with the possible appearance changes in association. On the other hand, too long a sampling window can include too many possible training pairs, and a large part of which might not really appear in the association problem. Adaptive affinity achieves best results in both SCT and MCT when sampling windows are set to the same size as the temporal sliding windows in data associations in SCT and MCT (600 and 2, 400), respectively. In this manner, the adaptive affinity modules learn to focus exactly on the possible appearance changes that might appear in associations.
Impact of different re-ID features. Tracking accuracy with different re-ID features is summarized in Table VI. Under both the SCT and MCT task, we find that the tracking performance of IDE, triplet, and PCB features similar. This finding is consistent with a previous report [12]: improvement in re-ID accuracy can have a diminishing improvement on the MTMCT system. The main reason is that the re-ID and MTMCT are two different problems in terms of matching scopes. Re-ID (global matching) deals with all possible appearance variations at once, requiring the features to be overall robust. MTMCT (local matching) deals with limited appearance changes in both SCT and MCT, and the more robust re-ID feature (higher re-ID performance) might not necessarily translate into being more discriminative in the matching problems in MTMCT. For example, in MCT, the matching scope within a temporal sliding window might have dozens of images, while that in re-ID has over 10k images. Within a much smaller matching scope, there is less requirement on feature's discriminative ability, and PCB would have a similar matching accuracy with IDE. Moreover, MTMCT also has several other components besides feature-based matching. Imperfectness in other components reduces the improvement brought about by the re-ID features.
Computation complexity. The metric network takes 20 minutes to train using one GTX 1080ti GPU. During testing, CNN features are extracted with GPU, and affinities the associations are computed on a 3.2Ghz Intel Xeon CPU. Overall, on DukeMTMC, the total tracker run time is 1,464 seconds using re-ID feature distances, and 1,537 seconds using the adaptive affinity (an acceptable 5% increase).

VI. CONCLUSION
This paper points out a previously overlooked problem in MTMCT: global re-ID feature distances might not be the best affinity estimation for local matching in either SCT or MCT. We design experiments to verify such misfit, and propose a simple-yet-effective adaptive affinity module for different data associations in MTMCT. Specifically, rather than trying to solve all possible appearance changes, we tailor the affinity metric to focus only on ones that might emerge in data associations in SCT or MCT. With temporal windows originally used for data associations, we introduce a new data sampling method for affinity metric learning. The proposed adaptive affinity introduces significant performance improvements on multiple datasets. In future works, we would like to investigate automatic tuning of sampling window lengths for further improvements.