{'fraction_non_alphanumeric': 0.027723406526922668, 'fraction_numerical': 0.0070109256557172855, 'mean_word_length': 4.624499599679744, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Ego2Top: Matching Viewers in Egocentric and Top-view Videos

Egocentric cameras are becoming increasingly popular and provide us with large amounts of videos, captured from the first person perspective. At the same time, surveillance cameras and drones offer an abundance of visual information, often captured from top-view. Although these two sources of information have been separately studied in the past, they have not been collectively studied and related. Having a set of egocentric cameras and a top-view camera capturing the same area, we propose a framework to identify the egocentric viewers in the top-view video. We utilize two types of features for our assignment procedure. Unary features encode what a viewer (seen from top-view or recording an egocentric video) visually experiences over time. Pairwise features encode the relationship between the visual content of a pair of viewers. Modeling each view (egocentric or top) by a graph, the assignment process is formulated as spectral graph matching. Evaluating our method over a dataset of 50 top-view and 188 egocentric videos taken in different scenarios demonstrates the efficiency of the proposed approach in assigning egocentric viewers to identities present in top-view camera. We also study the effect of different parameters such as the number of egocentric viewers and visual features.


Introduction
The availability of large amounts of egocentric videos captured by cellphones and wearable devices such as GoPro cameras and Google Glass has opened the door to a lot of interesting research in computer vision [2,3,4]. At the same time, videos captured with top-down static cameras such as surveillance cameras in airports and subways, unmanned aerial vehicles (UAVs) and drones, provide us with a lot of invaluable information about activities and events taking place at different locations and environments. Relating these two complementary, but drastically different sources of visual information can provide us with rich analytical power, and help us explore what can not be inferred from each of these sources taken separately. Establishing such a relationship can have several applications. For arXiv:1607.06986v2 [cs.CV] 13 Aug 2016 We aim to answer the two following questions: 1) Does this set of egocentric videos belong to the viewers visible in the top-view video? 2) Assuming they do, which viewer is capturing which egocentric video? example, athletes can be equipped with body-worn cameras, and their egocentric videos together with the top-view videos can offer new data useful for better technical and tactical sport analysis. Moreover, due to the use of wearable devices and cameras by law enforcement officers, finding the person behind an egocentric camera in a surveillance network could be a useful application. Furthermore, fusing these two types of information can result in better 3D reconstruction of an environment by combining the top-view information with first person views.
The first necessary step to utilize information from these two sources, is to establish correspondences between the two views. In other words, a matching between egocentric cameras and the people present in the top-view camera is needed. In this effort, we attempt to address this problem. More specifically, our goal is to localize people recording egocentric videos, in a top-view reference camera. To the best of our knowledge, such an effort has not been done so far. In order to evaluate our method, we designed the following setup. A dataset containing several test cases is collected. In each test case, multiple people were asked to move freely in a certain environment and record egocentric videos. We refer to these people as ego-centric viewers. At the same time, a top-view camera was recording the entire scene/area including all the egocentric viewers and possibly other intruders. An example case is illustrated in Figure 1.
Given a set of egocentric videos and a top-view surveillance video, we try to answer the following two questions: 1) Does this set of egocentric videos belong to the viewers visible in the top-view camera? 2) If yes, then which viewer is capturing which egocentric video? To answer these questions, we need to compare a set of egocentric videos to a set of viewers visible in a single topview video. To find a matching, each set is represented by a graph and the two graphs are compared using a spectral graph matching technique [5]. In general, this problem can be very challenging due to the nature of egocentric cameras. Since the camera-holder is not visible in his own egocentric video leaving us with no cues about his visual appearance. The input to our framework is a set of egocentric videos (in this case 5 videos), and one top-view video. The goal is defined as assigning the egocentric videos to the people recording them. A graph is formed on the set of egocentric videos (each node being one of the egocentric videos), and the other graph is formed on the top-view video (each node being one of the targets present in the video). Using spectral graph matching, a soft assignment is found between the two graphs, and using a soft-to-hard assignment, each egocentric video is assigned to one of the viewers in the top-view video. This assignment is our answer to the second question in 1.
In what follows we briefly mention some challenges concerning this problem and sketch the layout of our approach.
In order to have an understanding of the behavior of each individual in the top-view video, we use a multiple object tracking method [6] to extract the viewer's trajectory in the top-view video. Note that an egocentric video captures a person's field of view rather than his spatial location. Therefore, the content of a viewer's egocentric video, a 2D scene, corresponds to the content of the viewer's field of view in the top-view camera. For the sake of brevity, we refer to a viwer's top-view field of view as Top-FOV in what follows. Since trajectories computed by multiple object tracking do not provide us with the orientation of the egocentric cameras in the top-view video, we employ the assumption that for the most part humans tend to look straight ahead and therefore shoot videos from the visual content in front of them. Note that this is not a restrictive assumption as most ego-centric cameras are body worn (Please see Figure 4). Having an estimate of a viewer's orientation and Top-FOV, we then encode the changes in his Top-FOV over time and use it as a descriptor. We show that this feature correlates with the change in the global visual content (or Gist) of the scene observed in his corresponding egocentric video.
We also define pairwise features to capture the relationship between two egocentric videos, and also the relationship between two viewers in the top-view camera. Intuitively, if an egocentric viewer observes a certain scene and another egocentric viewer comes across the same scene later, this could hint as a rela- tionship between the two cameras. If we match a top-view viewer to one of the two egocentric videos, we are likely to be able to find the other viewer using the mentioned relationship. As we experimentally show, this pairwise relationship significantly improves our assignment accuracy. This assignment will lead to defining a score measuring the similarity between the two graphs. Our experiments demonstrate that the graph matching score could be used for verifying if the top-view video is in fact, capturing the egocentric viewers (See the diagram shown in Figure 7a).
The rest of this work is as follows. In section 2, we mention related works to our study. In section 3, we describe the details of our framework. Section 4 presents our experimental results followed by discussions and conclusions in Section 5.

Related Work
Visual analysis of egocentric videos has recently became a hot topic in computer vision [7,8], from recognizing daily activities [3,2] to object detection [9], video summarization [10], and predicting gaze behavior [11,12,13]. In the following, we review some previous work related to ours spanning Relating static and egocentric, Social interactions among egocentric viewers, and Person identification and localization.
Relating Static and Egocentric Cameras: Some studies have addressed relationships between moving and static cameras. Interesting works reported in [14,15] have studied the relationship between mobile and static cameras for the purpose of improving object detection accuracy. [16] fuses information from ego-centric and exocentric vision (other cameras in the environment) and laser depth range data to improve depth perception in 3D reconstruction. [17] predicts gaze behavior in social scenes using first-person and third-person cameras.
Social Interactions among Egocentric Viewers: To explore the relationship among multiple egocentric viewers, [18] combines several egocentric videos to achieve a more complete video with less quality degradation by estimating the importance of different scene regions and incorporating the consensus among several egocentric videos. Fathi et al., [19] detect and recognize the type of social interactions such as dialogue, monologue, and discussion by detecting human faces and estimating their body and head orientations. [20] proposes a multi-task clustering framework, which searches for coherent clusters of daily actions using the notion that people tend to perform similar actions in certain environments such as workplace or kitchen. [21] proposes a framework that discovers static and movable objects used by a set of egocentric users.
Person Identification and Localization: Perhaps, the most similar computer vision task to ours is person re-identification [22,23,24]. The objective here is to find the person present in one static camera, in another overlapping or nonoverlapping static camera. However, the main cue in human re-identification is visual appearance of humans, which is absent in egocentric videos. Tasks such as human-identification and localization in egocentric cameras have been studied in the past. [25] uses the head motion of an egocentric viewer as a biometric signature for determine which videos have been captured by the same person. [26] identifies egocentric observers in other egocentric videos, using their head motion. Relating geo-spatial location to user shared visual content has also been explored. [4] localizes the field of view of an egocentric camera by matching it against a reference dataset of videos or images (such as Google street view), and [27] refines the geo-location of images by matching them against user shared images. Landmarks and map symbols are used in [28] to perform self localization on the map. [29] use semantic cues for spatial localization, and [30] uses location information to infer semantic information.

Framework
The block diagram in Figure 2 illustrates different steps of our approach. First, each view (ego-centric or top-down) is represented by a graph which defines the relationship among the viewers present in the scene. These two graphs may not have the same number of nodes as some the egocentric videos might not be available, or some individuals present in the top-view video might not be capturing videos. Each graph consists of a set of nodes, each of which represents one viewer (egocentric or top-view), and the edges of the graph encode pairwise relationships between pairs of viewers.
We represent each viewer in top-view by describing his expected Top-FOV, and in egocentric view by the visual content of his video over time. This descrip-tion is encoded in the nodes of the graphs. We also define pairwise relationships between pairs of viewers, which is encoded as the edge features of the graph (i.e., how two viewers' visual experience relate to each other).
Second, we use spectral graph matching to compute a score measuring the similarity between the two graphs, alongside with an assignment from the nodes of the egocentric graph to the nodes of the top-view graph.
Our experiments show that the graph matching score can be used as a measure of similarity between the egocentric graph and the top-view graph. Therefore, it can be used as a measure for verifying if a set of egocentric videos have been shot in the same environment captured by the top-view camera. In other words, we can evaluate the capability of our method in terms of answering our first question. In addition, the assignment obtained by the graph matching suggests an answer to our second question. We organize this section by going over the graph formation process for each of the views, and then describing the details of the matching procedure.

Graph Representation
Each view (egocentric or top-view) is described using a single graph. The set of egocentric videos is represented using a graph in which each node represents one of the egocentric videos, and an edge captures the pairwise relationship between the content of the two videos.
In the top-view graph, each node represents the visual experience of a viewer being tracked (in the top-view camera), and an edge captures the pairwise relationship between the two. By visual experience we mean what a viewer is expected to observe during the course of his recording seen from the top view.
3.1.1 Modeling the Top-View Graph: In order to model the visual experience of a viewer in a top-view camera, we need to have knowledge about his spatial location (trajectory) throughout the video. We employ the multiple object tracking method presented in [6] and extract a set of trajectories, each corresponding to one of the viewers in the scene. Similar to [6], we use annotated bounding boxes, and provide their centers as an input to the multiple object tracker. Our tracking results are nearly perfect due to several reasons: the high quality of videos, high video frame rate, and lack of challenges such as occlusion in the top-view videos.
Each node represents one of the individuals being tracked. Employing the general assumption that people often tend to look straight ahead, we use a person's speed vector as the direction of his camera at time t (denoted as θ t ). Further, assuming a fixed angle (θ d ), we expect the content of the person's egocentric video to be consistent with the content included in a 2D cone formed by the two rays emanating from the viewer's location and with angles θ − θ d and θ + θ d . Figure 4 illustrates the expected Top-FOV for three different individuals present in a frame. In our experiments, we set θ d to 30 degrees. In theory, angle θ d can be estimated more accurately by knowing intrinsic camera parameters such as focal length and sensor size of the corresponding egocentric camera. 3.1.2 Modeling the Egocentric View Graph: As in the top-view graph, we also construct a graph on the set of egocentric videos. Each node of this graph represents one egocentric video. Edges between the nodes capture the relationship between two egocentric videos.
Node Features: Similar to the top-view graph, each node is represented using two features. First, we compute pairwise similarity between GIST features [31] of all video frames (for one viewer) and store the pairwise similarities in the matrix U GIST Ei , in which the element U GIST Ei (f p , f q ) is the GIST similarity between frame f p and f q of egocentric video i. Two examples of such features are illustrated in the left column of Figure 5 (a). The GIST similarity is a function of the euclidean distance of the GIST feature vectors.
In which g Ei fp and g Ei fq are the GIST descriptors of frame f p and f q of egocentric video i, and γ is a constant which we empirically set to 0.5.
The second feature is a time series counting the number of seen people in each frame. In order to have an estimate of the number of people, we run a pre-trained human detector using deformable part model [32] on each egocentric frame. In order to make sure that our method is not including humans in far distances (which are not likely to be present in top-view), we exclude bounding boxes whose sizes are smaller than a certain threshold (determined considering an average human height of 1.7m and distance of the radius of the area being covered in the top view video.). Each of the remaining bouding boxes, has a detection score (rescaled into the interval [0 1]) which has the notion of the probability of that bounding box containing a person. Scores of all detections in a frame are added and used as a count of people in that frame. Therefore, similar to the top-view feature, we can represent the node E i of egocentric video i with a 1 × T Ei vector U n Ei (Examples shown in the bottom row of figure 6.). Edge Features: To capture the pairwise relationship between egocentric camera i (containing T Ei frames) and egocentric camera j (containing T Ej frames), we extract GIST features from all of the frames of both videos and form a T Ei × T Ej matrix B GIST ij in which B GIST ij (f p , f q ) represents the GIST similarity between frame f p of video i and frame f q of video j (Examples shown in the left column of Figure 5 (b).).

Graph Matching
The goal is to find a binary assignment matrix x N e ×N t (N e being the number of egocentric videos and N t being the number of people in the top-view video).
x(i, j) being 1 means egocentric video i is matched to viewer j in top-view.
To capture the similarities between the elements of the two graphs, we define the affinity matrix A N e N t ×N e N t . a ik,jl is the affinity of edge ij in the egocentric graph with edge kl in the top-view graph. Reshaping matrix x as a vector x N e N t ×1 ∈ {0, 1} N e N t , the assignment problem is defined as the following: We compute a ik,jl based on the similarity between the feature descriptor of edge ij in the egocentric graph B GIST ij and the feature descriptor for edge kl in the top-view graph B IOU kl .
As described in the previous section, each of these features is a 2D matrix.
where * denotes cross correlation. For the elements of A for which i = j and k = l, the affinity captures the compatibility of node i in the egocentric graph, to node k in the top-view graph. The compatibility between the two nodes is computed using 2D cross correlation between U IOU k and U GIST Ei and 1D cross correlation between U n k and U n Ei . The overall compatibility of the two nodes is a weighted linear combination of the two: where α is a constant between 0 and 1 specifying the contribution of each term.
In our experiments, we set α to 0.9. Figure 5 illustrates the features extracted from some of the nodes and edges in the two graphs.
Soft Assignment We employ the spectral graph matching method introduced in [5] to compute a soft assignment between the set of egocentric viewers and top-view viewers. In [5], assuming that the affinity matrix is an empirical estimation of the pairwise assignment probability, and the assignment probabilities are statistically independent, A is represented using it's rank one estimation which is computed by argmin p |A − pp T |. In fact, the rank one estimation of A is no different than it's leading eigenvector. Therefore, p can be computed either using eigen decompositon, or estimated iteratively using power iteration. Considering vector p as the assignment probablities, we can reshape p N e N t ×1 into a N e × N t soft assignment matrix P , for which after row normalization P (i, j) represents the probability of matching egocentric viewer i to viewer j in the top-view video.
Hard Assignment Any soft to hard assignment method can be used to convert the soft assignment result (generated by spectral matching) to the hard binary assignment between the nodes of the graphs. We used the well known Munkres (also known as Hungarian) algorithm to obtain the final binary assignment.

Experimental Results
In this section, we will mention details of our experimental setup and collected dataset, the measures we used to evaluate the performance of our method, and the performance of our proposed method alongside with some baselines.

Dataset
We collected a dataset containing 50 test cases of videos shot in different indoor and outdoor conditions. Each test case, contains one top-view video and several egocentric videos captured by the people visible in the top-view camera.
Depending on the subset of egocentric cameras that we include, we can generate up to 2,862 instances of our assignment problem (will be explained in more detail in section 4.2.4). Overall, our dataset contains more than 225,000 frames. Number of people visible in the top-view cameras varies from 3 to 10, number of egocentric cameras varies from 1 to 6, and the ratio of number of available egocentric cameras to the number of visible people in the top-view camera varies from 0.16 to 1. Lengths of the videos vary from 320 frames (10.6 seconds) up to 3132 frames (110 seconds).

Evaluation
We evaluate our method in terms of answering the two questions we asked.

Ranking Top-view Videos:
We design an experiment to evaluate if our graph matching score is a good measure for the similarity between the set of egocentric videos and a top-view video. Having a set of egocentric videos from the same test case (recorded in the same environment), and 50 different topview videos (from different test cases), we compare the similarity of each of the top-view graphs to the egocentric graph. After computing the hard assignment for each top view video(resulting in the assignment vector x), the score x T Ax is associated to that top-view video. This score is effectively the summation of all the similarities between the corresponding nodes and edges of the two graphs. Using this score rank all the top-view videos. The ranking accuracy is measured by measuring the rank of the ground truth top-view video, and computing the cumulative matching curves shown in figure 7(a). The blue curve shows the ranking accuracy when we compute the scores only based on the unary features. The red curve shows the ranking accuracy when we consider both the unary and pairwise features for performing graph matching. The dashed black line shows the accuracy of randomly ranking the top-view videos. It can be observed that both the blue and red curves outperform the random ranking. This shows that our graph matching score is a meaningful measure for estimating the similarity between the two graphs. In addition, the red curve, outperforming the blue curve shows the effectiveness of our pairwise features. In general, this experiment answers the first question. We can in fact use the graph matching score as a cue for narrowing down the search space among the top-view videos, for finding the one corresponding to our set of the egocentric cameras.

Viewer Ranking Accuracy:
We evaluate our soft assignment results, in terms of ranking capability. In other words, we can look at our soft assignment as a measure to sort the viewers in the top-view video based on their assignment probability to each egocentric video. Computing the ranks of the correct matches, we can plot the cumulative matching curves to illustrate their performance.
We compare our method with three baselines in figure 7 (b). First, random ranking (dashed black line), in which for each egocentric video we randomly rank the viewers present in the top-view video. Second, sorting the top-view viewers based on the similarities of their 1D unary features to the 1D unary features of each egocentric camera (i.e., number of visible humans illustrated by the blue curve). Third, sorting the top-view viewers based on their 2D unary feature (GIST vs. FOV, shown by the green curve). Note that here (the blue and green curves), we are ignoring the pairwise relationships (edges) in the graphs. The consistent improvement of our method (red curve) over the baselines, justifies the effectiveness of our representation, and shows the contribution of each stage.

Assignment Accuracy:
In order to answer the second question, we need to evaluate the accuracy of our method in terms of node assignment accuracy. Having a set of egocentric videos and a top-view video containing the egocentric viewers, we evaluate the percentage of the egocentric videos which were correctly matched to their corresponding viewer. We evaluate the hard-assignment accuracy of our method and compare it with three baselines in figure 7(c). First, random assignment (Rnd), in which we randomly assign each egocentric video to one of the visible viewers in the top-view video. Second, Hungarian bipartite matching only on the 1D unary features denoted as H. Third, Hungarian bipar- Fig. 7: (a) shows the cumulative matching curve for ranking top-view videos. The blue curve shows the accuracy achieved only using the node similarities. Red is the accuracy considering both node and edge similarities in the graph matching. (b) shows the cumulative matching curve for ranking the viewers in the top-view video. The red, green and blue curves belong to ranking based on spectral graph matching scores, cross correlation between only the 2D, and only the 1D unary scores, respectively. The dashed black line shows random ranking accuracy (c) shows the assignment accuracy based on random assignment, using the number of humans, using unary features, and using spectral graph matching.
tite matching only on the 2D unary feature (GIST vs. FOV, denoted as G-F), ignoring the pairwise relationships (edges) in the graphs. The consistent improvement of our method using both unary and pairwise features in graph matching (denoted as GM) over the baselines shows the significant contribution of pairwise features in the assignment accuracy. As a result, the promising accuracy acquired by graph matching answers the second question. Knowing a top-view camera is capturing a set of egocentric viewers, we can use visual cues in the egocentric videos and the top-view video, to decide which viewer is capturing which egocentric video.

Effect of Number of Egocentric Cameras:
In sections 4.2.2 and 4.2.3, we evaluated the performance of our method given all the available egocentric videos present in each set as the input to our method. In this experiment, we compare the accuracy of our assignment and ranking framework as a function of the completeness ratio ( n Ego n T op ) of our egocentric set. Each of our sets contain 3 < N t < 11 viewers in the top-view camera, and 2 < N e < 8 egocentric videos. We evaluated the accuracy of our method and baselines when using different subsets of the egocentric videos. A total of 2 N e − 1 non-empty subsets of egocentric videos is possible depending on which egocentric video out of N e are included (all possible non-empty subsets). We evaluate our method on each subset separately. Figure 8 illustrates the assignment and ranking accuracies versus the ratio of the available egocentric videos to the number of visible people in the top-view camera. It shows that as the completeness ratio increases, the assignment accuracy drastically improves. Intuitively, having more egocentric cameras gives more information about the structure of the graph (by providing more pairwise terms) which leads to improvement in the spectral graph matching and assignment accuracy.

Conclusion and Discussion
In this work, we studied the problems of matching and assignment between a set of egocentric cameras and a top view video. Our experiments suggest that capturing the pattern of change in the content of the egocentric videos, along with capturing the relationships among them can help to identify the viewers in top-view. To do so, we utilized a spectral graph matching technique. We showed that the graph matching score, is a meaningful criteria for narrowing down the search space in a set of top-view videos. Further, the assignment found by our framework is capable of associating egocentric videos to the viewers in the topview camera. We conclude that meaningful features can be extracted from single, and pairs of egocentric camera(s) and incorporating the temporal information of the video(s).