{'fraction_non_alphanumeric': 0.03063914189524232, 'fraction_numerical': 0.0073335595166636215, 'mean_word_length': 4.814567526555387, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Game-Theoretic Approach to Multi-Pedestrian Activity Forecasting

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.


Introduction
The goal of this work is to imitate the predictive abilities of human cognition, by building a predictive model that takes into account complex reasoning about: (1) the interdependent interactions of multiple pedestrians and (2) important visual cues needed to infer individual behavior patterns. Consider the complexities of predicting the trajectories of multiple pedestrians as depicted in Figure 1, where four pedestrians are walking on the street. Given this single image, what would one forecast as their future trajectories? A simple prediction would be that all people will walk in a straight line (i.e., the minimum distance) to their goal as in Figure 1(b). This strategy, however, might lead to collisions between pedestrians (e.g., the young man (yellow) and the elder couple (green) may collide). A more thoughtful model might consider the possibility that one or more pedestrians will alter their trajectory based on their prediction of other pedestrians. A possible trajectory set is shown in Figure 1(c), where each pedestrian demonstrates preemptive collision avoidance. Going further, a more informed model might attempt to take into account the observation that on average, elderly couples tend to walk at a slower rate, while a young man is more likely to have a brisk walk. In this case, by considering velocity adjustments, we can reason that all pedestrians will arXiv:1604.01431v2 [cs.CV] 9 May 2016 roughly walk straight to their destination. This illustration serves to highlight the complex reasoning that is involved in predicting the walking trajectories of several people. Our goal is to mimic -computationally -this ability to reason about the dynamics of interactive social processes. Developing computational models for forecasting human activities is an important feature that autonomous system must possess in order to seamlessly interact with their human counterparts. For example, activity forecasting is an important feature needed for domestic robotic systems which need to be able to predict future activities of people to assist them or simply to stay out of their way (e.g., collision avoidance). Autonomous cars must be able to predict the possible trajectories of pedestrians to plan safer routes. Activity forecasting can also be used to detect anomalous activity: by comparing observed human actions against predicted normative behavior, an autonomous system can understand when its human counterpart is doing something out of the ordinary. The realization of such predictive capabilities requires new models of human activity which can reason about complex futures.
In this work, we directly address the interdependent nature of human interactions using the language and concepts of multi-player game theory. In particular, we utilize Brown's [1] classical notion of Fictitious Play to model the interaction between multiple pedestrians. Brown's fictitious play model assumes that each player will take the best next action based on an empirical distribution over the strategies of other players. As we will show, the multi-player game model has strong parallels to multi-pedestrian navigation, as each pedestrian pre-emptively plans his path according to his beliefs about how other pedestrians will move in the future.
To individualize the pedestrian model, we train a deep learning-based classifier to learn visual cues that are indicative of behavior patterns (e.g., age can affect speed). We use the classifier to estimate each pedestrians velocity based on sub-population statis-

CNN Fictitious Play
Old Woman tics. Furthermore, we visually estimate the initial body orientation such that the model is more likely to predict motion aligned to body direction at the start of a predicted trajectory. In this way, we integrate visual analysis with our prediction model. Figure 2 shows the overview of our approach. The contributions of this paper are as follows: (1) We present a novel framework to forecast multi-pedestrian trajectories from a single image by directly modeling the interplay between multiple people using concepts from game theory and optimal control; and (2) we build individualized predictive pedestrian models to generate more accurate forecasts of multi-pedestrian interactions.

Related Work
There has been growing interest in developing computational models of human activities that can extrapolate unseen information and predict future unobserved activities [2,3,4,5,6,7,8,9,10,11,12,13,14]. In the context of pedestrian dynamics, Helbing and Molonar [15] first integrated the concept of the social force model into a computational framework for understanding pedestrian dynamics. Their work incorporated ideas of goals, desired speed and the repulsion due to territorial affects of social forces. In computer vision, the social force model has been used to help aid visual tracking [16] and anomaly detection [17]. More recent work has focused on discovering the underlying potential field by observing human behavior such as patterns of motions [18], mutual gaze or regions of repulsion. In high-density crowds, the patterns of motion of people can be used to infer an underlying flow field for a given scene [19,20] and the interaction between stationary crowds and pedestrians can be used to predict pedestrians' future motions [21]. The global motion or the joint attention of sparse groups of people (e.g., sports scenarios) can also be used to infer basins of attractions or socially salient hot spots [22,23]. Patterns of avoidance can also be used to learn the hidden rewards or costs of physical spaces [24,25,26].
To make reliable predictions about the long-term future, many techniques often assume a static environment [24,26]. In a static environment, the cost topology is constant, where the environment and features do not change over time. In dynamic environments, the cost topology of the state space is constantly changing which means that any computational model must be continually updated. When the cost topology can be accurately updated over time, it can be used for short-term prediction [16,27,28] (or at least until the next update). As such, these techniques have been very effective for tracking multi-pedestrian trajectories. While methods have been proposed for long-term prediction in static environments and short-term prediction in dynamic environments, the task of long-term prediction in dynamic environments remains relatively unexplored in visual human activity analysis except [6,29]. In these work, the complex and intertwined interactions between agents is either ignored [6] or restricted to the perspective of a single agent (wide receiver in [29]). In contrast, we directly address the interdependent nature of human interactions using Fictitious Play and perform long-term prediction for all of the agents in the scene.

Forecasting Multi-Pedestrian Trajectories
Given a single image and initial pedestrians detections, we aim to develop a predictive model that can forecast plausible trajectories for all pedestrians. To do this we must model the complex predictive interplay between multiple pedestrians, while also considering individual differences that impact behavior, to obtain accurate predictions. To address these challenges, we utilize concepts from game theory to model the intricately coupled interactive prediction process. We also leverage recent success in deep neural network to build individual behavior models for each pedestrian from visual evidence. We describe how game theory can be used to frame our multi-pedestrian forecasting problem in Section 3.1 and present a method for mapping the visual appearance of pedestrians to estimate person-specific behavior parameters in Section 3.4. Notation and coordination system. We employ the same coordinate system as in [24]. We define the state space (the ground plane) as a 2D lattice, where each position is denoted by x = [x, y] ∈ X. A pedestrian can make a transition from state to state by taking an action a ∈ A which in the case of a 2D lattice (grid world) is the velocity [ẋ,ẏ]. A trajectory is a sequence of states, s = {x 1 , . . . , x K }. We note that all computations are performed under this coordinates. For example, each state x has an associated vector of features f (x) = [f 1 (x) . . . f J (x)], where f j (x) could represent properties of that state such as the output of a visual classifier, the distance to an object or predicted presence of another pedestrian. When computing speed statistics from the videos, we first project the locations of the pedestrians to our state space (3D floor plane), then compute their respective speed a. The grid world representation is shown in Figure 3.

Forecasting Interactions as Fictitious Play
Game theory [32] is a widely applicable discipline that aims to model adversarial and collaborative interactions between rational decision-makers. It has been applied to a range of disciplines including economic theory [33], politics [34] and computer science [35]. More importantly, it is well-suited for modeling our multi-pedestrian prediction scenario, as the social dynamics of collision avoidance can be modeled as a collaborative multi-player game.
To forecast long-term trajectories of multiple pedestrians, we utilize Fictitious Play (FP) [1], where we model each pedestrian to take a path based on his own predictions (a) (b) Fig. 3. Grid world representation. The ground plane is discretized into cells which represent states. As our state space is the 3D floor plane and as such, 2D image features, trajectories are all projected to the floor plane (assuming camera parameters are known) for computations. (a) and (b) visualize how we discretize the Town Centre Dataset [30] and the Zara dataset [31] respectively.
of how other pedestrians will move. By incrementally forward simulating pedestrian paths with this model, we can obtain a distribution over possible future paths of multiple pedestrians. Formally, each pedestrian n ∈ {1, . . . , N } has the ability to choose a macro-action s n ∈ S from a set of macro-actions. In our scenario, a macro-action s n is a very short trajectory whose length L n depends on the speed of the pedestrian n (detailed in Section 3.4). Each pedestrian has an utility function U n [s n , µ ¬n (s ¬n )] that maps a given macro-action to a value U n : s n → R based on forecasted distributions over macro-actions of all other pedestrians µ ¬n (s ¬n ). The set of trajectories s ¬n is a set of macro-actions of all other pedestrians, s ¬n = {s m |m = n}. We visualize a distribution over states sampled from µ ¬n in Figure 4 (right). Typically, in a repeated game, the empirical distribution over opponent macroactions µ ¬n (s ¬n ) is computed by counting how many times each macro-action was chosen by other players in the past. However in our model, we parametrize the empirical distribution in the following way. The empirical distribution over macro-actions of all other pedestrians can be decomposed into a product of distributions for each pedestrian: µ ¬n (s ¬n ) ∝ m =n µ m (s m ). This is called UPDATEEMPIRICAL in Algorithm 1. Each distribution is parametrized by a maximum entropy probability (also called Boltzmann or Gibbs) distribution, where f m (x) are the features of a state x along the trajectory s m for the pedestrian m, which are weighted by the vector of parameters θ. We will explain in Section 3.2 how the parameters θ can be learned from a dataset of demonstrated pedestrian behavior. Now in order to predict how each pedestrian will move over a sequence of time steps, and to compute how those predictions will affect the predictions of other pedestrian, we need to use a time-varying utility function for each pedestrian n, This is called UPDATEUTILITY in Algorithm 1. Notice that the utility function is also a maximum entropy distribution, where the empirical distribution of all other pedestrians µ  n (x) (details in Section 3.3). The utility function is updated every τ time steps, the frequency at which each pedestrian make predictions about the movement of others. A distribution over states sampled from U n in shown in Figure 4 (right).
It is important to make a connection between the utility function U and the empirical distribution µ at this juncture. In our formulation, U m is exactly equivalent to µ m (s m ). In general, U need not be a probability distribution, as it simply describes the value of one macro-action over another. In contrast, the empirical distribution µ is a probability distribution by construction and it describes which macro-action the opponent is likely to take. More importantly, the utility function U helps us to understand the dependency of the predicted path of pedestrian n on the predicted path of all other pedestrians ¬n. When we use the utility function to forecast the path of a single agent, that prediction influences the predicted path of all other predictions. This interplay between forecasted paths is precisely what we set out to model. Algorithm 1 describes the process of Fictitious Play. For every forecasting period τ , each pedestrian n forms beliefs about the future actions of other pedestrians by updating the empirical distribution µ  is then drawn from the probability distribution over the macro-actions of pedestrian n and later encoded into the feature vector of other pedestrians in the next forecasting period. Figure 4 illustrates the procedure where we employ fictitious play to model the interactions within three pedestrians. The three pedestrians, respectively colored in red, green, and yellow, sequentially make predictions (i.e., fictitious play) of others' macroactions based on µ n (s ¬n ) and then take the macro-action that maximize one's utility function. The forecasted state visitation distribution D (detailed in Section 3.3) of each n,soc +Dm end Algorithm 3: TAKEMACROACTION pedestrian is expressed in the corresponding color and grows incrementally forward with time.

A Decision-Theoretic Pedestrian Model
We now explain how to learn the maximum entropy distribution which is used for both the utility function U n and independent empirical distributions µ m . As we have alluded to earlier, the probability of generating a trajectory s is modeled to be drawn from a maximum entropy distribution, where the probability is proportional to the exponentiated sum of weighted features encountered over the trajectory, where Z is the normalization function (or partition function), θ is a vector of parameters and f (x) is a vector of features at state x.
In order to learn the parameters θ of this model from a set of demonstrated pedestrian trajectories, we utilize maximum entropy inverse optimal control [36]. We first make the assumption that each pedestrian is a rational agent and plans a path according to an underlying Markov Decision Process (MDP). The MDP describing a pedestrian n is defined by an initial state distribution P n (x 0 ), a transition model P n (x |x, a) and a reward function R (t) n (x). Following [24], the reward function is further defined as a weighted combination of features, R (t) n (x). Note however that our reward function R (t) n (x) is time indexed, as the the feature vector f (t) n (x) will be used to encode information about changes in the predicted behavior of other pedestrians.
To learn the parameters θ using maximum entropy IOC [36], we implement a gradient descent procedure that first computes a policy π(a|s; θ) based on the current estimate of θ and then compute the gradient update using difference between the estimated cumulative feature count and empirical cumulative feature count over demonstrated trajectories given that policy. When the features accumulated over trajectories generated by the MDP model converge to the empirical feature counts of the training data (i.e., likelihood under the maximum entropy distribution is maximized), the algorithm has obtained an optimal set of parametersθ, which will be used to define the empirical distribution µ.
An optimal policy for the maximum entropy distribution P (s; θ) can be computed as π(a|s) = exp{Q(x, a) − V (x)}, where the state-action soft value function Q(x, a) and state soft-value function V (x) can be computed by iterating the soft-maximum Bellman update equations: a). We call this procedure COMPUTEPOLICY in Algorithm 2. Recall that in our scenario, the policy is time-varying since the features of the states change over time. Therefore, the policy for each pedestrian must be recomputed each time features are updated, i.e. every forecasting period τ .

Features of the Reward Function
As we have mentioned in Section 3.1, the feature vector f n (s ¬n ). Then we explain the time invariant features in details. Social Compliance Feature: For each pedestrian, we first compute the policy π n (a|x) using the COMPUTEPOLICY procedure mentioned above. Given a policy π n (a|x), we can then generate a state visitation distribution D n of pedestrian n for trajectories of length L n by recursively computing: where D n (x) needs to be initialized to a probability distribution over start locations. Since D (l) n (x) is defined over the entire state space, it is the same size as the state space. We can sum visitation counts over time,D n (x) = l D (l) n (x) to generate a cumulative distribution over states that may be occupied by pedestrian n. The cumulative state visitation distributionD n (x) represents the states that are likely to be occupied by pedestrian n when sampling from the empirical distribution µ n (¬s). By aggregating the cumulative visitation distribution for all pedestrian except n, we can obtain a predicted occupancy map of all pedestrians in the environment,D ¬n (x) = m =nD m (x). Formally, this quantity encodes the empirical distribution µ n (s ¬n ) passed to the utility function (Equation 2). The process is summarized in Algorithm 2 and 3.
More intuitively, this quantity describes a social force field. Based on Helbing and Molonar's model of social forces [15] we define several social distance features that places a force field of varying size around the predicted trajectories of all other pedestrians in the environment. In particular, we defined three different sized force-fields, that roughly corresponds to Hall's proxemics zones [37], to encode a range of physical distances that people may maintain when walking in crowded scenes. The social compliance features f (t) soc (x) is indexed by time as the predicted paths of other pedestrians changes over time. Neighborhood Occupancy: This feature is a measurement of the amount of obstacles in a local neighborhood around a certain state. We calculate the number of pixels labeled as obstacles in a 5 × 5 grid and normalize it to provide a soft estimation of whether a state is a obstacle or not. The feature encodes how close pedestrians will walk near static objects in the scene. The neighborhood occupancy feature is denoted as f occ (x) which is not time varying as we assume the geometry of the scene to be static. Distance-to-Goal: The feature f dog (x) captures a pedestrian's desire to approach his goal quickly by computing the Euclidean distance between a state x and the goal x g . Body Orientation: Since a pedestrian's body orientation is a strong cue of the direction in which he will walk, we train a CNN (described in details in Section 3.4) to predict the initial walking direction of a pedestrian. We use the value of cosine distance minus one over the 8 connected neighbors centered at current pedestrian location. The value is greatest (0) in the direction of the predicted velocity direction and is the lowest (−2) in the opposite direction. The body orientation feature is denoted as f bod (x). Figure 5 visualize all features.

Walking Characteristics from Appearance
We further enhance the predictive power of our multi-pedestrian framework by allowing the model to maintain individualized walking models for each pedestrian based on appearance. In this section, we focus on visual information which conveys salient cues about how each individual in the scene may walk. For example, when we walk in crowds, the initial body orientation of a person may inform us of which direction that individual might walk. We may also perform high-level visual inference, predicting that an elderly couple might walk slow or a young business man might walk with at brisk pace. We proposed to use visual classifiers to identify various attributes of a pedestrian, and then map those attributes to walking direction and speed.
To extract attributes from a pedestrians visual appearance, we make use of a deep learning model. In particular, we employ a network structure similar to [38], but modify the top layer to generate three classification outputs: (1) age (old or young), (2) gender (male or female) and (3) body orientation (8 discretized direction). We train all three top layer classifiers jointly, as previous work has shown that multi-task learning helps to constrain the parameter learning [39,40]. The predicted body orientation is used to generate the body orientation feature mentioned in Section 3.3, while the output of the age and gender classifiers are used to build individualized pedestrian models.   To be concrete, we use the soft probabilistic output of the age and gender classifiers to estimate an individualized velocity parameter. For each pedestrian n, we compute his/her individual velocity v n as the weighted average over gender and age velocity averages, i.e. v n = a w a v stats a , where a ∈ {male, f emale, old, elder} denotes the attributes, w a denotes the softmax output from deep net, and v stats a represents the average speed of pedestrians with attribute a. The individualized speed v n is then incorporated into our model by multiplying the forecasting window size W , i.e. L n = W × v n . Recall that L n is the length of macro-actions s n and v n denotes speed, W can thus be interpreted as how many time steps into the future one will predict about others. In general, given a fixed W , the faster a pedestrian walks, the larger his occupancy map D n = t+W * vn l=t D (l) n may be. We note that when speed information is not available, we employ a constant speed C for every pedestrian, i.e. L n = W × C.

Experiments
To validate our claim that (1) our fictitious play based approach better describes the interplay between pedestrians and (2) our deep learning based classifier can effectively extract features that are indicative of behavior pattern, we evaluate forecasting performance on three different pedestrian interaction datasets: (1) the Zara Dataset [31], (2) the Town Centre Dataset [30], and (3) the LIDAR Trajectory Dataset.
As our goal is to learn how people reason about the behavior of others and model the interactive dynamics among them, we focus on the video clips where pedestrians demonstrate strategic reasoning. We make an important assumption that people tend to take the shortest available path subjecting to the environmental constraints (e.g. pedestrians should walk on the sidewalk, people cannot cross the obstacles). If people explicitly adjust their path, we argue that this is affected by others. In particular, we employ the modified Hausdorff distance (MHD) to measure the physical distance between one's ground truth trajectory (provided by tracker) and the shortest possible trajectory (subjecting to environmental constraints). The MHD allows for local time warping by finding the best local point correspondence over a small temporal window and has been widely used to evaluate the similarity of two sequences [24]. The larger the MHD is, the more different the actual trajectory is to the shortest path, and the more obvious the interaction may be. The MHD is defined as: where s gt and s short respectively denote the ground truth trajectory and the shortest possible trajectory towards goal, x gt,n and x short,w refer to the nth and wth state in sequence s gt and s short , W represents the small temporal window, and · denotes the euclidean distance function. If the MHD is larger than a certain threshold , we posit that there is an obvious interaction among pedestrians that leads to the change of trajectories. We thus include the trajectories of all pedestrians in the scene as one multi-pedestrian trajectory sequence. If the MHD is smaller than the threshold, we assume there is no obvious interactions and reject it. Following this criteria, we respectively obtain 16 multi-pedestrian trajectory sequences from the Zara Dataset [31] and the Town Centre Dataset [30]. Figure 6 illustrates our selection process.
To show that our model can also work with other modes of trajectory data, we introduce a LIDAR-based Trajectory Dataset. This dataset consists of 20 interactive trajectories satisfying the criteria mentioned above. Subjects are initialized at various start locations in a small room (7m×7m) with a few obstacles. They are then directed to walk towards a goal location without colliding with other pedestrians in the scene. We show samples of each dataset in Figure 7. We note that our approach is a generalization of [24] to the multi-agent scenario. While we train only on multi-pedestrian sequences, we can always perform single agent forecasting by setting the social compliance feature to zero (detailed in Section 4.4).

Metrics
We measured the performance of our forecasting model using two different metrics: (1) negative log loss (NLL) for consistency and (2) state collision rate (SCR) for robustness. Firstly, NLL computes the likelihood of drawing a certain trajectory from learned policy. It is defined as: where trajectory s denotes a sequence of state x, and action a corresponds to the transition between two states. In our case, both s and a are computed from ground truth trajectory sequences. NLL thus describes how consistent the learned policy π is when applied to the demonstrated examples. In addition to NLL, we also employ a metric based on the behavior of multiple people to assess the robustness of a forecasting model. While NLL is a good metric for quantifying how well a model describes the behavior of a single agent, it ignores the interactions among various agents. For instance, walking straight towards goal may suffice for one person due to low NLL, but not when there are multiple people due to potential collisions (Figure 1(b)). Thus, we introduce the second metric, state collision rate (SCR). It is defined as: where n denotes pedestrian ID, D n (x) represents the expected state visit count at state x at time t, i.e. the probability of being at certain state at a certain time, and Z refers to the normalization factor. By taking into account the distribution of multiple pedestrians and taking the intersection jointly, the expected state visit counts of all agents represent a region of collision.

Multi-Pedestrian Forecasting Performance
We validate the effectiveness of our model by performing comparative experiments against the following three baselines. N-Independent MDP (nMDP). This baseline model is the approach of [36] applied to images to forecast the trajectory of a single pedestrian [24]. We extend their approach to meet our multi-agent scenario. In particular, we use N instantiations of their MDP model and run them in parallel. MDP + Constant Velocity (MDPCV). The second baseline model is a modification of the N-Independent MDP model but with a collision region features added to the reward function. By assuming constant velocity, we can compute regions of collision (i.e., the intersection regions of linear motion models) in advance and incorporate such information into the reward function. The N pedestrians are thus no longer independent as in nMDP. mTA. Based on the work of Pellegrini et al. [16], the third approach is a modified Trajectory Avoidance (mTA) model. In [16] every agent chooses a velocity that minimizes its energy function at every time step. Formulated as an MDP, this corresponds to a reward function using only a constant feature. As for modeling the social force features (e.g., comfortable distance among agents), we use the social compliance features described in Section 3.3. We emphasize here that this baseline model has less information than the original model described in [16] where every agent knows the positions and velocities of others. The information is not available due to our problem setup, i.e., single image input. For all three datasets, we performed 5-fold cross validation. The forecasting window size is set to W = 3 and forecasting period is τ = 1. The results of the proposed approach with comparisons are shown in Table 1. We observe that our fictitious play based approach outperforms all three approaches with respect to NLL and SCR. This shows that our iterative predicting and planning process better predicts human interactions and also generates the most collision-free trajectories.
We further incorporate speed information into our model using the method mentioned in Section 3.4. We evaluate the effectiveness of speed information only on the Town Center dataset as the resolution of the Zara dataset is too low and the LIDARbased Trajectory dataset does not provide visual information. We collected ≈ 16000 pedestrian patches from the Town Centre Dataset [30], with three labels for each patch, i.e. age, gender, and body orientation. As the TownCentre Dataset already provides the ground truth data for trackers, we do not need to label age and gender for all image patches directly. We ask 2 in-house annotators to label 5 randomly sampled patches for each pedestrian. If the labels for a pedestrian are consistent, we treat them as ground truth and propagate to other frames of a track. Otherwise, we repeat. We also assume that one's body orientation is well aligned with one's trajectory as in [41]. We can thus automatically compute the label for all patches using the ground truth trajectories. Fig-Fig. 8. Labeling attributes for Town Centre dataset: As the Town Centre dataset already provides the ground truth data for trackers, we do not need to label age and gender for all image patches directly. We simply annotate each pedestrian once and then propagate the labels to other frames of a track. We can also obtain the body orientation label for free by assuming that one's body orientation is well aligned with one's trajectory as in [41]. ure 8 shows how we generate labels for age and gender. The images are randomly split into ≈ 12000 training images and ≈ 4000 validation images by pedestrians. We train a deep classifier using the network structure mentioned in Section 3.4 and the performance on the validation set is shown in Table 2(right). We also computed the average speed of people with different attributes as shown in Table 2(left). With the speed statistics and the output of the deep network model, we compute an individualized model for each pedestrian. From Table 1 we can observe that our model performs better after considering pedestrian's visual appearance and individualize the predictive model. Selected qualitative results of predicted trajectories are shown in Figure 9. Each pedestrian is marked with a colored bounding box, and a corresponding forecasting distribution is shown in the same color. The more saturated the color is, the higher the probability. Many of the predicted trajectories are smoothly curved (and do not exhibit abrupt changes in trajectories), indicating that the proposed approach mimics human behavior of taking preemptive actions to avoid collisions. We note that we consider all pedestrians for quantitative experiments but only visualize forecast distributions for a limited number of pedestrian to improve visualization.

Forecasting Window Size and Forecasting Period
We now take a deeper look at the parameters that define the recursive reasoning parameters of the fictitious play algorithm. How far into the future (forecasting window) should the model predict others' movements? How often (forecasting period) should   Table 3. Performance of our model using various fictitious play parameters. W denotes forecasting window (how far into future one should predict), while τ refers to forecasting period (how often one should predict).
each agent update predictions? In order to better understand how forecasting window size W and forecasting period τ affect the performance of our approach, we perform grid search over the two parameters on the Town Centre dataset, with the estimated speed information included. From Table 3, we can see that given a fixed forecasting window size W , NLL and SCR always increase (gets worse) as the forecasting period τ becomes larger. This phenomenon implies that no matter how far one sees into the future, it is better to predict the actions of others as frequent as possible. This result is reasonable since a larger τ means we assume no changes in the dynamics of the scene over a longer period of time. As for fixing τ and increasing W , SCR decreases initially and then increases with W as expected. This is reasonable since at first the more future steps one forecasts, the more conservative one would be, and therefore less likely to collide with another person. However, as W becomes very large, all agents become overly conservative and may result in collision. Note that NLL does not have a consistent relationship with W , and the breaking point (where scores change from decreasing to increasing) for NLL and SCR is different with various τ . However, in most cases both metrics reach their minimum when W = 3. This result to some extent implies that pedestrians in our datasets tend to forecast three steps into the future of others.

Features for Forecasting
We further evaluate the effects of the features used in our proposed model. Three models with different feature sets are trained in advance: (1) models trained only on  We set forecasting window size W = 3 and forecasting period τ = 1 as before.
The average NLL and SCR for the Town Center dataset using different features are shown in Fig. 10. The performance of other approaches are also shown for reference. Note that N-Independent MDP and MDP+Constant Velocity still consider only physical features and body orientation features even in the + Social Compliance column, since nMDP assumes all entities in the scene are motion-less. Also, our model performs almost identical to nMDP when considering only physical features and body orientation features. This result is expected as there are no social compliance features to change the cost topology over time. If there is only one agent (and there is no social compliant feature), our proposed model reduces to nMDP. The most important result is that with the inclusion of the social compliance feature, our proposed models better explains the interactions between multiple pedestrians. The FP+Speed model attains a NLL of 10.892 compared to the next best performing model mTA at 14.2471, resulting in a 23.5% improvement in the NLL.

Destination Forecasting
In previous sections, we have assumed that the destinations of each pedestrian is known to understand the role of social compliance features and to decouple the effect of uncertain about the goal. In many situations, the final destination of a pedestrian is not known and needs to be inferred. Following [24], we densely generate potential goals on the map and perform the same forecasting experiment, which can be done effectively by inverting the role of goals and start locations. For more details, please refer to [24].
We evaluate the performance of our destination forecasting model on the Town Centre dataset and the Zara dataset. The only information used is the start locations for each pedestrian. No observations are included. Results show that our FP based approach consistently outperforms others even without knowing the destinations ahead of time. The absolute performance of all models degrade due to uncertainty about the goal. From Table 4 we observe that mTA still performs the second best, while nMDP still performs the worst. Together with Table 1, experimental results show that our model which models the interplay and visual evidence outperforms the baseline models.

Conclusion
We present a novel framework to forecast multi-pedestrian trajectories from a single image by directly modeling the interplay between multiple people using concepts NLL nMDP [24] MDPCV mTA FP FP + Speed Zara [31] 98. from game theory and optimal control. We also develop various predictive models to show how different modes of information help to reason about the future actions of multi-pedestrian scenarios. By building individualized pedestrian models for each person based on his visual appearance, we generate more accurate prediction of multipedestrian interactions. We have compared our Fictitious Play based approach with other state-of-the-art algorithms. Our evaluation on multiple pedestrian interaction datasets has shown that our proposed approach is able to attain more accurate long-term predictions of pedestrian activity.