{'fraction_non_alphanumeric': 0.028091450186349225, 'fraction_numerical': 0.007231462424208711, 'mean_word_length': 4.803098773402195, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Protocol for Validating Social Navigation Policies

Enabling socially acceptable behavior for situated agents is a major goal of recent robotics research. Robots should not only operate safely around humans, but also abide by complex social norms. A key challenge for developing socially-compliant policies is measuring the quality of their behavior. Social behavior is enormously complex, making it difficult to create reliable metrics to gauge the performance of algorithms. In this paper, we propose a protocol for social navigation benchmarking that defines a set of canonical social navigation scenarios and an in-situ metric for evaluating performance on these scenarios using questionnaires. Our experiments show this protocol is realistic, scalable, and repeatable across runs and physical spaces. Our protocol can be replicated verbatim or it can be used to define a social navigation benchmark for novel scenarios. Our goal is to introduce a protocol for benchmarking social scenarios that is homogeneous and comparable.


I. INTRODUCTION
One of the main prerequisite of making robots ubiquitous and generally applicable is to endow them with the ability to move around people in a socially acceptable manner. A robot must be able to accomplish navigation tasks while adhering to social norms in shared spaces and respecting human actions and behaviors. We refer to such type of navigation as Social Navigation. Recently, the robotics community has witnessed an increased interest in Social Navigation. Among many other directions, researchers investigate the importance of respecting personal space [1], maintaining social dynamics [23] and velocities [9], socially-acceptable approaching behavior [6], and navigation in the presence of groups of people [10].
While these approaches are a testament for the rapid progress in this direction, the evaluation and comparison of algorithms has proven to be difficult for Social Navigation research. To facilitate progress in the community it is of paramount importance to have shared, realistic, repeatable, and scalable benchmarks. If one is to define such a benchmark, then simulation is one tool of choice -we have seen an increase in robotic simulation environments that focus on physics and visual realism [19], [24], [14]; simulation is scalable and repeatable across labs. However, simulating the intricacies of human behavior at different levels of abstraction, ranging from atomic actions and motion dynamics to more complex activities and behavior, has proven to be difficult. Therefore, simulation of humans falls short of providing a realistic medium for social navigation benchmarking. A different approach for evaluation is to perform demonstrations and studies in uncontrolled real settings in the wild, e. g. accessible public spaces such as university campuses. While such evaluations are by definition realistic, they face several limitations. For one, they are not guaranteed to be behaviorally natural as one has to enact social interactions in real studies, which may lead to undesired patterns in the observations (e.g. such as teetering motions). More importantly, though, real experiments are difficult to repeat and to conduct at scale. Every run, even under controlled conditions, will differ from prior runs and running experiments repeatedlywith human in the loop -can be extremely costly. Finally, social interactions in real environments are defined by a wide range of variables (e.g. differences in human behavior and appearance, environmental settings, etc.) that make obtaining meaningful measurements difficult.
To address these challenges, we aim to propose a protocol for establishing a social navigation benchmark. The desired properties of our benchmark are: • Realism: The benchmark is implemented in a real environment, real robot, and real humans; • Scalability: The benchmark allows for testing on a diverse set of social situations, with a cost which allows for frequent evaluations; • Repeatability: The benchmark is repeatable across different runs and instantiations in different physical spaces. To achieve the above properties we propose a real benchmark based on predefined set of social scenarios evaluated using user surveys. In more detail, we introduce a set of social navigation scenarios ( Figure 2) implemented in realworld settings ( Figure 1). Each scenario is a canonical example of a common human-robot interaction that can occur when performing navigation tasks. The idea is that reducing a human-robot interaction to its essence allows us to better understand and validate social behavior of humans and robots. To this end, we define the scenarios in a way that they can be replicated by different labs with low effort and so as to avoid high variance in the human-robot interaction.  This addresses the challenge of repeatability as enacting the canonical scenarios will be comparable. Second, we propose an in-situ metric based on questionnaires to obtain ratings of humans who experienced the interaction with the robot. Unlike other metrics, this allows us to validate the performance of navigation policies w.r.t. social expectations. We show initial results of validating different policies and that questionnaires can be used to measure meaningful gradients for validating social navigation policies. Third, we present guidelines and good practices for defining social navigation benchmarks for other scenarios and environments. Altogether, we hope that our protocol will prove useful for the community to converge to more standardized validation setups.

II. RELATED WORK
Driven by its importance to robotics, social navigation has become the focus of a growing body of research. Because of social navigation's complexity and growth, we cannot comprehensively discuss all related work. For an overview touching on the validation of social navigation, the interested reader is referred to the recent survey papers of Gao and Huang [5], Charalampous et al. [4], Mavrogiannis et al. [15], Kruse et al. [12], Rios-Martinez [21], and more recently, Xiao et al. [26] and Mirsky et al. [16]. Closest to our work are methods that use metrics to measure human discomfort [20], [11], [17] or sociability [18], [22], [2], datasets for social navigation [8], as well as approaches that employ questionnaires for validation purposes [3], [7], [25].

III. BENCHMARK FOR SOCIAL NAVIGATION
To establish a social navigation benchmark our goal is to define a set of canonical social navigation scenarios. Each scenario represents a common interaction between a human and a robot performing a navigation task. Specifically, we define frontal approach, intersection wait, intersection gesture, narrow doorway, and blind corner. For each scenario we define start and end points and task the robot to navigate along the trajectory between the two points (see Fig. 3). Each scenario is then enacted by a human, who is provided with a short description of what is expected to happen. For example, for our frontal approach scenario, we simply say "Please walk along this trajectory, start walking when the robot is here, the robot is expected to yield." The human is then walking in the opposite direction of the robot, while  Fig. 3. Illustration of the locations we used for each of the social scenarios: frontal approach (red), intersection wait (green), intersection gesture (blue), narrow doorway (yellow), and blind corner (magenta). Grey round boxes represent obstacles present in the building (e.g., such as chairs or desks). Arrows with round ends and solid lines indicate human trajectories and arrows with square ends and dashed lines robot trajectories. the robot is driving toward its goal position. By defining a specific scenario with a constrained trajectory we define a canonical example of a social interaction.
Please note that we aim to keep the definition of a social scenario as lightweight as possible -we only define the social scenario along with a brief description and the start and end points of the trajectory for the robot. We do not specify any constraints for the environment, human behavior, and human appearance. This allows us to implement and then validate the social scenarios in various different environments, while we can also repeatedly measure the performance of a policy on a defined social scenario. To establish a benchmark we define each scenario two times in different locations of our building (Fig. 3).

A. Social Navigation Scenarios
For our current benchmark we have selected five social scenarios of human-robot interactions as detailed below. For most of our scenarios we focus on observant and passive robot behavior; i.e. the robot is expected to yield and make room for the human. Consequently, our ideal social navigation policy would generate robot behavior in a way that the human would almost not notice the robot. An illustration of the social scenarios of our benchmark is shown in Fig. 2. Examples of our real setup for each social scenario are shown in Fig. 1, 4, and 5.
Frontal: Robot and human are approaching each other from two ends of a straight trajectory; enough space is Examples of the social scenarios: narrow doorway (a, b), intersection gesture (c, d), and intersection wait (e, f). Each scenario is defined by enacting the social interaction with a human and a robot that is manually controlled by a human operator.
provided for the robot to yield. Robot and human are walking toward each other and the robot is expected to yield early on to avoid socially-intimidating behavior. Human and robot are alternating their start and end positions.
Intersection Wait: Robot and human approach each other on perpendicular trajectories. The human does not stop walking down their path. The robot is expected to drive slowly when it approaches the human and it has to come to a complete stop to yield to the human. After the human is out of sight, the robot continues on its trajectory.
Intersection Gesture: Robot and human approach each other on perpendicular trajectories. Human and robot come to a complete stop. The human recognizes the robot and then gestures -with a waving hand motion along the trajectory of the robot -that they yield to the robot. The robot interprets the gesture and continues its path.
Narrow Doorway: Human and robot cross each other's paths by moving through a narrow doorway. Robot and human alternately start inside or outside a room and try to get in or out. In this scenario, the human or the robot has to yield to the respective other agent. If the robot arrives at the door before the human it is allowed to continue on its path. If the human arrives at the door first, the robot has to wait outside the door and yield to the human.
Blind Corner: Human and robot cross each other's paths at a blind corner. Human and robot move down a hallway toward the corner and 'surprise each other' by meeting at the corner at the same time. Both agents have to come to a complete stop to then resolve the situation. The robot is expected to either yield to the human after the collision or to avoid the collision by anticipating the situation.

IV. IN-SITU VALIDATION
To validate the social-compliance of a policy we define an in-situ metric based on a questionnaire for each social scenario. We use a five-level Likert scale [13] to define Frontal Approach 1 The robot moved to avoid me.

2*
The robot obstructed my path. 3 The robot maintained a safe and comfortable distance at all times.

4*
The robot nearly collided with me. 5 It was clear what the robot wanted to do. Intersection Wait 6 The robot let me cross the intersection by maintaining a safe and comfortable distance. 7 The robot changed course to let me pass. 8 The robot paid attention to what I was doing. 9 The robot slowed down and stopped to let me pass. Intersection Gesture 10 The robot maintained a safe and comfortable distance at all times. 11 The robot slowed down and stopped. 12 The robot followed my command. 13 I felt the robot paid attention to what I was doing. Narrow Doorway 14* The robot got in my way. 15 The robot moved to avoid me. 16 The robot made room for me to enter or exit. 17* It was clear what the robot wanted to do. Blind Corner 18 The robot moved to avoid me. 19 The robot stopped to let me pass. 20* I had to move around the robot. 21* The robot nearly collided with me head-on. 4 -5 questions for each scenario and ask participants to rate their agreement toward these questions based on the following scale: 1=strongly disagree, 2=disagree, 3=neutral, 4=agree, 5=strongly agree. Additionally, we allow for the rating 0=cannot tell/something went wrong. The questions for each scenario are provided in Table I: "How much do you agree with the statement ... ?". We run the scenario with human and robot and immediately ask the participant to provide the rating before performing another scenario. This allows us to get reliable in-experience ratings.

V. EXPERIMENTS
To begin validating our benchmark, we have conducted experiments demonstrating that our setup is scalable, reliable and repeatable. For most of the experiments we use a simple iLQR-based model predictive controller (MPC) to generate linear and angular velocity commands for our robot (provided by Everyday Robots, Alphabet). We repeatedly run the policy and ask the human participants to answer the questionnaire (Table I) after each run.
In Table II we show the results of running the MPC policy against all of our five social scenarios in both of the defined locations (Fig. 3). For this experiment, we recorded 10 runs for each social scenario in both of the defined locations. We report the per question average (QAVG), as well as the overall average (SAVG) along with the standard deviation for each scenario. For negatively formulated questions (labeled with a '*' in Table I) we reverse coded the ratings to make them comparable to the positively formulated ones. Across the different scenarios we obtain similar average ratings. This suggests that our questionnaire based metric can be used to obtain meaningful results for different social scenarios.   Moreover, we observed that the measurements for the same social scenario in different locations corresponded with each other. This indicates that our setup can be replicated for the same social scenarios in other locations (e.g. other labs) or for other social scenarios. One run of a social scenario commonly takes 15-45 seconds, while answering the 4-5 questions for each scenario requires 20-40 seconds.
To test our setup for individual human biases, we ran the same Location (F2) frontal approach scenario 30 times with three different participants. Table III shows that for each participant we obtained similar average ratings for each question and for the entire social scenario. This suggests that our questionnaire-based metric provides good inter-rater reliability.
To measure the variance of our setup when running the same experiment at different dates and times, we ran the Location (F2) frontal approach scenario 90 times with the same human participant. Table IV shows similar average ratings and standard deviations across the different runs of the same social scenario. This suggests that our questionnaire based metric can be used to obtain reliable measurements across different validation runs. Furthermore, this suggests that it may not be necessary to capture large quantities of runs to obtain reliable results but that batches of up to 30 runs already provide meaningful results.
To test our benchmark for different polices we compare the MPC policy with a behavioral cloning (BC) policy. Here we captured 300 trajectories of expert data of the frontal Fig. 5. Validation run of the blind corner scenario. Robot and human are moving in opposite directions around a blind corner on a colliding path. The MPC policy is not able to anticipate the human interaction causing a collision -the robot drove onto the person's foot. Consequently the rating for this social interaction was: Q18=strongly disagree, Q19=disagree, Q20=neutral, Q21=strongly agree. approach scenario and trained a convolutional neural network to predict intermediate waypoints for the MPC policy. We use this policy to generate socially-compliant behavior. Table V shows the average questionnaire ratings of 20 runs of each policy. These results indicate that we are able to measure the capabilities of this more advanced policy compared to the common MPC policy (indicated by the higher averages).
In Fig. 5 we show a validation run of our blind corner scenario that highlights the advantages of obtaining in-situ ratings from human participants. For this run, the robot briefly collided with the human, touching the person's foot. While this event is hardly noticeable for an external observer or in an ex-situ setting (Fig. 5, right) it generated a clear negative response when the participant provided their ratings.

VI. CONCLUSION
We have introduced a novel protocol for establishing a benchmark for social navigation scenarios. The proposed approach is based on defining a set of common social interactions that occur for navigation tasks. Each social scenario is defined in a canonical manner to support the repeated validation of policies. Additionally, we have proposed and piloted a questionnaire-based metric to obtain insitu ratings of human participants that allow us to assess the social compliance of navigation policies. We only rely on a lightweight specification for each social scenario. Therefore, our benchmark and our questionnaire can readily be extended by additional scenarios. As future work, we plan to extend our benchmark and to use it to validate existing and novel socially-compliant navigation policies.
VII. ACKNOWLEDGMENTS We thank our robot operators April Zitkovich, Jake Lee, Khem Holden, Rosario Jauregui Ruano, and Diego Reyes for their diligent work collecting all social navigation data samples reported in the paper.