{'fraction_non_alphanumeric': 0.0363458463594492, 'fraction_numerical': 0.013746032503639358, 'mean_word_length': 4.614147909967846, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semantic Visual Navigation by Watching YouTube Videos

Semantic cues and statistical regularities in real-world environment layouts can improve efficiency for navigation in novel environments. This paper learns and leverages such semantic cues for navigating to objects of interest in novel environments, by simply watching YouTube videos. This is challenging because YouTube videos don't come with labels for actions or goals, and may not even showcase optimal behavior. Our proposed method tackles these challenges through the use of Q-learning on pseudo-labeled transition quadruples (image, action, next image, reward). We show that such off-policy Q-learning from passive data is able to learn meaningful semantic cues for navigation. These cues, when used in a hierarchical navigation policy, lead to improved efficiency at the ObjectGoal task in visually realistic simulations. We improve upon end-to-end RL methods by 66%, while using 250x fewer interactions. Code, data, and models will be made available.


Introduction
Consider the task of finding your way to the bathroom while at a new restaurant. As humans, we can efficiently solve such tasks in novel environments in a zero-shot manner. We leverage common sense patterns in the layout of environments, which we have built from our past experience of similar environments. For finding a bathroom, such cues will be that they are typically towards the back of the restaurant, away from the main seating area, behind a corner, and might have signs pointing to their locations (see Figure 1). Building computational systems that can similarly leverage such semantic regularities for navigation has been a long-standing goal. Even though you don't see a restroom, or a sign pointing to one in either of these images, going straight ahead in the left image is more likely to lead to a restroom than going straight in the right image. This paper seeks to learn and levarage such semantic cues for finding objects in novel environments, by watching egocentric YouTube videos.
Such use of videos presents three unique and novel challenges, that don't arise in standard learning from demonstration. Unlike robotic demonstrations, videos on the Internet don't come with any action labels. This precludes learning from demonstration or imitation learning. Furthermore, goals and intents depicted in videos are not known, i.e., we don't apriori know what each trajectory is a demonstration for. Even if we were to label this somehow, the depicted trajectories may not be optimal, a critical assumption in learning from demonstration [49] or inverse reinforcement learning [41].
Our formulation, Value Learning from Videos or VLV, tackles these problems by a) using pseudo action labels obtained by running an inverse model, and b) employing Q-learning to learn from video sequences that have been pseudo-labeled with actions. We follow work from Kumar et al. [36] and use a small number of interaction samples (40K) to acquire an inverse model. This inverse model is used to pseudo-label consecutive video frames with the action the robot would have taken to induce a similar view change. This tackles the problem of missing actions. Next, we obtain goal labels by classifying video frames based on whether or not they contain the desired target objects. Such labeling can be done using off-the shelf object detectors. Use of Q-learning [58] with consecutive frames, intervening actions (from inverse model), and rewards (from object category labels), leads to learning optimal Q-functions for reaching goals [53,58]. We take the maximum Q-value over all actions, to obtain value functions. These value functions are exactly γ s , where s is the number of steps to the nearest view location of the object of interest (γ is the Q-learning discount factor). These value functions implicitly learn semantic cues. An image looking at the corner towards the back of the restaurant will have a higher value (for bathroom as the semantic target) than an image looking at the entrance of the restaurant. These learned value functions when used with a hierarchical navigation policy, efficiently guide locomotion controllers to desired semantic targets in the environment.
Learning from such videos can have many advantages, some of which address limitations of learning from direct interaction (such as via RL). Learning from direct interaction suffers from impractical sample complexity (the policy needs to discover high-reward trajectories which may be hard to find in sparse reward scenarios) and poor generalization (limited number of instrumented physical environments available for reward-based learning, or sim2real gap). Learning from videos side-steps both these issues. Our experiments in visually realistic simulations show 66% better performance than RL methods, while at the same time requiring 250× fewer active interaction samples for training.

Related Work
This paper tackles semantic visual navigation in novel environments. Our proposed solution is a hierarchical policy that employs value functions learned from videos. We survey different navigation tasks, the different representations used to tackle them, and the different training methodologies employed to build those representations.
Navigation Tasks. Navigation tasks take many forms [3], but can largely be grouped into two categories based on whether they require exploration or not. Finding paths in known environments [64], or going to a known relative offset in a previously unknown environment [25], do not require very much exploration. On the other hand, tasks such as finding an object [25] (or a given image target [10]) in a novel environment, or exhaustively mapping one [9,11], require exploration and are thus more challenging. Our down-stream task of finding objects in previously unseen novel environments falls into this second category. Most current work [15,25,42,63] on this task employ end-to-end, interaction-heavy learning to get at necessary semantic cues. Our work instead seeks to mine them from videos with minimal active interaction.
Representations. Solving navigation tasks, requires building and maintaining representations for space. These range from explicit metric maps [18, 54,61] or topological representations [10,33,47], to more abstract learned implicit representations [38]. Such learned representations can effectively learn about semantic cues. Research has also focused on making classical metric and topological representations more semantic: explicitly by storing object detector or scene classifier outputs [8,23,28,34,39,43,60], or implicitly by storing abstract learned feature vectors useful for the end-task [25]. In our work, we use a hybrid topological and metric representation that incorporates implicit semantic information. Our focus is on investigating alternate ways of learning such semantic information.
Hierarchical Policies. Researchers have pursued many different hierarchical policies [7] for navigation: no hierarchy [38], macro-actions [25,64], low-level controllers [6,31], and sub-policies [9,14,23]. In particular, Chaplot et al. [9,10] decompose exploration policies into a global policy, for high-level semantic reasoning, and a local policy, for low-level execution to achieve short-term goals produced by the global policy. We follow a similar decomposition, but tackle a different task (object goal), and investigate learning from unlabeled passive data vs. active interaction or strong supervision.
Training Methodology. Different papers pursue different strategies for training navigation policies: no training [54], supervised learning for collision avoidance [20,22], behavior cloning, DAgger [25,35,45], reinforcement learning with sparse and dense rewards [9,38,46,59,64], and combinations of imitation and RL [11,13,44]. In contrast, this paper designs a technique to derive navigation policies by watching YouTube videos. This is most similar to work from Kumar et al.
[36] that studies how to learn low-level locomotion sub-routines from synthetic videos. In contrast, we learn high-level semantic cues from actual YouTube videos.
Learning for Acting from Videos. Learning about affordances [19], state-transitions [2,29], and task-solving procedures [12], with the goal of aiding learning for robots, is a long-standing goal in computer vision. Our work is also a step in this direction, although our output is directly useful for building navigation policies, and our experiments demonstrate this.
Imitation Learning from Observations. A number of recent papers [5,16,17,21,50,55,56] focus on learning policies from state-only or observation-only demonstrations. All of these works focus on learning a policy for the same task in the same environment that is depicted in the observationonly demonstrations (with the exception of Gangwani et al. [21] who show results in MDPs with different transition dynamics). Our work relaxes both these assumptions, and we are able to use video sequences to derive cues that aid solving novel tasks in novel environments.

Proposed Approach
The final task we tackle is that of reaching semantic goals in a novel environment, i.e., at test time we will place the agent in a novel environment and measure how efficiently it can find common house-hold objects (bed, chair, sofa, table and toilets).
Overview. We design a 2-level hierarchical policy. The high-level policy incrementally builds a topological graph and uses semantic reasoning to pick promising directions of exploration. It generates a short-term goal (within 2m) for the low-level policy, that achieves it or returns that the short-term goal is infeasible. This process is repeated till the agent reaches its goal. We describe the details of this hierarchical policy in Supplementary Section S6. Our central contribution is the procedure for learning the semantic reasoning function, which we call a value function (following RL terminology [53]), for the high-level policy from videos, and we describe this next.

Value Learning from Videos
Given an image I and a set of object categories C, we seek to learn a function f (I, c) that can predict the value for image I for reaching an object of category c ∈ C. Images that show physical space close to instances of category c should have a higher value than images that show regions far away from it.  . This prepares them for c) Q-learning, which can extract out optimal Q-functions for reaching goals purely by watching egocentric videos. See Section 3.1 for more details.
Let's say we have V, a set of egocentric video tours of indoor spaces. We seek to learn this function from such videos. We follow a three step procedure: a) imagining robot actions that convey the robot between intervening frames, b) labeling video frames of images containing instances of the desired object category, and c) Q-learning on the resulting reward-labeled image-action sequence trajectories. Figure 2 shows an overview of this process, we describe it in more detail below.
Action grounding. Such videos don't come with any information for how one image is related to another. We follow the pseudo-labeling approach from [36,55], to imagine the actions the robotic agent would have taken to induce the depicted transformation. We collect a small amount of interaction data, where a robot executes random actions in a handful of environments. This data is in the form of image action sequences, . . . , I t , a t , I t+1 , . . ., and importantly, has information of the action that was executed to go from I t to I t+1 . We use this interaction dataset to train a one-step inverse model ψ [1, 30] that uses I t and I t+1 to predictâ t = ψ (I t , I t+1 ). ψ is trained via a cross-entropy loss between its predictionâ t and ground truth a t . We use this inverse model ψ to pseudo-label the video dataset V with action labels to obtainV.
Labeling Video Frames with Goals. Our next step involves labeling video frames with the presence of object instances from categories in C. This can simply be done by using an off-the-shelf object detector D (such as Mask RCNN [26]) trained on the MS-COCO dataset [37]. We assign a binary reward value r c (I) for each category c for each video frame I: +1 if object detected, and 0 otherwise.
Value Learning via Off-policy Q-Learning. Our next step is to derive value function f (I, c) for the different categories. The above two steps, generate reward-labeled, image-action trajectories for traversals in indoor environments. For each category c ∈ C, these are in the form of quadruples I t ,â t , I t+1 , r c t+1 , where I t and I t+1 are consecutive frames,â t is the pseudo-label as predicted from the inverse model ψ, and r c t+1 is the label for category c for image I t+1 . These quadruples can be thought of as transitions from a Markov Decision Process (MDP) [53], where the agent gets +1 reward for entering into a location close to the desired target object, and 0 reward otherwise.
Thus, even though we don't have access to the physical environment, a simple video traversal of an indoor space can be pseudo-labeled to obtain transition samples from the underlying MDP operating in this environment. Under mild conditions, such samples are all that are necessary for learning optimal value functions via Q-learning [58]. Thus, instead of directly learning the value function f (I, c), we learn a Q-function Q(I, c, a) that predicts the Q value of executing action a when at image I and seeking to find object from category c.
Q-learning takes the following form, where we seek to learn the fixed point, Q * of the following Bellman equation (for each category c): Q * (I t , a t , c) = max a r c t+1 + γQ * (I t+1 , a , c) . This is done by finding the Q that minimizes the following objective function, over transition quadruples fromV (we parameterize Q as a convolutional neural network (more details in Section 4)): (1) Value function f (I, c) can be obtained by simply taking a maximum of the Q-values over all actions, i.e., f (I, c) = max a Q(I, a, c). This gives us our desired value function.
Note, Q-learning can learn optimal Q-functions independent of where transition quadruples come from (as long as they cover the space), and in particular, can learn from off-policy data. This allows us to learn optimal value functions even though the video dataset may not follow optimal paths to any targets. This also leads us to favor Q-learning over the simpler alternative of employing Monte Carlo or TD(0) policy evaluation [53]. Policy evaluation is simpler as it does not involve reasoning about intervening actions, but consequently only learns the value of the underlying policy depicted in the video, rather than the optimal policy. Our experiments demonstrate this contrast between these two design choices, in scenarios where videos don't show the optimal goal reaching behavior.
The learned Q-function, and the associated value function f (I, c), implicitly learn semantic cues for navigation. They can learn what images lead to the desired category, and what don't. Relative magnitude of their prediction can be used to pick directions for exploration. It is worth noting, this obtained value function is the optimal value function under the dynamics of the agent recording the video. We are implicitly assuming that optimal value function under the robot's action space or dynamics would be similar enough. This assumption may not always be true (specially at fine temporal scales), but is true in a number of situations at coarser time scales.

Experiments
We show results on the ObjectGoal task in novel environments [3]. Our experiments test the extent to which we are able to learn semantic cues for navigation merely by watching videos, and how this compares to alternate techniques for learning such cues via direct interaction. We also compare against alternate ways of using passive video data for learning, and show visualizations of our learned value functions.
Video Dataset. We mined for real estate tours from YouTube. This YouTube House Tours Dataset consists of 1387 videos with a total run length of 119 hours. A sample video is shown in supplementary video. We sample a frame every 2 seconds resulting in 170K transitions tuples I t , I t+1 for Q-learning (after pre-processing to remove outdoor scenes and people). We will denote this dataset as V yt .
Experimental Setup. We work with a simulated robot in visually realistic simulation environments. We use the Habitat simulator [48] with the Gibson environments [62] (100 training environments from the medium split, and the 5 validation environments from the tiny split). These environments are derived from scans of real world environments, and thus retain the visual and layout complexity of the real world, but at the same time allow for systematic experimentation.
We split the 105 environments into three sets: E train , E test , and E video with 15, 5, and 85 environments respectively. The robot has access to, and can directly interact with environments in E train . E test is same as the official Gibson tiny validation set that comes with human verified semantic class labels [4]. It is used to setup downstream semantic navigation tasks for evaluation. E train and V yt are used for learning via our proposed formulation. Learned policies are evaluated on E test . For some additional control experiments, we also create a dataset of synthetic videos V syn using the 85 environments in E video (generation procedure described in supplementary). Our splitting procedure ensures: a) final testing happens in novel, previously unseen environments, and b) the robot does not have direct access to environments in which videos were shot (neither the E video used to generate V syn , nor the real estate shown in YouTube House Tours Dataset V yt ).
Robot Model. We use a simplified robot action space with four actions: move forward by 25cm, rotate left 30 • , rotate right 30 • and stop. We assume perfect localization, that is, the robot exactly knows where it is relative to its previous location. This can be achieved by running a SLAM system, or using additional sensors such as an IMU. The robot is a 1.25m long cylinder of radius 10cm, and has a RGB-D camera with 90 • field of view, mounted at a height of 1.25m.
Semantic Visual Navigation Task. We set up the ObjectGoal task [3] in E test for testing different models. Note that E test is same as the Gibson tiny validation set (and does not overlap with environ-ments in E train or E video ), and comes with human-verified annotations for semantic classes. We use these semantic annotations to set up the ObjectGoal task for 5 categories: bed, chair, couch, dining table, and toilet. We sample 1075 test episodes, equally distributed among these 5 classes. For each episode, the agent is initialized at the starting location, and asked to go to the chosen object category. An episode is considered successfully solved if the agent reaches within 1m of any instance of the target category. We report both the success rate and SPL [3]. Minimum geodesic distance to any instance of the target category, is used as the reference path length for computing SPL. We consider two settings: Oracle Stop (episode is automatically terminated and deemed successful when the agent is within 1m of the target category), and Policy Stop (agent needs to indicate that it has reached the goal). We report results along with a 90% bootstrap confidence interval.

Implementation Details
Action Grounding. Inverse model ψ processes RGB images I t and I t+1 using a ResNet-18 model [27], stacks the resulting convolutional feature maps, and further processes using 2 convolutional layers, and 2 fully connected layers to obtain the final prediction for the intervening action. We train ψ on 40K interaction frames gathered by randomly executing actions in E train . We use this inverse model to pseudo-label video dataset V yt and V syn to obtainV yt andV syn .
Object Detectors. We use Mask RCNN [26] trained on MS-COCO dataset [37] as our detector D coco . Frames with detections with score in the top 10% are labeled as +1 reward frames. D coco also predicts a foreground mask for each detection. We use it to evaluate a stopping criterion at test time.
Q-Learning. We represent our Q-function with ResNet 18 models, followed by 1 convolutional layer, and 2 fully connected layers with ReLU non-linearities. We use Double DQN (to prevent chronic over-estimation [57]) with Adam [32] for training the Q-networks, and set γ = 0.99. As our reward is bounded between 0 and 1, clipping target value between 0 and 1 led to more stable training.
Semantic Navigation Policy. High-level policy stores 12 images for each node in the topological graph (obtained by rotating 12 times by 30 • each). It uses the learned value function, f (I, c), to score these 12n images (for a n node topological graph), and samples the most promising direction for seeking objects of category c. The sampled direction is converted into a short-term goal by sampling a location at an offset of 1.5m from the chosen node's location, in the chosen view's direction. Low-level policy [24] uses occupancy maps (built using depth images) [18] with fast marching planning [52] to execute robot actions to reach the short-term goal. It returns control on success / failure / timeout. The High-level policy also factors in the distance to the sampled direction, and score from D coco while sampling directions. Stopping criterion: The agent chooses to stop if D coco fires with confidence ≥ τ c and median depth value in the predicted mask is ≤ d c distance. More details are provided in Supplementary Section S6. Table 1 reports performance on the ObjectGoal task for our method and compares it to other methods for solving this task. An important aspect to consider is the amount and type of supervision being used by different methods. We explicitly note the scale (number of frames, environments) and type (reward signals) of active interaction used by the different methods. For Policy Stop setting, for all methods, we found our stopping criterion to work much better than using the method's own stop signal. We use it for all methods. Using only 40K reward-less interaction samples from E train , along with in-the-wild YouTube videos our proposed method is able to achieve an OS-SPL (Oracle Stop SPL) of 0.5 and PS-SPL (Policy Stop SPL) of 0.21 respectively in the Oracle and Policy stop settings. We put this in context of results from other methods.

Results
Topological Exploration exhaustively explores the environment. It uses our hierarchical policy but replaces f (I, c) with a random function, and ignores scores from D coco to score different directions.
As the topological map grows, this baselines systematically and exhaustively explores the environment. Thus, this is quite a bit stronger than executing random actions (OS-SPL of 0.15). It is able to find objects often (67%), though is inefficient with OS-SPL of 0.30.
Detection Seeker also does topological exploration, but additionally also uses scores from D coco to seek the object once it has been detected. This performs quite a bit better at 0.46 SPL. This indicates that object detectors provide a non-trivial signal for object goal navigation. Even lower confidence detection scores for more distant but partially visible objects will guide the agent in the right direction.
Our method captures more out of view context, and consequently does better across all settings.
End-to-end RL. We also compare against many variants of end-to-end RL policies trained via direct interaction. We use the PPO [51] implementation for CNN+GRU policies that are implemented in Habitat [48]. We modify them to work with ObjectGoal tasks (feeding in one-hot vector for target class, modifying rewards), and most importantly adapt them to use ImageNet initialized ResNet-18 models [27] for RGB (given no standard initialization for Depth image, it is still processed using the original 3-layer CNN in Habitat code-base). The fairest comparison is to train using sparse rewards (dense rewards will require environment instrumentation not needed for our method) in E train for 40K interaction samples with RGB-D sensors. This unsurprisingly did not work (OS-SPL: 0.17 and OS-SR: 37%). Thus, we aided this baseline by providing it combinations of more environments (E train ∪ E video ), many times more samples, and dense rewards. Even in these more favorable settings, end-to-end RL didn't perform well. The best model had a OS-SPL of 0.29 and OS-SR of 56% (vs. 0.50 and 75% for our method), even when given interaction access to 6× more environments, 250× more interaction, and dense rewards (vs. no rewards). This demonstrates the power of our proposed formulation that leverages YouTube videos for learning about spatial layout of environments. Policy stop evaluation is computationally expensive so, we report the score only for the strongest model.
Behavior Cloning (BC) on Pseudo-Labeled VideosV. We pre-process the videos to find trajectories that lead to objects of interest (as determined by D coco ). We train CNN+GRU models to predict the pseudo-labeled action labels on these trajectories. As this is passive data that has already been collected, we are limited to using behavior cloning wth RGB input as opposed to richer inputs or the more sophisticated DAgger [45]. This behavior cloning performs fairly similarly to RL methods and with negligible sample complexity, though still lags far behind our proposed method that utilizes the exact same supervision. Perhaps this is because our proposed method uses pseudo-labeled action indirectly and is more tolerant to mismatch in action space. In contrast, behavior cloning is critically reliant on action space similarity. This is brought out when we useV syn instead ofV yt where the action space is more closely matched. Behavior cloning performs much better at 0.36 OS-SPL, though our method still performs better than all the baselines even when trained on videos inV syn .
Behavior Cloning+RL. We also experimented with combining behavior cloning and RL. We use the behavior cloning policies obtained above, and finetune them with RL. For the same reasons as above, this policy is limited to use of RGB inputs. When finetuning from behavior cloning policy trained on V yt we found performance to remain about the same (OS-SPL 0.23). When starting off from a policy trained onV syn , we found the performance to drop to OS-SPL of 0.24. We believe that the dense reward shaped RL may be learning a qualitatively different policy than one obtained from behavior cloning. Furthermore, use of dense rewards for RL, may limit the benefit of a good initialization.
Strong Supervision Value Function. While our focus is on learning purely from passive data, our semantic navigation policy can also be trained using strong supervision obtained using semantically labeled maps. We train f (I, c) to predict 'ground-truth' Q-values computed using the number of steps to the nearest instance of category c on the meshes from environments in E video . This performs the strongest at OS-SPL of 0.53. This serves as a very competitive ObjectNav policy in the regime where we allow such strong supervision. Our proposed method that uses significantly less supervision (in-the-wild videos from YouTube vs. environment scans) is very close to this oracle at 0.50 OS-SPL.
Thus, in conclusion, value functions learned via our approach from YouTube video tours of indoor spaces are effective and efficient for semantic navigation to objects of interest in novel environments. They compare favorably to competing reinforcement learning based methods, behavior cloning approaches, and strong exploration baselines, across all metrics.

Ablations
We present ablations when testing policies on E train in Oracle Stop setting. Note E train was only used to train the inverse model, and not the Q-learning models that we seek to compare. The base setting from which we ablate corresponds to training f (I, c) onV syn with pseudo-labeled actions, D coco based reward labels, and the use of f (I, c) and spatial consistency for sampling short-term goals. This achieves an OS-SPL of 0.40 ± 0.02. We summarize results below, table in supplementary.
We notice only a minor impact in performance when a) using true actions as opposed to actions from inverse model ψ (0.41 ± 0.03), b) using true detections as opposed to detections from D coco (0.40 ± 0.03), c) using true reward locations as opposed to frames from which object is visible as per D coco (0.41 ± 0.03) (the proposed scheme treats frames with high-scoring detections as reward frames as opposed to true object locations), and d) using optimal trajectories as opposed to noisy trajectories (0.43 ± 0.03). Albeit on simulated data, this analysis suggests that there is only a minor degradation in performance when using inferred estimates in place of ground truth values.
Perhaps, a more interesting observation is that there is a solid improvement when we additionally use D coco score to sample short-term goals (0.46 ± 0.03). We believe use of D coco produces a more peak-y directional signal when the object is in direct sight, where as differences in f (I, c) are more useful at long-range. Secondly, we found that use of 360 • images at training time also leads to a strong improvement (0.47 ± 0.02). We believe use of 360 • images at training time prevents perceptual aliasing during Q-learning. In the base setting, Q-values can erroneously propagate via an image that looks directly at a wall. Presence of 360 • context prevents this. While this is useful for future research, we stick with the base setting as we are limited by what videos we could find on YouTube. Is action pseudo-labeling necessary? As discussed in Section 3.1, we favored use of Q-learning over action agnostic methods, such as policy evaluation, as this allows us to learn optimal value functions as opposed to value of the policy depicted in the video. To test this, we train different methods in the branching environment as shown in the figure on the right (top). Desired goal locations are labeled by G near and G far . We investigate the learned behavior at the branch point B, by initializing the agent at random locations in the circle S. Desired behavior is for agent to reach G near . In departure from all other experiments, here we train and test in the same branching environment. This is a deliberate choice as we seek to understand how different methods interpret the training data.
Videos in this branching environment are a 50 − 49.5 − 0.5% mix of trajectories T 1 , T 2 , and T 3 . T 1 and T 2 are sub-optimal trajectories to reach G near and G far respectively, while T 3 is the optimal trajectory to reach G near . The policy evaluation method doesn't use any action labels, and correctly infers the values for the policy from which videos are sampled. As expected, this causes it to pursue the sub-optimal goal (red paths in bottom figure). In contrast, Q-learning with pseudo-labeled actions, estimates the optimal value function, and consistently reaches G near (green paths).

Discussion
We presented a technique to enable learning of semantic cues for finding objects in novel environments from in-the-wild YouTube videos. Our proposed technique employs Q-learning on pseudo-labeled transition quadruples. This allows learning of effective semantic cues even in the absence of action grounding and goal-directed optimal behavior. When coupled with a hierarchical navigation policy, these cues convey the agent to desired objects more effectively than competitive exploration baselines and RL methods at a fraction of interaction cost. In the future, we will test our policies on real robots and extend to other navigation tasks.  Figure S4: Hierarchical navigation policy. High-level policy does semantic reasoning (using the learned value functions) over images in different directions and outputs short-term goals, that are consumed by the low-level policy. The low-level policy employs classical mapping and planning to achieve the short-term goal, and returns control to the high-level policy if it achieves the short-term goal, or determines it to be infeasible. Black nodes depict nodes stored by the high-level policy in the topological graph, and blue nodes show the value predictions in different directions from each of the black nodes (size indicates predicted value, we use 12 uniformly sampled directions but only show few for clarity). Current location is indicated by the hollow circle. High-level policy outputs the most promising direction to pursue as the short-term goal. Relative offset of this location from the current location (∆Pose) is passed to the low-level policy. Low-level policy incrementally builds occupany map. It uses the fast-marching method to plan a path to the desired short-term goal, and outputs low-level robot actions. Low-level policy returns control on success (reaching the short-term goal), infeasible goal (short-term goal determined to be in occupied space), or timeout.
We use the learned value function f (I, c) from Section 3.1. in a hierarchical navigation policy for semantic navigation. Our hierarchical policy is motivated by Chaplot et al. [10], and consists of a high-level policy and a low-level policy. The high-level policy outputs short-term goals that are achieved by the low-level policy. The high-level policy uses value predictions on images seen so far (at short-term goal locations), to sample a short-term goal in the most promising direction. This short-term goal is expressed as a relative offset from the agent's current location. The low-level policy emits low-level robot actions to navigate to this short-term goal, or returns that the short-term goal is infeasible. This process is repeated, i.e., the high-level policy takes feedback from the low-level policy, along with the image at the agent's new location to sample the next short-term goal. We describe these two policies in more detail below. Figure S4 shows an overview of this navigation policy.

S6.1 High-level policy
The high-level policy, Π builds a hybrid spatial and topological representation. It stores 360 • images along with their locations at each short-term goal location. 360 • images are obtained by incrementally rotating the agent 12 times by 30 • each. High-level policy also stores the value prediction from f (I, c) on these 12 images, for the category of interest c. These 12 values denote the promise of exploring in the different directions for reaching the objects of the desired class. These predicted values are combined with object detector output and a spatial consistency term to give the final score: As it is expensive to get these images (it costs 12 steps), we only store these at locations where the short-term policy returns control to the high-level policy (we call these locations as semantic reasoning locations, and these are marked in Figure S4 with black dots).
The high-level policy maintains a priority heap of all of these 12N values (along with their location and associated direction vectors in the agent's coordinate frame), where N is the number of semantic reasoning nodes currently stored in the topological graph. At each time step, the high-level policy pops the highest of these 12N values 1 from the priority heap, and samples k (= 100) short-term goals in this direction (±7 • ) that are between 1m and 2m from the parent node. These k short-term goals are passed onto the low-level policy, which pursues the first of these k goals that is not known to be infeasible, and returns control to the high-level policy if it succeeds, or determines that the sampled short-term goal is infeasible or too far away.

S6.2 Low-level policy
The low-level policy uses metric occupancy maps [18] along with fast-marching method (FMM) path planners [52] to incrementally plan paths to provided short-term goals. The low-level policy filters the provided k goals for feasibility (using the current occupancy map). It takes the first one of these filtered short-term goals, plans a path to it, and outputs planned robot actions. Low-level policy continues to re-plan when the occupancy map updates. Low-level policy executes actions output from the FMM planner. It stops and returns control when a) it has reached the goal, b) it has already executed enough steps (based on estimate from original FMM computation), or c) the short-term goal turns out to be infeasible or much further than originally anticipated (as more of the map becomes visible). We assume access to depth images, and adapt code from the map and plan implementation from [24], to implement the low-level policy.
As our focus is on high-level semantic cues, for simplicity we assume access to perfect agent pose for this hierarchical policy. This can be achieved using additional sensors on the robot (depth cameras, and IMU units), or using a SLAM system [40], or just with RGB images by using learned pose estimators and free space estimators [9].

S6.3 Stopping Criteria
We elaborate on the stopping criteria used for Policy Stop setting. At every semantic reasoning step, we compute a proxy measure for whether we are close to an object of the desired category or not by using the depth image and Dcoco. For all high-scoring detections for class c from Dcoco (detection score more than τc = 0.75), we approximate the distance to the detected object instance by the median depth value within the predicted instance segmentation mask. If any detected instance is within a distance dc, the agent emits a stop signal. dc is a per-category hyper-parameter (as object sizes vary drastically across categories). We set it using 100 episodes sampled in Etrain.
As noted in Section 4.2, we found that this hand-crafted stopping criteria also led to best performance for all methods that we compare to (as opposed to using the method's own stopping method). Threshold τc was fixed to 0.75 for all methods, while dc was optimized for each category for each method on the same 100 episodes from Etrain using the exact same procedure. For behavior cloning and RL methods, stopping criteria is evaluated at all times steps, where as for our method and baselines based on our method, it is evaluated at every semantic reasoning step. We plot the distribution of difficulty (distance to nearest object of interest) of the evaluation episodes in Etest in figure on right. We group these episodes into 3 difficulty levels, based on distance to the nearest instance of the target category: easy (≤ 3m, green), medium (3m to 5m, orange), and hard (5m to 15m, red). In total there were 313 easy, 324 medium and 438 hard episodes.

S7.3 Generation of V syn
We use environments in Evideo to render out egocentric navigation tours. We employ a path planner to compute shortest path between random pairs of points in each environment. We render out panorama images (4 images: straight facing, left facing, back facing, and right facing, relative to the direction of motion) along these shortest paths and throw out the sequence of actions that were executed, to arrive at the dataset of videos Vsyn. To make these tours more realistic, we execute a random action with 20% probability at each time step (and replan accordingly). We sample 300 trajectories in each of the 85 environments. Average trajectory length is 40 steps.

S7.4 More Implementation Details
We note further implementation details for our method and baselines.
1. Topological Exploration and Detection Seeker are implemented by setting (λ1, λ2, λ3) to be (0, 0, 1), and (0, 1, 1) respectively in Eq. S2. This assures a fair comparison between the three methods, and tests the effectiveness of our learned function f (I, c).

For
End-to-End RL, we experimented with different architectures as noted in Table 1 in the main paper. Baselines as part of Habitat [48] use a 3 layered CNN (denoted by 3CNN and SimpleCNN interchangeably in the main paper) to represent RGB, Depth or RGB-D input. We report performance with this default network (RL (RGB-D 3CNN, RL Depth 3CNN)) in Table 1 in main paper. We found that using a ResNet-18 model (initialized by pre-training on ImageNet) worked better than using this SimpleCNN to represent RGB images. Thus we additionally also reported performance with 3. Our Q-learning models were optimized using Adam with a learning rate of 10 −4 , β1 = 0.9 and β2 = 0.999. Model was trained for 300K mini-batches of size 16 and the model after the last update was used for experiments.

Architecture of Q-network:
The architecture of the Q-network was based off of ResNet-18. We used a ResNet-18 pretrained on ImageNet removing the last convolution layer and all later layers. We add to the pre-trained head, an additional convolution layer with kernel size 3 × 3 and 64 channels.       Figure S7: Example panoramas from novel environments with scores from our value network. Scores for each object class (Bed, Chair, Couch, Dining Table, and Toilet) are reported. We can see that value is high in the likely direction of objects even if the object is not directly visible.   Figure S10: Maps representing the value of different locations in novel environments as predicted by our method trained onV syn . We can see that high value regions fall off smoothly as the distance from object goals increases.  Figure S11: The predicted value in the branching environment using models trained with Q-learning, and policy evaluation via TD(0) and Monte Carlo. We see that the policy evaluation methods drastically under estimate the value in the optimal direction at the branch point. This leads to sub-optimal policies for those methods while the Q-learning based value function finds the optimal trajectory. See Section 4.3 for details.