{'fraction_non_alphanumeric': 0.027724639633571333, 'fraction_numerical': 0.014657146706183484, 'mean_word_length': 4.775011669519216, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning to See by Moving

The dominant paradigm for feature learning in computer vision relies on training neural networks for the task of object recognition using millions of hand labelled images. Is it possible to learn useful features for a diverse set of visual tasks using any other form of supervision? In biology, living organisms developed the ability of visual perception for the purpose of moving and acting in the world. Drawing inspiration from this observation, in this work we investigate if the awareness of egomotion can be used as a supervisory signal for feature learning. As opposed to the knowledge of class labels, information about egomotion is freely available to mobile agents. We show that given the same number of training images, features learnt using egomotion as supervision compare favourably to features learnt using class-label as supervision on visual tasks of scene recognition, object recognition, visual odometry and keypoint matching.


Introduction
Recent advances in computer vision have shown that visual features learnt by training neural networks for the task of object recognition using more than a million labelled images are useful for many computer vision tasks like semantic segmentation, object detection and action classification [22,13,17,1,38]. However, object recognition is one among many tasks for which vision is used. For example, humans use visual perception for recognizing objects, understanding spatial layouts of scenes and performing actions such as moving around in the world. Is there something special about the task of object recognition or is it the case that useful visual representations can be learnt through other modes of supervision? Clearly, biological agents perform complex visual tasks and it is unlikely that they require external supervision in form of millions of labelled examples. Unlabelled visual data is freely available and in theory this data can be used to learn useful visual representations. However, until now unsupervised learning approaches [6,27,34,37] have not yet delivered on their promise and are nowhere to be seen in current applications on complex real world imagery.
It is believed that biological agents developed perceptual systems for obtaining sensory information about their environment that can aid them in performing actions required for accomplishing their goals [16,12]. The motor system of both biological and robotic agents is responsible for executing these actions. The existence of close links between perceptual and motor systems in biological agents is supported by the discovery of neurons in mammalian brain that show increased neural activity not only when the animal itself performs a certain action, but also when the animal perceives another animal performing the same action (Mirror Neurons; [35]). Although motor theories of perception have a long history [7], there has been little work in formulating computational models of perception that make use of motor information. In this work we explore how agents can use information provided by their own motor system as an intrinsic source of supervision for learning useful visual features.
Useful visual features must possess two characteristics -(1) they should be adept at solving a variety of visual tasks and (2) they should enable the agent to perform novel target tasks by learning from minimal demonstrations of the same task performed by an external teacher. For example, if the agent is presented with only a few labeled examples of cars by an external teacher, the agent must be able to accurately identify cars in future images it views.
A motor signal that is readily available to mobile agents is the awareness of their egomotion (i.e. self-motion). For example, the vestibular system provides the sense of orientation in many mammals. In humans and many animals, the brain has access to information about movement of eyes, limbs and the actions that the animal performs [12]. A mobile robotic agent can estimate its egomotion either from the motor commands it issues to move or from odometry sensors like gyroscopes and accelerometers mounted on the robot itself. Stated differently, information about egomotion is available for "free" to mobile agents We hypothesize that agents can learn useful visual representations by performing the simple task of correlating their visual stimuli with their egomotion. As a mobile agent can be treated like a camera moving around in the world, the knowledge of egomotion is same as the knowledge of the how the camera moves. Using this insight, we pose the problem of correlating visual stimuli with egomotion as the problem of predicting the camera transformation from the consequent pairs of images that the agent receives while it moves. Intuitively, the process of predicting camera transformation between two images should force the agent to learn features adept at identifying common visual elements present in both the images(i.e. visual correspondence). This means that egomotion can serve as supervision for learning features that are useful for determining visual correspondence. In the past, features such as SIFT, that were hand engineered for finding correspondences, were found to be very useful for tasks such as object recognition [28,23]. This suggests that features learnt using egomotion may also be useful for such tasks.
In order to test our hypothesis, we trained multilayer neural networks to predict the camera transformation between pairs of images. As a proof of concept, we first demonstrate the usefulness of our approach on the MNIST dataset [26]. We show that if class-label supervision is available only for a limited number of examples, then the features learnt using our method outperform previous approaches of unsupervised feature learning (section 3.3) Next, we evaluated the efficacy of our approach on real world imagery. For this purpose, we used image and odometry data recorded from a car moving through urban scenes, made available as part of the KITTI [15] and the San Francisco (SF) city [10] datasets. This data mimics the scenario of a robotic agent moving around in the world. The quality of features learnt from this data were evaluated on four tasks -(1) Scene recognition on SUN [42] (section 5.1), (2) Visual odometery (section 5.3), (3) Keypoint matching (section 5.2) and (4) Object recognition on Imagenet [36] (section 5.4). Our results shows that for the same number of labelled images, features learnt using egomotion as supervision compare favorably to features learnt using class-label as supervision. To the best of our knowledge, this work provides the first effective demonstration of learning visual representations from non-visual egomotion information in real world setting.
The rest of this paper is organized as following: In section 2 we discuss the related work, in section 3, 4, 5 we present the method, dataset details and we conclude with the discussion in section 6. Barlow [4,3] hypothesized that good visual representations can be learnt by finding features that minimize the redundancy of visual information. This hypothesis inspired research in fully unsupervised learning from visual imagery and led to popular algorithms such as ICA [5,20], autoencoders [24,9], sparse coding [32], restricted boltzmann machines [27,37] and others [6]. The aim of such approaches has been to discover compact and rich representations of images that are also sufficient to reconstruct these images. Another line of work has focused on learning features that are invariant to transformations irrelevant to the identity of objects, either from images [14,34], or from video [41].

Related Work
To the best of our knowledge, we are among the first to explore feature learning from non-visual egomotion information in the real world setting. Closest to our method is the the work of transforming auto-encoders [19] that used non-visual egomotion to reconstruct the transformed image from an input source image. This work was purely conceptual in nature and the quality of learned features was not evaluated. In contrast, our method uses egomotion as supervision by predicting the transformation between two images using a siamese-like network model [11]. Another piece of work by [29] considers modeling spatial transformations using boltzmann machines but did not assess the utility learnt features for performing visual tasks.
In the past, there has been little work on using intrinsic reward signals available in robotic agents for feature learning. [18] proposed using intrinsic reward signals from the robots to learn a system for predicting traversability of paths while the robot explores its environment. [33] trained neural networks for driving vehicles directly from visual input. The recent work of [30] considered joint learning of visual and motor representations by training a system for playing Atari video games directly from image pixels, but their method made use of extrinsic rewards in form of game scores. The goal of these works has been to develop a system for performing a specific task. In contrast, the motivation of our work is to investigate how useful are the visual features learnt using egomotion for a variety of visual tasks.

A Simple Model of Motion-based Learning
When a mobile agent equipped with visual sensors moves from a point A to a nearby point B in its environment (i.e. performs egomotion), it receives two views of the world as its visual inputs. The visual system of the agent is modelled by a randomly initialized Convolutional Neural Network (CNN, [25]). By trying to perform the task of predicting its own egomotion using its visual inputs, the agent optimizes its own visual representations (i.e. optimizing the weights of CNN). As the problem of predicting the egomotion is equivalent to the problem of predicting camera transformation between the images, this task can be modelled as training a Siamese style Convolutional Neural Network (SCNN) [11] (illustrated in figure 2) for predicting the camera transformation between the pair of input images. The process of optimizing the visual representations (i.e. feature learning) has been referred to as pretraining in rest of this paper.

SCNN, BCNN and TCNN
The SCNN is a two streamed CNN, where each stream independently computes features for one image. The two streams of the SCNN share the same architecture and the same set of weights and consequently perform the same set of operations for computing features. We call each of these streams as a Base-CNN (BCNN). Features from the two BCNNs are concatenated and passed downstream into another CNN called as the Top-CNN (TCNN). The TCNN is responsible for using the BCNN features to predict the camera transformation between the input pair of images.
For evaluating the quality of learnt features, the TCNN is removed and a single BCNN is used as a standard CNN for feature computation. For the target tasks of digit classification, scene recognition and keypoint matching, the pretrained model weights were not modified and were used "as is" (i.e. no finetuning) for feature computation. For the target tasks of object recognition and visual odometry, all layers of the BCNN were finetuned for the target tasks.

Shorthand for CNN architectures
The abbreviations Ck, Fk, P, D, Op represent a convolutional(C) layer with k filters, a fully-connected(F) layer with k filters, pooling(P), dropout(D) and the output(Op) layers respectively. We used ReLU non-linearity after every convolutional/fully-connected layer, except for the output layer. The dropout layer was always used with dropout of 0.5. The output layer was a fully connected layer with num-ber of units equal to the number of desired outputs. As an example of our notation, C96-P-F500-D refers to a network with 96 filters in the convolution layer followed by ReLU non-linearity, a pooling layer, a fully-connected layer with 500 unit, ReLU non-linearity and a dropout layer. We used [21] for training all our models.

Proof of Concept using MNIST
MNIST was used for providing a proof of concept because several methods for unsupervised feature learning have been evaluated on this dataset. In the MNIST setting, egomotion of the agent was emulated by generating synthetic data consisting of random transformation (translations and rotations) of digit images. From the training set of 60K images, digits were randomly sampled and then transformed using two different sets of random transformations to generate image pairs. CNNs were trained for predicting the transformations between these image pairs. In section , we describe the details of training and in section 3.3.3 we present the results.

Data
The relative translation between the digits was constrained to be an integer value in the range [-3, 3]. The relative rotation θ was constrained to lie within the range [-30 • , 30 • ]. The prediction of transformation was posed as a classification task with three separate soft-max losses (one each for translation along X, Y axes and the rotation about Zaxis). SCNN was trained to minimize the sum of these three losses. Translations along X, Y were separately binned into seven uniformly spaced bins each. The rotations were Base-CNN Stream-2 Top-CNN Figure 2: Description of the method for feature learning. Visual features are learnt by training a Siamese style Convolutional Neural Network (SCNN, [11]) that takes as inputs two images and predicts the transformation between the images (i.e. egomotion). Each stream of the SCNN (called as Base-CNN or BCNN) computes features for one image. The outputs of two BCNNs are concatenated and passed as inputs to a second multilayer CNN called as the Top-CNN (TCNN) (shown as layers F 1 , F 2 ). The two BC-NNs have the same architecture and share weights. After feature learning, TCNN is discarded and a single BCNN stream is used as a standard CNN for extracting features for performing target tasks like scene recognition.
binned into bins of size 3 • each resulting into a total of 20 bins (or classes). A total of 5 million image pairs were used for pretraining.

Network Architectures
The BCNN architectures used in this work are detailed in table 1. The two BCNN streams were concatenated using this TCNN architecture: F1000-D-Op. SCNN was pretrained for 40K iterations (i.e. 5M examples) using an initial learning rate of 0.01 which was reduced by a factor of 2 after every 10K iterations.
The following architecture was used for classification: BCNN-F500-D-Op. In order to evaluate the quality of BCNN features, the learning rate of all layers in the BCNN was set to 0 when training for digit classification. The network was trained for 4K iterations (which is equivalent to training for 50 epochs for 10K labelled training examples) with a constant learning rate of 0.01.

Results
The BCNN features were evaluated by computing error rates on the task of digit classification using 100, 300, 1K and 10K class-labelled examples for training. These sets were constructed by randomly sampling digits from the standard training set of 60K digits. For this part of the ex- periment, the original digit images were used (i.e. without any transformations or data augmentation). The standard test set of 10K digits was used for evaluation and error rates averaged across 3 independent runs are reported in table 1. Our best performing architecture (C96-P-C256-P), outperforms convolutional deep belief networks [27], a previous approach based on learning features invariant to transformations [34] and features learnt by the default autoencoder network provided with Caffe [21]. These results demonstrate that predicting relative transformation (i.e. egomotion) is a useful mode of supervision for learning visual features adept at digit classification.

Learning Visual Features From Egomotion in Natural Environments
We used two main sources of real world data for feature learning: the KITTI and SF datasets, which were collected using cameras and non-visual odometry sensors mounted on a car driving through urban scenes. Details about the data, the experimental procedure, the network architectures and the results are provided in sections 4.1, 4.2, 4.3 and 5 respectively.

KITTI Dataset
The KITTI dataset provided odometry and image data recorded during 11 short trips of variable length made by a car moving through urban landscapes. The total number of frames in the entire dataset was 23,201. Out of 11, 9 sequences were used for training and 2 for validation. The total number of images in the training set was 20,501.
The odometry data was used to compute the camera transformation between pairs of images recorded from the car. As significant camera transformations in the KITTI data were either due to translations along the Z/X axis or rotation about the Y axis, only these three dimensions were used to express the camera transformation. The rotation was represented by euler angles. The task of predicting the transformation between pair of images was posed as a classification problem. The three dimensions of camera transformation were individually binned into 20 uniformly spaced bins each. As it was unreasonable to expect that visual features can be used to infer big camera transformations, the training image pairs were chosen from images that were atmost ±7 frames apart The SCNN was trained to predict camera transformation from pairs of 227 × 227 pixel sized image regions. Each image was of overall size 370 × 1226 pixels and the coordinates for cropping image regions were randomly chosen for each pair of images. Figure 1 illustrates typical image crops.

SF Dataset
SF dataset provides camera transformation between ∼ 136K pairs of images (constructed from a set of 17,357 unique images). The images and the camera transformation were obtained from Google StreetView data [10]. ∼ 130K image pairs were used for training and ∼ 6K pairs for validation.
Just like KITTI, the task of predicting camera transformation was posed as a classification problem. Unlike KITTI, significant camera transformation was found along all six dimensions of transformation (i.e. the 3 euler angles and the 3 translations). As, it was unreasonable to expect that visual features can be used to infer big camera transformations, rotations between [-30 • , 30 • ] were binned into 10 uniformly spaced bins and two extra bins were used for rotations larger and smaller than 30 • and -30 • respectively. The three translations were individually binned into 10 uniformly spaced bins each. Images were resized to a size of 360 × 480 and randomly cropped image regions of size 227 × 227 were used for training the SCNN.

Network Architecture
The BCNN architecture closely followed the architecture of first five AlexNet layers [22]: C96-P-C256-P-C384-C384-C256-P. The TCNN architecture was: C256-C128-F500-D-Op. The spatial extent of convolutional filters in the TCNN was 3 × 3 for all the layers. The networks were trained for 60K iterations with a batch size of 128. The initial learning rate was set to 0.001 and was reduced by a factor of two after every 20K iterations.
In the remainder of the paper, the networks pretrained on KITTI and SF datasets are called KITTI-Net and SF-Net respectively. Figure 3 shows the layer-1 filters of KITTI-Net and SF-Net. A large majority of layer-1 filters are color detectors, while some of them are edge detectors. As color is a useful cue for determining correspondences between closeby frames of a video sequence, learning of color detectors as layer-1 filters is not surprising. The fraction of filters that detect edges is higher for the SF-Net as compared to the KITTI-Net. One possible explanation for this is the fact that a higher fraction of images in the SF dataset contain structured objects like buildings and cars.

Evaluating Egomotion Based Learning
For evaluating the merits of the proposed approach, features learned using egomotion based supervision were compared against features learned using class-label based supervision on the challenging tasks of scene recognition, intraclass keypoint matching and visual odometry. In addition, the usefulness of egomotion based pretraining for the task of object recognition was also evaluated. The ultimate goal of feature learning is to find features that can generalize from only a few supervised examples on a new task that the agent needs to perform. Therefore it makes sense to evaluate the quality of features when only a few labelled examples for the target task are provided. Consequently, the scene and object recognition experiments were performed in the setting when only 1-50 labelled examples per class were available for training.
The KITTI-Net and SF-Net (examples of models trained using egomotion based supervision) were trained using only only ∼ 20K unique images. To make a fair comparison with class-label based supervision, a model with AlexNet architecture was trained using only 20K images taken from the training set of ILSVRC12 challenge (i.e. 20 examples per class). This model has been referred to as AlexNet-20K. In addition, some experiments presented in this work also make comparison with AlexNet models trained with 100K and 1M images that have been named as AlexNet-100K and AlexNet-1M respectively.

Scene Recognition
We used the SUN dataset for evaluating scene recognition performance. This dataset consists of a total of 397 indoor and outdoor scene categories and provides 10 standard training splits constructed using 1, 5, 10, 20 and 50 training images per class and associated test set splits containing 50 images per class. Due to computational constraints, recognition accuracy was evaluated only for the settings of 5 and 20 labelled examples per class and 3 train/test splits.
For evaluating the utility of CNN features produced by different layers, separate linear (SoftMax) classifiers were trained on features produced by individual CNN layers (i.e. BCNN layers for KITTI-Net and SF-Net). Table 2 reports recognition accuracy (averaged over 3 train/test splits) for various networks considered in this study. The performance of KITTI-Net is superior to SF-Net and comparable to AlexNet-20K. This indicates that given a fixed budget of pretraining images, egomotion based supervision learns features that are almost as good as the features using classbased supervision on the task of scene recognition.
The KITTI-Net outperforms hand-engineered GIST [31] features developed for scene classification, but is outperformed by Dense SIFT with spatial pyramid matching (SPM) kernel [23]. The KITTI-Net was trained using limited visual data (≈ 20Kf rames) containing visual imagery of limited diversity. The KITTI data majorly contains images of roads, buildings, cars, few pedestrians, trees and some vegetation. It is in fact surprising that a network trained on data with such little diversity is competitive on classifying indoor and outdoor scenes with the AlexNet-20K that was trained on a much more diverse set of images. We hope that with more diverse training data for egomotion based learning, the performance of learnt features will be superior than currently reported numbers.
The KITTI-Net outperforms the SF-Net except when the performance of first layer (L1, conv-1 layer of AlexNet architecture) is compared. From figure 3 notice that layer 1 of SF-Net learns a greater number of edge filters. This may explain why the performance of L1 for SF-Net is higher. The KITTI-Net performed better than SF-Net possibly because a greater number of 227 × 227 image region pairs were available for egomotion based training from the KITTI dataset (see section 4.1, 4.2). As KITTI-Net was found to be superior to the SF-Net in this experiment, the KITTI-Net was used for all other experiments described in this paper.

Intra-Class Keypoint Matching
Identifying the same part (or keypoint) of an object across different instances of the same object class (for instance, identifying "eyes" among different instances of a dog) is an important visual task. Visual features learned by motion based and class-label based supervision were evaluated for this task using the keypoint annotations on the PAS-CAL dataset [8].
The keypoint matching was calculated in the following way: First, ground-truth object bounding boxes (GT-BBOX) from PASCAL-VOC2012 dataset were extracted and resized (while preserving the aspect ratio) to ensure that the smaller side of the boxes was of length 227 pixels. Next, feature maps from layers 2-5 of AlexNet-10K, AlexNet-100K, AlexNet-1M and KITTI-Net were computed for every GT-BBOX. The keypoint matching score was computed between all pairs of GT-BBOX belonging to the same object class. For given pair of GT-BBOX, the features associated with keypoints in the first image were used to predict the location of the same keypoints in the second image. The normalized pixel distance between the actual and predicted keypoint locations has been referred to as error in the keypoint matching. More details about this procedure have been provided in appendix A.
Intuitively, matching keypoints across object instances that are related by a large transformation (i.e. viewpoint distance) should be harder than matching keypoints across instances with a small viewpoint distance. In order to obtain a holistic understanding of the quality of features learnt using different modes of supervision on this task, matching error was computed as a function of viewpoint distance [40]. Figure 4 reports the matching error averaged across all keypoints, all pairs of GT-BBOX and all classes using features extracted from layers conv-3 and conv-4. Matching error for layers 2 and 5 have been provided in appendix A.4 The KITTI-Net, trained only with 20K unique frames was superior to AlexNet-20K and AlexNet-100K and inferior only to AlexNet-1M. A network with AlexNet architecture initialized with random weights (Alexnet-Rand), surprisingly performed better than AlexNet-20K. One possible explanation for this observation is that with only 20K examples, features learnt by AlexNet-20K only capture coarse global appearance of objects and are therefore poor at keypoint matching. SIFT has been hand engineered for finding correspondences across images and performs as well as the best Alexnet-1M features for this task (i.e. conv-4 features). See figure 5 for a visualization of the keypoint matching results.
The results of this experiment indicate that egomotion based supervision is superior to class-label based supervision for learning features adept at keypoint matching. These results further support the intuition that different kinds of supervision can provide features that are useful for different tasks.

Visual Odometry
Visual odometry is the task of predicting the egomotion (or camera transformation) between pairs of input images. The performance of SCNNs formed by using KITTI-Net, Table 2: Comparing the accuracy of neural networks pre-trained using motion-based and class-label based supervision for the task of scene recognition on the SUN dataset. The performance of layers 1-6 (labelled as L1-L6) of these networks was evaluated after finetuning the network using 5/20 images per class from the SUN dataset. The performance of the KITTI-Net (i.e. motion-based pretraining) fares favorably with a network pretrained on Imagenet (i.e. class-based pretraining) with the same number of pretraining images (i.e. 20K). AlexNet-1M and a network with AlexNet architecture initialized with random weights as BCNNs were compared on the task of predicting camera transformations between image pairs taken from the validation set of the SF dataset. All layers of the SCNNs were finetuned for 25K iterations using image pairs from the training set of SF dataset. Table 3 reports the performance of different networks. The KITTI-Net was found to be either superior or as good as the AlexNet-1M on the task of visual odometry prediction. This indicates that on this task, the features learnt using motion based supervision using only 20K unique images are as good as features learnt using class-label based supervision using 1M images.

Object Recognition
If it is the case that motion based pretraining learns useful features for object recognition, a network intialized with weights from KITTI-Net should outperform a network intialized with random weights on the task of object recognition. For testing this, four different networks (intialized with random weights) were trained using 1, 5, 10 and 20 images per class from the ILSVRC-2012 challenge. As this dataset contains 1000 classes, the total number of training examples available for training for these networks were 1K, 5K, 10K and 20K respectively. Four more networks, each initialized from KITTI-Net weights were also finetuned us- Table 4: Top-5 accuracy on the task of object recognition on the ILSVRC-12 validation set. The train from scratch network was initialized with random weights and the KITTI-Net used weights learned using egmotion-based supervision on the KITTI dataset. These networks were finetuned using 1, 5, 10, 20 examples per class. The KITTI-Net clearly outperforms the network trained from scratch.

Discussion
In this work, we have shown that egomotion is a useful source of intrinsic supervision for visual feature learning in mobile agents. In contrast to class labels, knowledge of egomotion is "freely" available. On MNIST, egomotion-based feature learning outperforms previous unsupervised methods of feature learning. Given the same budget of pretraining images, on task of scene recognition, egomotion-based learning performs almost as well as class-label-based learning. Further, egomotion based features outperform features learnt by a CNN trained using class-label supervision on two orders of magnitude more data (AlexNet-1M) on the task of visual odometry and one order of magnitude more data on the task of intra-class keypoint matching. In addition to demonstrating the utility of egomotion based supervision, these results also suggest that features learnt by  AlexNet trained for object recognition may not be generic for a wide variety of visual tasks. This means that future work should look at what kinds of supervision are useful for feature learning. One potential criticism of our work is that we have trained and evaluated high capacity deep models on relatively little data (e.g. only 20K unique images available on the KITTI dataset). In theory, we could have learnt better features by downsizing these networks. For example, in our experiments with MNIST we found that pretraining a 2-layer network instead of 3-layer results in better performance (table 1). In this work, we have made a conscious choice of using standard deep models because the main goal of this work was not to explore novel feature extraction architectures but to investigate the value of egmotion for learning visual representations on architectures known to perform well on practical applications and previous work [1] has shown that such networks outperform conventional visual features when trained using only about ∼ 10K images. While egomotion is freely available to mobile agents, there are currently no publicly available datasets as large as Imagenet. Consequently, we were unable to evaluate the utility of motion-based supervision across the full spectrum of training set sizes. Future research focused on exploring architectures that are better suited for egomotion based learning and more nuanced learning formulations for using egomotion as supervision can only make a stronger case for this line of work.
In this work, we chose to first pretrain our models using a base task (i.e. egomotion) and then finetune these models for target tasks. A more interesting setting is that of online learning where the agent has continuous access to intrinsic supervision (such as egomotion) and occasional explicit access to extrinsic teacher signal (such as the class labels). We believe that such a training procedure is likely to result in learning of better features. Our intuition behind this is that seeing different views of the same instance of an object (say) car, may not be sufficient to learn that different instances of the car class should be grouped together.
The occasional extrinsic signal about class labels may prove useful for the agent to learn such concepts.
Another worthwhile direction of future research is to revisit active vision [2]. Our current work makes use of passively collected egomotion data and it would be interesting to investigate if the agent is able to construct better visual representations if it is allowed to decide on how it explores its environment.
• Pad the images by 30 pixels along each dimension.
• Resize each image so that the smallest side is 227 pixels. The aspect ratio of the image is preserved.

A.1. Keypoint Matching using CNN
Assume that the l th layer of the CNN is used for feature computation. The feature map produced by the l th layer is of dimensionality I × J × M , where (I, J) are the spatial dimensions and M is the number of filters in the l th layer. Thus, the l th layer produces a M dimensional feature vector for each of the I × J grid position in the feature map.
The coordinates of the keypoints are provided in the image coordinate system [8]. For the keypoints in the first image, we first determine their grid position in the I × J feature map. Each grid position has an associated receptive field in the image. The keypoints are assigned to the grid positions for which the center of receptive field is closest to the keypoints. This means that each keypoint is assigned one location in the feature map.
Let the M dimensional feature vector associated with the k th keypoint in the first image be F k 1 . Let the M dimensional feature vector at grid location C ij for the second image be F 2 (C ij ). The location of matching keypoint in the second image is determined by solving: C * is transformed into the image coordinate system by computing the center of receptive field (in the image) associated with this grid position. Let this transformed coordinates be C im * and the coordinates of the corresponding keypoint (in the second image) be C im gt . The matching error for the k th keypoint (E k ) is defined as: where, L 2 D is the length of diagonal (in pixels) of the second image. As different images have different sizes, dividing by L 2 D normalizes for the difference in sizes. The matching error for a pair of images of instances belonging to the same class is calculated as: The average matching error across all pairs of the instance of the same class is given by E class : where, #pairs is the number of pairs of object instances belonging to the same class. In Figure 4 of the main paper we report the matching error averaged across all the 20 classes.

A.2. Keypoint Matching using SIFT
SIFT features are extracted using a square window of size 72 pixels and a stride of 8 pixels using the open source code from [39]. The stride of 8 pixels was chosen to have a fair comparison with the ConvNet features. The ConvNet features were computed with a stride of 8 for layer conv-2 and stride of 16 for layers conv-3, conv-4 and conv-5 respectively. The matching error using SIFT was calculated in the same way as for the CNNs.

A.3. Effect of Viewpoint on Keypoint Matching
Intuitively, matching instances of the same object that are related by a large transformation (i.e. viewpoint distance) should be harder than matching instances with a small viewpoint distance. Therefore, in order to obtain a holistic understanding of the accuracy of features in performing keypoint matching it is instructive to study the accuracy of matching as a function of viewpoint distance. [40] aligned instances of the same class (from PASCAL-VOC-2012) in a global coordinate system and provide a rotation matrix (R) for each instance in the class. To measure the viewpoint distance, we computed the riemannian metric on the manifold of rotation matrices ||log(R i R T j )|| F , where log is the matrix logarithm, ||.|| F is the Frobenius norm of the matrix and R i , R j are the rotation matrices for the i th , j th instances respectively. We binned the distances into 10 uniform bins (of 18 • each). In Figure 4 of the main paper we show the mean error in keypoint matching in each of these viewpoints bin. The matching error in the k th bin is calculated by considering all the instances with a viewpoint distance ≤ k×18 • , for k ∈ [1,10]. As expected we find that keypoint matching is worse for larger viewpoint distances.

A.4. Matching Error for layers 2 and 5
The matching errors using features computed from layers 2 and 5 of different networks are shown in figure 6. kitti-20k-cv5 alexnet-rand-cv5 alexnet-20k-cv5 alexnet-100k-cv5 alexnet-1M-cv5 sift Figure 6: Intra-class keypoint matching error as a function of viewpoint distance averaged over 20 PASCAL objects using features extracted from layers pool-2 (left) and conv-5 (right) of various networks used in this work. Please see section 5.2 for more details.