{'fraction_non_alphanumeric': 0.026929400050941254, 'fraction_numerical': 0.00893787482344224, 'mean_word_length': 4.804838709677419, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Road with 16 Neurons: Mental Imagery with Bio-inspired Deep Neural Networks

This paper proposes a strategy for visual prediction in the context of autonomous driving. Humans, when not distracted or drunk, are still the best drivers you can currently find. For this reason we take inspiration from two theoretical ideas about the human mind and its neural organization. The first idea concerns how the brain uses a hierarchical structure of neuron ensembles to extract abstract concepts from visual experience and code them into compact representations. The second idea suggests that these neural perceptual representations are not neutral but functional to the prediction of the future state of affairs in the environment. Similarly, the prediction mechanism is not neutral but oriented to the current planning of a future action. We identify within the deep learning framework two artificial counterparts of the aforementioned neurocognitive theories. We find a correspondence between the first theoretical idea and the architecture of convolutional autoencoders, while we translate the second theory into a training procedure that learns compact representations which are not neutral but oriented to driving tasks, from two distinct perspectives. From a static perspective, we force groups of neural units in the compact representations to distinctly represent specific concepts crucial to the driving task. From a dynamic perspective, we encourage the compact representations to be predictive of how the current road scenario will change in the future. We successfully learn compact representations that use as few as 16 neural units for each of the two basic driving concepts we consider: car and lane. We prove the efficiency of our proposed perceptual representations on the SYNTHIA dataset. Our source code is available at https://github.com/3lis/rnn_vae


Introduction
Road traffic injuries are the leading cause of death for the age group between 5 and 29 years [59]. In the Global status report on road safety of 2018, the World Health Organization reported that the number of road traffic deaths was 16 times larger than in war conflicts. This suggests that avoidance or mitigation of harm caused by motor vehicle accidents will be probably the most beneficial outcome one can expect from artificial intelligence and automation [15]. In fact, in the US only 2% of vehicle crashes are due to technical failures. Among the major causes of accidents are driver's inattention, fast or reckless driving, illegal maneuvers, and tiredness [51].
Self-driving cars will be immune to all the risky factors depending on human drivers. The development of fully autonomous vehicles has always be considered a coveted achievement for the modern society. The research on this field has a long history that dates back to the late 70s [11], but it became a reality -at an unusually fast pace -no longer than a decade ago [25]. While most of the components of a self-driving system (such as sensors) have improved at the typical rate of technological progress without any specific crucial innovations, the impressive advances have been mainly fueled by the emerging "deep" version of artificial neural networks [22,50,32]. Since their early beginnings, the greatest challenge for autonomous driving systems is the perception and understanding of the road environment, and this is precisely the most successful field of application of deep neural models [33,54,56]. Therefore, deep neural models have quickly become the method of choice for driving scene perception [6,4,18,58]. However, despite the impressive progress, perception remains the major obstacle towards fully autonomous vehicles. The core of this issue can be identify in the narrow conception of "perception" usually assumed in autonomous driving, which lacks the fundamental aspect of gathering knowledge about objects and events in the environment, to the point of being able making predictions for action [35,26].
In this respect, it might be useful to reflect on how humans are able to drive. When not distracted, or asleep, or deliberately engaged in dangerous maneuvering, humans are excellent at driving, as at many other complex and highly specialized sensorimotor behaviors. How the brain realizes such sensorimotor behaviors is far from being fully understood, but there are few general neurocognitive theories trying to shed light on this. We deem it useful to borrow two theoretical ideas, in particular, to design the perception strategy of autonomous vehicles.
The first neurocognitive theory we take inspiration from concerns how sensorial information is coded into lowdimensional representations. These perceptual representations are able to capture aspects relevant to actions, and they allow their original content to be recreated in an approximated form during the phenomenon of mental imagery [29,37]. One of the first piece of evidence of such representations was found in the work of Damasio [10], who identified neuron ensembles exhibiting a convergent structure, where neural signals are projected onto multiple cortical regions in a many-to-one fashion. Damasio later developed a broader theory [36] identifying more sophisticated neural structures he called convergence-divergence zones (CDZs). In this case, the very same neuron ensembles are able to perform both convergent and divergent projections, depending on the current action the brain is engaged with: the convergent flow is dominant during perceptual recognition, while the divergent flow occurs during mental imagery. CDZs were recognized as a crucial component in the formation of concepts in the brain [40]. Therefore, we believe it useful to design a model with a similar hierarchical architecture to learn the abstract concepts relevant to the driving context.
The second theoretical idea concerns the nature of the neural representations in the brain. In most cases, neural representations are not neutral but functional to the predic-tion of the state of affairs in the future environment. Actually, the ability to predict appears to be the main goal of intelligence [27,20]. There is evidence for the existence of various circuits in the brain that provide prediction from perceptual representations. In particular, two forms of prediction -procedural and declarative -are typically acknowledged in different brain structures [12]. However, one of the most popular theories in the field interprets the mental mechanism of prediction in mathematical terms [16,17]. This theory, called predictive brain, explains the behavior of the brain as minimization of the free-energy, a quantity that can be expressed in mathematical form. We will show how this formulation can actually be adopted as loss function to train our model.
The aim of our work is to learn conceptual representations of the driving scenario from visual information. Our intention is to learn compact and informative representations that can be useful for a variety of downstream driving tasks. We propose a cognitive-inspired approach that enforces the representations to be not neutral but oriented to the driving tasks, under two distinct perspectives. From a static perspective, we force groups of neural units in the compact representation to distinctly represent specific concepts which are crucial in the driving task. Specifically, we use as few as 16 neurons for each of the two basic concepts: cars and lanes. From a dynamic perspective, we encourage the compact representations to be predictive of how the current road scene would change in the future.
We achieve the conceptual representations by implementing an artificial neural model that is in line with the two aforementioned neurocognitive theories. The term "neural" in artificial neural models by no means implies a faithful replication of the computations performed by biological neurons. On the contrary, the mathematics of deep learning bears little resemblance to the way brain works [46,9]. However, we identify two methods within the framework of artificial neural networks (ANNs) that appear, at least in part, rough algorithmic counterparts of the neurocognitive theories described above. Specifically, the CDZs may find a correspondence in the idea of convolutional autoencoders [55], while the predictive brain theory resonates with the adoption of Bayesian variational inference in combination with autoencoders [28,45].
This work is part of the H2020 Dreams4Cars 1 project, aimed at developing an artificial driving agent inspired by the neurocognition of human driving [43]. In the following section we discuss the most significant related works. In §3 we describe the implementation of 4 different neural models that successfully learn informative and compact representations. Lastly, Section §4 presents the results of our models on the SYNTHIA dataset.

Related Works
It is not uncommon for works adopting ANNs for perception in autonomous vehicles to declare virtues of a neurocognitive inspiration [41,7,61]. Among these are important claims, but they do not transfer the specific brain mechanisms into algorithms. To the best of our knowledge, the two neurocognitive principles embraced by this work -Damasio's CDZs and Friston's predictive brain -have not been proposed in any work on perception for autonomous driving. In addition, the striking similarity between the formulation of brain predictivity given by Friston and the variational autoencoder algorithm seems to remain unnoticed, with few exceptions [39].
The idea of autoencoder has been at the heart of the "deep" turn of ANNs [23,30,21], and their variational version has rapidly gained attention [60]. However, it has not yet been widely adopted for autonomous vehicle perception. One of the most popular strategies, instead, is the endto-end approach, where images from a front-facing camera are fed into a stack of convolutions followed by feedforward layers, generating low-level commands. The first attempt in this direction dates before the rise of deep learning [38], and it has been the groundwork for the later NVIDIA's PilotNet [3,4]. One of the most serious drawbacks of end-to-end systems based on static frame processing is the erratic variation of steering wheel angle within short time periods. A potential solution is to provide temporal context in the models, combining convolutions with recurrent networks [14].
But the most appealing feature of the end-to-end strategy -to dispense with internal representations -is also the major source of its troubles. Learning the entire range of road scenarios from steering supervision alone, with all possible appearances of objects relevant to the drive, is not achievable in practical settings. For this reason several more recent proposals suggest inclusion of intermediate representations, such as the so-called mid-to-mid strategy [2] used in ChauffeurNet, Waimo's autonomous driving system. ChauffeurNet is essentially made of a convolutional network which consumes the input data to generate an intermediate representation with the format of top-down view of the surrounding area and salient objects. In addition, Chauf-feurNet has several higher-level networks which iteratively predict information useful for driving. In [57] the main aim is to overcome the object agnosticism of the end-to-end approach, the Authors propose a object-centric deep learning system for autonomous vehicles. In their proposal there is one convolutional neural module taking an RGB image and producing an intermediate representation. Then, the downstream networks are diversified depending on a taxonomy of objects-related structures in the intermediate representation, which are lastly converted into discrete driving actions. An internal representation is used also in the system by Valeo Vision [52], constructed using a standard ResNet50 model [19] with the top fully-connected layers removed. The feature representation is shared across a multitude of tasks relevant to visual perception in automated driving such as object detection, semantic segmentation, depth estimation. All the downstream tasks are realized using the top parts of standard models such as YOLO [44] for object detection or FCN8 [34] for semantic segmentation.
None of the works reviewed so far builds the internal representations through the idea of autoencoder. We found just two notable exceptions in the field of perception for autonomous driving. The first one is by comma.ai [49], in their model the latent representation has dimension 2048 neurons, and it is obtained with a variational autoencoder that restores the input images of 160 × 80 pixels with 4 deconvolutional layers. Once trained, the latent representations are used for predicting successor frames in time with a recurrent neural network. The second exception is a work by Toyota in collaboration with MIT [1], using a variational autoencoder of dimension 25 neurons. This entire internal representation is decoded to restore the input image of size 200 × 66 as in a standard autoencoder. In addition, one neuron of the representation is interpreted as steering angle, therefore an end-to-end supervision for this neuron is mixed in the total training loss.
There are similarities between these last two approaches and the one we present, but also fundamental differences. The latent compact representation of Amini et al. does not take into account the crucial time dimension of the perceptual driving scenario. On the other hand, Santana and Hotz include their internal representation in a recursive network for prediction, but time dependency is not exploited when learning the compact representation. Moreover, the comma.ai's model is agnostic about the meaning of the neurons composing the latent representation, while Amini et al. assign meaning to just the single neuron coding steering angles. One key strategy in our model is to encourage the assignment of conceptual meaning to segregate groups of neurons in the latent representation. In contexts different from autonomous vehicles the idea is not new. For example, [31] in human heads generation [31] proposed a latent space with separate representations for viewpoints, lighting conditions, and shape variations. Also in [62] the latent vector is partitioned in semantic content and geometric coding.

The Neural Models
During the development of this work, we experimented a number of different architectures, all sharing the common feature of an hierarchical arrangement similar to the CDZs in the brain, according to the strategy described in the Introduction. The first group of neural models was developed with the aim of generating a latent representation of the driving environment. The challenge here is to ensure the latent space is informative enough to represent the wide va-riety of driving scenarios, but keeping at the same time a low dimensionality so that it can still be inspected and explicable.
Here we present: • three different neural network (Net1, Net2, Net3) with encoder-decoder architectures, adopting increasingly sophisticated approaches to learn compressed and disentangled latent representations, • a fourth neural network (Net4) performing prediction in time of future driving scenarios as a rudimentary form of mental imagery, working exclusively within the latent representation created by the three previous models.

Net1: Variational Autoencoder
When talking about representation learning, the first architecture that comes to mind is the autoencoder. This is the simplest model of the family, composed of two subnetworks: The first sub-network is called encoder and computes the compact representations z ∈ Z of a high dimensional input x ∈ X . This network is determined by its set of parameters Φ. The second sub-network is the decoder, often called the generative network, which reconstructs high-dimensional data x ∈ X taking as input low dimensional compact representations z ∈ Z. The network is fully determined by the set of parameters Θ. When training the autoencoder, the parameters Θ and Φ are learned by minimizing the error between input samples x i and the outputs f (g(x i )). The next big improvement in the field of representation learning is the variational autoencoder, we refer to A for a detailed mathematical definition. The variational autoencoder is able to learn a more ordered representation with respect to the standard autoencoder. However, there is a lot space for improvements, especially in our case where we want to focus only on learning representations of driving scenarios. Therefore, we implemented the variational autoencoder model mainly with the intent to be used as a baseline for further improvements and comparison. Table 4 shows the numbers of layers and the parameters adopted in the final version of the variational autoencoder (Net1). The input of the network is a single RGB image of 256 × 256 pixels, the encoder is composed of a stack of 4 convolutions and 2 fully-connected layers, converging to a latent space of 128 neurons. The decoder has a symmetric structure with respect to the encoder, mapping the 128 neurons back to an image of 256 × 256. The network is trained to optimize the loss function in equation (27) in a totally unsupervised way.

Net2: Topological Autoencoder
As discussed in the Introduction, our brain naturally projects sensorial information -especially visual -into CDZs representations, including all those that constitute the conceptual space, where neural activations represent the nature of entities present in the environment that produced the perceptual stimuli. Hence, we can take inspiration from this theory and use the CDZs hierarchical architecture as a "blueprint" to design a more sophisticated neural network, able to learn representations that are not only in terms of visual features, but also in terms of concepts. In the driving context the entire road scenario is informative, however, from a conceptual point of view it is not immediately necessary to infer categories for every entity present in a scene. It is useful to project in conceptual space only the entities mostly relevant to the driving task. Therefore, for simplicity, in this model we choose to consider the two main concepts of cars and lanes . Fig. 2 depicts the architecture of this topological autoencoder (Net2), composed of one shared encoder and three independent decoders. The choice of parameters is similar to the architecture of our variational autoencoder (Net1), Table 5 shows the parameters of the final model considered. The encoder and each of the 3 decoders maintain the same structure as before, even the overall number of neurons in the latent space is the same, it is their inner organization that is strongly improved. As shown in Fig. 2, the grey decoder is the one reconstructing in the visual space -just like the decoder of Net1 -mapping all the 128 neurons of the latent vector z into an RGB image. This decoder learns to reconstruct the input image therefore is trained in a totally unsupervised way. The decoder colored in green, instead, takes only a vector z C of 16 neurons of the latent space and produces a matrix x C of 256 × 256 probability values. The latent vector of 16 neurons is trained to represent the cars concept, and the output matrix can be interpreted as a semantic segmentation of the input image, where values indicates the probability of presence of cars entities. Similarly, the violet decoder maps only the vector z L of 16 neurons representing the lanes concepts into a probability matrix x L for lanes entities. These two decoders require a supervised learning: their output is converted into binary images, by applying a threshold, and trained to minimize the reconstruction error with semantic segmentation of the input images.
Note that the idea of partitioning the entire latent vector into meaningful components is not new, as mentioned in §2. But our approach is different: while we keep the two segments z C and z L disjointed, it is the entire z that learns representations in visual space. This way, we adhere entirely to the CDZ theoretical idea, and explicitly encourage the network to pay attention to the cars and lanes entities in the environment. We would like to highlight another advantage of our approach in partitioning the latent space, concerning the crucial issue of lack of transparency in deep neural network. In most models no information is available about what exactly make them arrive at their predictions [48,42]. By assigning meaning to components of the inner representation, the issue is mitigated.
To give a mathematical description, the overall model is composed of 4 sub-networks: where the subscript V denotes the visual space, and the subscripts C and L refer to the cars and lanes concepts respectively. For each vector z in the latent space: In the first expression, z C and z L are the two segments inside the latent vector z representing the cars and lanes concepts, respectively. The segment in between, z, encodes the remaining generic visual features, and the entire latent vector z is a representation in the visual space. In the final version of the model, we choose to have N V = 128 and N C = N L = 16.
By calling Θ = [Θ V , Θ C , Θ L ] the vector of all parameters in the three decoders, the loss functions of the model is derived from the basic equation (27). At each batch iteration b, a random batch B ⊂ D is presented, and the following loss is computed: where Few observations are due for the differences between this loss function and the basic one (27). First of all, there is a delay in including the contribution of the Kullback-Leibler divergence in the term E K , because initially the encoder is unlikely to provide any meaningful probability distribution q Φ (z|x). There is a cost factor for the KL component, set initially at a small value k 0 and gradually increased up to 1.0, with time constant κ. This strategy is called KL annealing and was first introduced in the context of variational autoencoders for language modeling [5]. The remaining terms E V , E C , E L are errors in the reconstruction of driving scenario and conceptual entities, and their relative contributions are weighted by the parameters λ V , λ C , λ L . The purpose of these parameters is mainly to normalize the range of the errors, which is quite different from visual to conceptual spaces. For this reason, typically λ V = λ C = λ L . The term E V computes the error in visual space, using the entire latent vector z, and corresponds precisely to the second component in the basic loss (27). The last two terms E C and E L compute the error in the conceptual space and are slightly different. Only the relevant portion of the latent vector z is considered, as indicated by the projection operators Π C , Π L . In addition, a variant of the standard cross entropy is used, indicated with the symbols p ΘC and p ΘL , in order to account for the large unbalance between the number of pixels belonging to a concept and all the other pixels, which is typical in ordinary driving scenes. Following the method first introduced in the context of medical image processing [53], we compensate this asymmetry by weighing the contribution of true and false pixels with P , the ratio of true pixels over all the pixels in the dataset, computed as follows: where N is the number of pixels in an image, M is the number of images in the training dataset. The parameter s is used to smooth the effect of weighting by the probability of ground truth, a value evaluated empirically as valid is 4.

Net3: Temporal Autoencoder
Our third model aims at including in the compact representation also the predictivity the future state of affairs. The idea here is to enforce the model to learn representations that are consistent also in the temporal dimension. By imposing temporal consistency, we aim at further regularizing the network latent representations (whereas longer-term predictions will be the subject of the next Net4). Now we combine in the latent space the distinct representations of crucial concepts together with the ability to predict of how such concepts will change in future driving scenarios. Let us introduce the notation x (t) to indicate the frame t steps ahead of frame x. Similarly, z (t) refers to the the latent representation of the image t steps ahead to that represented by z. Fig. 3 shows the architecture of this temporal autoencoder (Net3), with the final parameters described in Table 6. The model has substantially the same architecture of the topological autoencoder (Net2), except for an additional module based on a simple recursive neural network. The training procedure is also different: at each iteration, two subsequent frames x and x (1) are fed as input to the common encoder, which computes two latent representations z and z (1) . These two latent vectors are fed to a RNN trained to predict the latent vector z (2) containing the representation of the consecutive frame in the sequence. All three latent vectors are then expanded using the same 3-decoders structure already seen in Net2, so the overall model is trained to generate visual and segmented output images for all the three frames x, x (1) , x (2) .
The novel sub-network of the model can be described by the function: where h Ψ plays the role of an autoregressive dynamic model of order 2, and it is implemented using a basic recursive neural network (RNN) [13] with time window of 2 and set of parameters Ψ. The formulation of the loss used in training the network is similar to equation (4) with additional terms for the recursive prediction: L(Θ, Φ, Ψ|B) = L(Θ, Φ|B)+ where the first term is the same loss of equation (4), and the expressions of the new terms are the following: The contributions of equations (12), (13), (14) is similar to that of E V , E C , E L , i.e. the errors in the reconstruction of the driving scenario and conceptual entities applied to the successor of z. The predictive relevance is carried by the errors in equations (15), (16), (17) where the 3 decoders are applied to a latent vector resulting from the recursive subnetwork h Ψ (·).

Net4: Recurrent Network
Once trained, the temporal autoencoder (Net3) can be deployed in its encoding part to generate latent representations for all the images of the dataset, and the actual prediction in time can be implemented with a novel recursive network that works in the latent space only. The advantage of a very compact representation allows a much more complex recursive network than the simple h Ψ (·) used in Net3.
As shown in Fig. 4, the proposed recurrent network (Net4) has a first module composed of multiple levels of stacked recurrent sub-networks, and a second module composed of multiple parallel recurrent sub-networks, each x, x (1) RNN z, z (1) z (2) x, x (1)    predicting future frames with increasing time steps ahead. Each stacked recurrent sub-network feeds its entire output sequence as input of the next stacked sub-network. Instead, the parallel recurrent sub-networks yield only the last output in time. All sub-networks of this model share the same core architecture: we experimented the same simple RNN [13] used for h Ψ (·), Long Short-Term Memory (LSTM) [24] and Gated Recurrent Units (GRU) [8], using the latter as final choice.
The mapping of the proposed recurrent model can be described as a function r Ξ : Z NI → Z NO where N I is the number of consecutive inputs, N O is the number of predictions in time, and Ξ is the set of parameters of the model. The function r Ξ (·) is used as follows: In the final version of the model, we choose N I = 8 and N O = 4, moreover we use 2 stacked GRUs and 4 parallel GRUs, as described in Table 7.

Dataset
We choose to train and test our models on the SYNTHIA benchmark [47]. The dataset offers a large collection of image sequences representing various driving scenarios. It is realized using the game engine Unity, and it is composed of ∼ 100k frames of urban scenes recorded from a simulated camera on the windshield of the ego car. We allocated 70% of the dataset to the training set, 25% to validation and 5% to the test set. Despite being generated in 3D computer graphics, this dataset offers a wide variety of quite realistic illumination and weather conditions, resulting occasionally even in very adverse driving conditions. Each driving sequence is replicated on a set of different environmental conditions including seasons, weather and time of the day. Moreover, the urban environment is very diverse as well, ranging from driving on freeways, through tunnels, congestion, "NewYorklike cities" and "European towns" -as the creators of the dataset describe it.

Results of Topological and Temporal Autoencoders
Here we present the comparison between the results of the topological and the temporal autoencoder (Net2 and Net3), the final networks were trained for 200 epochs. Table 1 displays the IoU (Intersection over Union) scores for the cars and lanes classes, for each model. The scores are grouped into 4 categories: driving in urban environments, driving on freeways, driving in sunny conditions, driving with adverse lighting conditions or during bad weather. The Table  includes also the general scores for all the frames in the dataset. These scores nicely show how the temporal model (Net3) is able to learn a more consistent latent representation, with respect to the topological model (Net2), in all the categories of driving sequences. However, it is also evident how the task of recognizing the cars concept always ends up in better scores compared to the lanes concept. An explanation of why the latter task is more difficult can be the very low ratio of pixels belonging to the class of lanes over the entire image size, and consequently how easily the lane markings get occluded by other elements in the scene. We would like to stress that the purpose of our networks is not mere segmentation of visual input. The segmentation operation is to be considered as a support task, used to enforce the networks to learn a more robust latent space representation, which now is explicitly taking into consideration two of the concepts that are fundamental to the driving tasks.
To visually appreciate the representation learned by the temporal autoencoder (Net3), Fig. 5 shows the images produced by the model for 4 different input images, one for each category of driving sequences mentioned above. We take an input image (showed on the left of the Figure) and make the network produce its corresponding latent representation. The latent vector is passed to the decoders of the network to reconstruct the initial image (output showed on the center of the Figure), in which the class of cars concepts is colored in cyan and the class of lanes concepts in yellow. The images on the right are just displayed as a reference, they show the target images with the colored overlay of the two classes.
To further visualize the performance of our temporal autoencoder (Net3), we present the result of interpolating between different latent spaces. In Fig. 6, each column shows what happens when taking the latent representation of a first frame (first row in the Figure) and linearly interpolate it with the latent representation of a second frame (last row). We generate 5 intermediate latent vectors, which are passed to the decoders of the temporal autoencoder to produce novel frames. The images are a smooth and gradual shift from the first input to the second, and successfully provide new plausible driving scenarios never seen before by the network.

Results of Recurrent Network
Lastly, we present the results of our final network, the recursive model (Net4), trained for 100 epochs on a dataset of latent representations computed by the temporal autoencoder (Net3) over the SYNTHIA frames. Table 2 contains the IoU scores obtained by the model in the different categories of driving sequences used before. As described in §4.3, the network takes as input a sequence of 8 frames and predicts the 4 subsequent frames. The Table  shows the scores for the 4 predicted frames, separated as usual in the cars and lanes classes. It is immediate to note the cars scores are always higher than the lanes scores, just like we saw in Table 1. However, the cars predictions worsen more significantly for the distant frames, where we can see a decay of 16%, while the lanes scores lose only 9%. This result can be explained by the fact that, generally, in a driving sequence the lane markings change in a more smooth and predictable way with respect to the cars, City Freeway Sunny Dark Input Output Target Figure 5. Results of our temporal autoencoder (Net3) in reconstructing an image. In the first row, the input frames belonging to different categories of driving scenarios. In the last row, the same input frames plotted with a colored overlay showing the target cars entities in cyan and the lanes entities in yellow. In the center row, the output of the network.  Table 2. IoU scores of cars and lanes classes obtained by our recurrent network (Net4) when predicting 4 future frames from an input sequence of 8 frames. The scores are computed over all the SYNTHIA dataset, and are organized into 4 different categories of driving conditions. which can for example suddenly modify their trajectory.  read. The model is able to predict an overtake maneuver from the left as well as from the right ("freeway" and "sunny" cases). Another interesting result is the different kind of predictions when facing a crosswalk, in the "city" scenario there is a car moving perpendicularly to the lane of the ego car, so the network correctly predicts to hold still at the cross walk. In the "dark" scenario there are cars driving in the same direction of the ego car, therefore the model predicts to not stop at the crosswalk move forward.
As a further test, we tried to replicate the phenomenon of mental imagery using our recurrent model (Net4), i.e. the network is called iteratively and at each iteration the output is fed back as input of the next iteration. In our specific case we choose to take the 1st of the 4 output vectors and use it as the 8th input vector of the next iteration. Fig. 8 presents the results of 9 iterations of imagery for two different sce-  Table 3. Simple statistics on the latent representations learned by our 3 autoencoder models. For both indicators, the lower the better.
narios, along with the corresponding reference frames (the input images are, again, omitted for practical reasons). Note that, while the imagery process must inevitably start with all input frames taken from the dataset, the results provided in the Figure are obtained from forward iterations, that is when all input vectors are computed by the network as results of previous iterations. In both driving scenarios, it is possible to appreciate how the model is able to predict a quite plausible future from just its own representation of the world.

Latent Representations
We performed additional validations of the latent representations learned by the models with simple statistical evalu-frame A raw frame A segmented frame B segmented frame B raw Figure 6. Examples of interpolation between latent representations learned by our temporal autoencoder (Net3), for two different driving scenarios. The first 2 rows show the first input frame, with and without the colored overlay highlighting the cars and lanes entities. The last 2 rows show the second input frame. The 5 central rows are the result of the linear interpolation between the latent representations of the inputs.
ations of their ability to predict in the future, and their consistency with respect to the temporal dynamics. This sort of evaluation has been useful in the development of the models, in Table 3 we report the results obtained with the 3 final encoder-decoder models. The first indicator ξ evaluates the degree of temporal coherence by the ratio between the difference of two latent vectors that are contiguous in time, and the variance over the entire dataset of latent Z. The evaluation is done independently for each component of the latent vector, and then averaged: where z i is the i-th element of z, υ (i) Z is the i-th element of the variance vector of z over Z, and M is the cardinality of Z. The ideal temporal coherence should be ξ Z = 0.
The second indicator ρ is the mean square of the residual obtained when using two consecutive latent vectors to predict one neuron of a third vector, by linear regression. In order to make computation time acceptable, this index is computed on a subspace Z ten times smaller than Z. By calling ε(A, b) the residual of the least squares approximation of the normal equation Ax = b, ρ can be written as follows: As a final visualization of the performance of our models, we present an inspection of the latent representations learned by the topological and temporal autoencoders (Net2 and Net3), Fig. 9 and 10 respectively. In each Figure Comparing Fig. 9 and 10 it is immediately clear how the temporal autoencoder (Net3) learns a more robust representation, the variation in the neurons encoding the cars and lanes concepts is minimum. Also the variation in the general 96 neurons are very localized, the neurons exhibit a similar overall distribution, and this fits with the fact that the 4 frames have the same surrounding (the trees, the soil on the right). Conversely, the representation learned by topological autoencoder (Net2) does not appear as consistent. The cars and lanes neurons change significantly for each input frame, and even the other 96 visual features do not share any particular pattern in the 4 cases.

Conclusions
This paper presented an algorithm for perception of driving scenarios, our work takes inspiration from some principles on how the brain of the current best drivers -humans -works. We did not have to invent anything from scratch: the deep learning framework already offers tools that -when used in a proper way -can implement the neurocognitive theories we aim at. Specifically, we used autoencoders for implementing the theoretical idea of coding perceptual concepts using the lowest possible dimension, as in neural convergence-divergence zones. Then we followed the theory of predictive brain by encouraging the probabilistic representation learned by the autoencoder to capture information about the future.
Compared to other research on perception of driving scenarios based on autoencoder, our approach is unique in combining the conventional training of the encoder by minimizing the loss on the decoded reconstruction of the input, with the ability of the encoded representation to perform prediction in time. Experimenting with the SYNTHIA Output Target Output Target Figure 8. Examples of mental imagery on our recurrent model (Net4), for two different driving scenarios. Odd columns show the result of iteratively feed the predicted output of the model back as input of the next iteration. Even columns are a reference on the corresponding frames.
dataset, we were able to converge the visual input down to a representation of just 16 neurons for the concept of cars and other 16 neurons for the lanes concept. With this compact representation we achieved good performances in predicting future frames up to 4 time steps ahead. The system described in this paper is certainly not a complete solution to perceptual understanding of driving scenarios, it is a strategy. In several driving contexts there are more concepts than cars and lanes , such as pedestrians and cyclists, not taken into account in our system. An even more crucial task is the projection of the inner representation to more complex spaces than the visual and conceptual ones: the space of actions. In fact, the ongoing research in our group is in decoding the latent representation into the two-dimensional space of affordable longitudinal and lateral controls. computational resources funded by Fondazione CARITRO.

A. Variational Inference
The variational inference framework takes up the issue of approximating the probability distribution p(x) of a high dimensional random variable x ∈ X . This approximation can be performed by a neural network such as that in equation (2). The neural network by itself is deterministic, but its output distribution can be easily computed as follows: where N (x|µ, σ) is the Gaussian function in x, with mean µ and standard deviation σ. Using this last equation it is now possible to express the desired approximation of p(x): It is immediate to recognize that the kind of neural network performing the function f Θ (·) is exactly the decoder part in the autoencoder, corresponding to the divergence zone in the CDZ neurocognitive concept. In the case when X is the domain of images, f Θ (·) comprises a first layer that rearranges the low-dimension variable x in a two dimensional geometry, followed by a stack of deconvolutions, up to the final geometry of the x images. In equation (22) there is clearly no clue on what the distribution p(z) might be, but the idea behind variational autoencoder is to introduce an auxiliary distribution q from which to sample z, and it is made by an additional neural network. Ideally, this network should provide the posterior probability p Θ (z|x) -which is unknown -and should be a network like the kind of equation (1). Its probability distribution is: While the network f Θ (·) behaves as decoder, the network g Φ (·) corresponds to the encoder part in the autoencoder, projecting the high-dimensional variable x into the low dimensional space Z. It continues to play the role of the convergence zone in the CDZ idea. The measure of how well p Θ (x) approximates p(x) for a set of x i ∈ D sampled in a dataset D is given by the log-likelihood: This equation cannot be solved because of the unknown p(z), and here comes the help of the auxiliary probability q Φ (z|x). Each term of the summation in equation (24) can be rewritten as follows: (Θ|x) = log p Θ (x, z)dz where in the last passage we used the expectation operator E[·]. Being the log function concave, we can now apply Jensen's inequality: Since the derivation in the last equation is smaller or at least equal to (Θ|x), it is called the variational lower bound, or evidence lower bound (ELBO). Note that now in (Θ, Φ|x) there is also the dependency from the parameters Φ of the second neural network defined in (23). It is possible to rearrange further (Θ, Φ|x) in order to have p Θ (x|z) instead of p Θ (x, z) in equation (26), moreover, we can now introduce the loss function L(Θ, Φ|x) as the value to be minimized in order to maximize ELBO: where the last step uses the Kullback-Leibler divergence ∆ KL . Still, this formulation seems to be intractable because it contains the term p Θ (z), but there is a simple analytical formulation of the Kullback-Leibler divergence in the Gaussian case (see Appendix B in [28]): where µ i and σ i are the i-th components of the mean and variance of z given by q Φ (z|x).    Total parameters 600.000 Table 7. Parameters describing the architecture of the recurrent network (Net4).