{'fraction_non_alphanumeric': 0.025765951466316873, 'fraction_numerical': 0.010087760392267092, 'mean_word_length': 4.669263456090651, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-modal Experts Network for Autonomous Driving

End-to-end learning from sensory data has shown promising results in autonomous driving. While employing many sensors enhances world perception and should lead to more robust and reliable behavior of autonomous vehicles, it is challenging to train and deploy such network and at least two problems are encountered in the considered setting. The first one is the increase of computational complexity with the number of sensing devices. The other is the phenomena of network overfitting to the simplest and most informative input. We address both challenges with a novel, carefully tailored multi-modal experts network architecture and propose a multi-stage training procedure. The network contains a gating mechanism, which selects the most relevant input at each inference time step using a mixed discrete-continuous policy. We demonstrate the plausibility of the proposed approach on our 1/6 scale truck equipped with three cameras and one LiDAR.


I. INTRODUCTION
The advancements in deep learning [1] enabled artificial neural networks to invade a diverse set of real-life applications that need to rely on systems capable of learning, among which autonomous driving is an important and still largely unsolved task. Using multiple sensors improves perception abilities, leads to a better understanding of the environment, and consequently increases the safety and reliability of deeplearning-based autonomous platforms (some examples in the context of self-driving cars include [2], [3], [4]). Nevertheless, it is highly non-trivial to teach the autonomous vehicle to use the information registered by different sensors potentially capturing various modalities. When there exists an overlap of information as is the case for camera and LiDAR registrations that cover the same or partially overlapping field of view, one needs to ensure the network is evenly trained to use any of the available relevant inputs. Otherwise, the failure of the sensor that the network was preferring at training leads to significant performance degradation. This could be avoided if instead of ignoring the information provided by other relevant sensors, the network would adapt to them as well. Another obstacle when using many sensors is the explosion of computational complexity. Addition of an extra sensor requires the expansion of the network to accommodate an extra feature extractor and increases computational cost. This largely increases the inference time step and eventually paralyzes vehicle's real-time operation. The objective of this work is to address the above challenges.
We propose a network that contains a feature extractor per each sensor and a gating mechanism that activates only a subset of feature extractors at each inference time step. The key component of the proposed architecture is the gating mechanism, which selects the most informative input in real time under strict computational constraints. It therefore efficiently switches the attention of the network among inputs (in other words experts). In our setting, we have three camera inputs that are mutually exclusive, i.e. they have nonoverlapping field of views (each of them covers 60 degrees), and one LiDAR sensor that covers 180 degrees 1 and therefore has an overlapping field of view with the cameras. We consider two schemes of input selection. The first one is the discrete one and occurs when selecting the proper sensor. The second one is the continuous one and occurs when selecting a relevant part of the information delivered by a single sensor. This is done for LiDAR, which has a very wide field of view. The proposed concept naturally provides a defensive mechanism against sensor failures as those will be considered as irrelevant ones and will not contribute to the final prediction. Moreover, wasteful computations correlated with extracting features from sensory inputs irrelevant or redundant for driving are avoided.
We also propose a multi-stage training procedure and show that it is superior compared to the other end-to-end learning techniques. We show that the end-to-end scheme does not result in the network that has the ability to select most relevant sensors. Note that the small gating network has its own feature extractors which should be correlated with the main feature extractors. The reason for the failure of the end-to-end approach is that it does not learn this correlation.
We empirically verify our approach on our autonomous platform equipped with three cameras and LiDAR and provide evidence that our network correctly selects the proper sensor and works well in the discrete and continuous selection regimes, robustly predicts the vehicle's steering command, and requires significantly fewer computations than common baselines. This paper is organized as follows: Section II provides a literature review, Section III describes our approach, Section IV demonstrates experimental results, and Section V concludes the paper.
II. RELATED WORK This paper considers an end-to-end imitation learning framework for steering autonomous driving platforms that take raw sensor inputs and predict the steering command. We consider deep learning framework since it allows for the automatic extraction of data features that are furthermore useful for a particular learning task and potentially more scalable to real-life systems. Alternative non-deep learning techniques for multi-sensor data fusion for autonomous driving were investigated in [5], [6], [7], [8]. The functional mapping between inputs and outputs in the end-to-end setting was realized in past works using fully-connected networks trained on synthetic data [9] or convolutional neural networks (CNNs) trained on real recorded data without [10], [11], [12] and with [3], [11] augmentation. All of the above mentioned works rely only on a single camera sensor at inference. The conditional imitation learning approach [13] relies on a single camera and uses camera images and navigation commands to steer the vehicle through road intersections in desired directions. This work was further extended [14] to use both camera images and maps to generate a probability distribution over possible steering commands. Finally, [15] proposes a reconfigurable network that uses images captured from three front-facing cameras as inputs to navigate the autonomous driving car in an indoor environment. There are only a few works proposing end-to-end learning frameworks accommodating multiple sensing modalities such as LiDARs and cameras [16], [17], [18], [19]. Our work concerns precisely such setting and engages the gating mechanism for relevant sensor selection unlike existing approaches.
The gating mechanism was explored in the literature in various settings and applications. The first system incorporating this mechanism [20] was the modular version of a multilayer supervised network, where many separate networks (experts) were operating on the same input but they were handling different learning sub-tasks. The gating mechanism has often been used as an enabling technique for scaling algorithms to large data sets and has been flexible enough to adapt to different learning systems, including those with experts realized as SVMs [21] or neural networks [22], [23], [24], [25] and those where experts were hierarchicallystructured [26]. The works on conditional computation [22], [23], [24] explore the notion that irrelevant or redundant experts need not be activated at all when forming a prediction. Sparsely-gated mixture-of-experts [25] realizes an extreme version of this concept, where a large number of experts is used, but only a very small subset is activated at each inference time step. None of the gating schemes discussed above was applied to autonomous driving, but they were explored in the context of more traditional AI problems such as language modeling, machine translation, or image classification. Our work falls into the category of conditional computations schemes, but as opposed to existing methods experts do not share the same input. Finally, gating mechanism was also used for sensor fusion [27], [17] in autonomous driving, where all experts are active. These approaches therefore are not concerned with constraints on the computational budget since they allow wasteful feature extractions from irrelevant or redundant sensors. Our work fundamentally differs from these works as in our case such experts are identified at each inference time step and are deactivated, which leads to computational savings.

III. MULTI-MODAL EXPERTS NETWORK
A. Network architecture In the Camera Modality, there are three inputs, each from a different camera, followed by the corresponding feature extractors (Camera Experts). In the LiDAR Modality, the single input from the LiDAR is used. All the camera inputs and the entire LiDAR input are passed to the Main Gating Network that decides which input contains the most relevant information for driving at the moment and activates the corresponding expert. If LiDAR is selected, its input is first processed by the LiDAR Gating Network that decides which part of the input will be cropped and used for further processing. The cropped LiDAR input is passed to the LiDAR feature extractor (expert). Finally, the output of the LiDAR Gating Network is concatenated with the output of the LiDAR expert. The output of the Main Gating Network is concatenated to the output of the active expert and passed to the final Fully-Connected Network (FC) that forms the prediction of the steering command.
The proposed multi-modal experts network is illustrated in Fig. 1. The network is designed to handle inputs coming from three cameras and one LiDAR. Thus the network has four inputs in total coming from two sensory modalities. We will refer to those inputs as x = [x 1 , x 2 , x 3 , x 4 ], where x 1 , x 2 , x 3 are the camera inputs and x 4 is the LiDAR input. The final predictionŷ of the multi-modal experts network can be therefore expressed in the following way: , the resulting feature vector is formed by concatenating the feature vector coming out from the chosen expert and the output of the Main Gating Network. F is the action of the final fully-connected network and transforms the features into the final prediction label. This construction ensures that the resulting feature vector encodes the information extracted from the selected input as well as the identity of the selected sensor, and in case this sensor is LiDAR also the information about the part of the input that was used.

B. Training procedure
We propose a carefully tailored procedure for training the multi-modal experts network. The training procedure is designed to enable efficient and robust learning of the behavior of the gating mechanism from the training data, which do not carry any information about the identity of the most relevant sensor at any given moment in time. The network training procedure that we propose consists of three main steps. In the first step, we train the network equipped with the gating mechanism to predict the steering command based on the LiDAR input. Thus the network effectively learns which part of the LiDAR input is relevant (continuous sensor selection). In the second step, we train another network, also equipped with the gating mechanism, to predict the steering command from either one of the cameras or LiDAR input. Thus the network effectively learns to choose between sensors and modalities (discrete sensor selection). In the third step, we incorporate the LiDAR network trained in the first step into the multi-modal network trained in the second step and finetune everything to simultaneously choose the proper sensor (learned in step 2) and the relevant part of the LiDAR input (learned in step 1), if its modality was chosen.
Moreover, for the purpose of training the gating networks (either the LiDAR Gating Network from the first step or the Main Gating Network from the second step) realizing the gating mechanism, we consider two sub-steps: • We train the entire network (all feature extractors and the gating mechanism that uses those extractors) to predict the steering command. The by-products of this training are the labels for the gating network that indicate relevant sensors for driving. Since the gating network in this step uses the same feature extractors as the main network it is able to learn sensor relevance without explicit labels. We train the network with the architecture given in Fig. 2. Its input is formed by splitting the LiDAR input into five partially overlapping segments (they overlap one-half), each covering 60 degree field of view. Each segment is processed by the corresponding expert. The feature vectors obtained from the experts are processed by the fully-connected network that realizes a function of the gating mechanism. Therefore, the experts and the mentioned fully-connected network form a gating network. Finally, the feature vectors are scaled by the outputs of the gating network and passed to another fully-connected network that predicts the steering command. We train this entire model in an end-toend fashion. We designed a cost function comprising three carefully selected terms to ensure desired performance of the gating behavior. The first term is the prediction loss measuring the discrepancy between the true and predicted steering command and to ensure the final output of the network can successfully steer the robot.
L prediction = y −ŷ 2 2 . To ensure that the output of the gating network is sparse so that the most informative input can easily stand out, we introduced the second term that we refer to as sparsity loss: where g i s are the outputs of the gating network. The outputs of the gating network are obtained after a softmax layer, is the input to the softmax layer), thus ∑ i g i = 1. In this stage we allow more than one g i to be non-zero. The final term of the loss is the negative entropy computed with respect to the distribution of the gating network outputs. It is given as is the i th output of j th gating network observation in one batch and B is the training batch size. Minimizing this term prevents the output of the gating network from collapsing and favoring a small and fixed subset of experts over the entire data batch, while zeroing out the rest.
The objective function has therefore the following form: where α and β are the hyper-parameters controlling the importance of sparsity and negative entropy terms. We aim at minimizing this objective. In the result, we obtain the trained gating network that we then use for generating labels for the desired gating mechanism.

Training the LiDAR Gating Network
We train a separate LiDAR Gating Network that using knowledge distillation [28]. This one predicts which part of the LiDAR input is the most relevant for driving. We train the LiDAR Gating Network using the gating mechanism obtained in the previous training step as the teacher network. Specifically, the labels g L for training the new gating mechanism are calculated as g L = ∑ i g L i r i , where g L i are outputs of the reference gating mechanism obtained in the first step of the training procedure and r i = [−1.0, −0.5, 0.0, 0.5, 1.0] are the values of the g L corresponding to the segments of the LiDAR image described before.

Fine-tuning
In the third sub-step of the training procedure, we train the network with the architecture depicted in Fig. 3. Here we are using the LiDAR Gating Network obtained in the previous step of training and we keep its weights fixed. The network uses LiDAR input which first is processed by the LiDAR Gating Network. Based on the LiDAR Gating Network output, part of the input is cropped. In particular, we crop a segment covering 60 degree field of view. The segment range is calculated as c l = 60 o g L − 30 o ; c r = 60 o g L + 30 o , where c l and c r are the left and right edges of the cropped segment respectively and g L is the output of the LiDAR Gating Network that is in the range [−1, 1]. Next, the cropped segment is processed by the feature extractor (expert). The output of the LiDAR Gating Network and the obtained feature vector are concatenated together and fed to the fullyconnected layer, which produces the steering command. At the end of this step, we obtain a functional network with the single LiDAR input for the autonomous driving and we refer to this network as "LiDAR with gating" network.
2) Second Step: Discrete Sensor Selection 2.1 Obtaining labels for the gating mechanism First, we train the network with the architecture depicted in Fig. 4. The architecture uses three inputs from camera modality and one input from LiDAR modality and is capable of handling sensor fusion. The feature vectors obtained from three camera experts and one LiDAR full expert (feature extractor of the entire LiDAR input) are also used by the gating network as feature extractors, thus it can be seen that the additional fully connected layers realize the functionality that we call the gating mechanism (selecting relevant inputs). Finally, the feature vectors are scaled by the outputs of the gating network and passed to another fully-connected network that predicts the steering command. We train this entire model in an end-to-end fashion using the same objective function as the one introduced in the 1.1 sub-step.

Training the Main Gating Network
As a result of the 2.1 step of training, we obtain the trained gating network that serves as a reference delivering labels for training the Main Gating Network from Fig. 1. The latter network realizes the gating mechanism showing which input (one of three camera inputs or LiDAR input) is most relevant for driving. Note that we used the same training strategy in the 1.2 step of training. To be more specific, the labels g M i for training the Main Gating Network are calculated as j = arg max g M i , where g M i are the outputs of the reference gating mechanism obtained in the previous step of the training procedure. In this case, the Main Gating Network is trained to select one, the most relevant input for driving.

3) Third Step: Training the Multi-Modal Experts Network
In the final step of the training procedure, we train our proposed multi-modal experts network depicted in Fig. 1. In LiDAR modality, we use part of the network architecture in the first step but without final fully-connected layer to process LiDAR information. In this way, if the LiDAR expert is selected, the computation can be further optimized. Here we keep the weights of the LiDAR Gating Network obtained in the first step and the Main Gating Network obtained in the second step fixed. Thus, in this step, we train all the experts and the final fully-connected layer that produces the steering command. In the result, we obtain the final network that identifies the most relevant input for predicting the steering command and activates only the corresponding expert at each inference time step. The block diagram of the autonomous platform that we used for data collection and system testing is shown in Fig. 5. The platform is based on the Traxxas X-Maxx remote control truck and equipped with Drive PX2 that performs computations. We incorporated the following sensors on the platform:

A. Hardware Overview
• three SEKONIX AR0231 GMSL cameras that are facing the front of the platform and covering nonoverlapping views. Each camera has 60 degrees horizontal field of view. The center camera is oriented straight. The side cameras are mounted at angles to allow capturing front side views. • single Velodyne VLP-16 LiDAR with 16 lasers that cover a 30 degree vertical field of view and 360 degree horizontal field of view.
Controlling the actuators of the autonomous platform was done using Micro Maestro 6-Channel USB Servo Controller. When collecting the data, the wireless game-pad was used to drive the platform. The platform was controlled by an experienced human driver to ensure it drives in the center of the path without hitting the walls of the corridors or obstacles. The captured steering commands were used as ground truth for training and evaluation. The steering commands together with the corresponding cameras and LiDAR inputs captured by the sensors were saved on the Drive PX2 local storage. At testing, the platform runs in real time and is steered with the commands predicted by the deep learning gating-based system. The speed is dictated by the operator of the platform.

B. Data Pre-processing
The SEKONIX AR0231 GMSL cameras have resolution 1920 × 1208. We scale down the images captured by the cameras to the size 192 × 120. We use three channels (RGB) for each camera.
The LiDAR captures data in the form of a point cloud, which we convert into a depth map. We utilize only the frontfacing 180 degrees field of view of the LiDAR input as the car is only required to drive forward. The resulting depth map has size 450 × 16. We use two channels, one for distance and one for reflectivity. The exemplary images from cameras and LiDAR are shown in Fig. 1  examples uniformly at random from each category to balance the data set. We also collected the test data containing 2, 384 examples. The train and test data sets were collected in different parts of the same building. All reported results in the next sections were obtained on the test data set.

C. Training Details and Evaluation
We use ADAM optimizer in all steps of the training procedure. Hyperparameters are tuned by grid search to ensure best performance. In the 1.1 step, we use the learning rate equal to 0.001 and the following hyper-parameters for the objective function: α = 0.001, β = 0.0016. The performance of the gating mechanism obtained in the 1.1 step of the training procedure is reported in Fig. 6(a)(top). In the 1.2 step, we train the LiDAR Gating Network using knowledge distillation. The loss terms consist of distillation loss and student loss. The distillation loss is measured as the KL divergence loss between soft labels of the teacher network and soft predictions of the student model. The student loss is defined as the mean square error (MSE) between the predicted labels and the labels produced by the reference gating mechanism. In this step, the learning rate was set to 0.001, the distillation temperature is set to 2 and the weight on the distillation loss is 0.9. In the 1.3 step, the learning rate was set to 0.001 and the test performance of the network is shown in Fig. 6(a)(bottom). The network clearly can learn to predict the correct steering command and properly selects most relevant part of the LiDAR input at given time instant.
In the 2.1 step of the training process, we use the learning rate of 0.0001. The hyper-parameters of the objective function are α = 0.002 and β = 0 (this is because there is no shared information between any two different inputs). In this step, we assume that at most one sensor will encounter a sensor failure. Thus, we separately train different instances of the same network with architecture depicted in Fig. 4, where Fig. 6. Top: The output of the gating network obtained from the 1.1 step of the training procedure indicating the relevance of each expert for predicting the steering command. The corresponding network architecture is shown in Fig. 2. Bottom: The comparison between the actual and predicted steering command. The network (shown in Fig. 3) obtained from the 1.3 step of the training procedure is used for prediction. In red we show the output of the LiDAR Gating Network. Only the first 250 test frames are shown here due to the space limitation.  Fig. 4. Bottom: The comparison between the actual and predicted steering commands for the 5 th network. Results of the other networks are similar. Fig. 8. The test performance of the final network (shown in Fig. 1) when the inputs are (1)- (5). The curves show the comparisons between the actual steering command and predicted steering command. The color-shaded areas indicate the selection of the Main Gating Network. for each instance we disable different sensors. We consider the following cases: (1) LiDAR is enabled and at most one of the camera inputs is disabled (2) LiDAR is disabled and all the camera inputs are enabled (3) Left part of LiDAR is disabled and all the camera inputs are enabled (4) Center part of LiDAR is disabled and all the camera inputs are enabled (5) Right part of LiDAR is disabled and all the camera inputs are enabled. The test performance of the resulting five networks is depicted in Fig. 8. Clearly, the gating network can successfully select the most relevant sensor. For example, when LiDAR sensor is enabled the gating network always selects the LiDAR expert as the best sensor for driving. Otherwise, when the LiDAR sensor is disabled, the gating network chooses the most relevant camera. This is intuitive as LiDAR input is more informative compared with RGB camera image and the information from the side sensors will be more important when the car makes turns.
In the 2.2 step of the training procedure, the distillation  see Table II 204.69M *NetGated(NG) see Table II 204.90M *Sensor Dropout(SD) see Table II 204.69M

Multi-step training
LiDAR with gating 0.023 28.15M *Ours see Table II 35.71M chosen sensor: LiDAR *Ours see Table II 58.15M chosen sensor: camera loss is the same and the student loss is the cross-entropy loss between the predicted labels and the labels produced by the reference gating mechanism. In this step we use the ADAM optimizer with the learning rate set to 0.001. The distillation temperature is set to 4 and the weight on the distillation loss is 0.9. We train the Main Gating Network using all the five gating networks obtained in the previous step as reference. The performance of the obtained Main Gating Network is reported in Table I. The network achieves high performance accuracies of choosing the correct sensor.
In the third step of the training procedure, we train the final multi-modal experts network. We use the pre-trained experts from previous steps and set the learning rate to be 0.0001. In particular, we use the pre-trained LiDAR expert obtained from the first step and pre-trained camera experts obtained in the second step in the case where LiDAR was disabled. Finally, we use all five settings of enabled/disabled sensors, to fine-tune the entire network in an end-to-end fashion. In particular, we lock the gating networks (will not update the parameters) and train the whole network. The test results of the obtained final network are shown in Fig. 8 and Table II, where the test performance is measured as the mean squared error(MSE) between the predicted and the reference steering command. The obtained network correctly chooses the useful sensor for driving and accurately predicts the steering command. Finally, we compare the test performance of our model with several different approaches. The first three methods listed in Table III describe models that use inputs from single modality. They are constructed by one or more feature extractors followed by a fully-connected layer to make the final prediction. The baseline model is obtained by concatenating the features from all of the experts without any gating mechanism. It is trained end-to-end. NetGated and Sensor Dropout were introduced in [17] and [19] respectively. We followed the setups described in their original papers to train the networks. Also note that our approach is orthogonal to the existing network compression methods [29], [30], [31], [32] since compression schemes are applicable to our approach and can lead to further computational reductions. To make fair comparisons, each network uses the same feature extractor architectures and the gating networks are designed to require significantly less computations than feature extractors, they are all described in the Appendix. We compare MSE between the predicted and actual steering command and the amount of computations, i.e. the number of floating-point operations (FLOPs) needed to perform a single inference. For the multisensory models, their performances are measured in five different scenarios when one of the sensors is disabled. The results are captured in Table III. The proposed network achieves comparable performance to the baseline model and outperforms the majority of the competitive methods in terms of test loss except for NetGated. However, both the baseline and NetGated require significantly more computations and they do not have the ability to interpret the importance of each sensor at any given moment. Meanwhile, our method uses almost the same amount of computations as the single modality networks. Therefore multi-modal experts network is superior over other sensor fusion techniques as it can make accurate predictions and handle sensor failures better while at the same time is computationally much more efficient. We tested our proposed network on the autonomous platform and used the predicted steering angles to steer the vehicle in an indoor environment. The snapshots of the test video are shown in Fig. 9.
D. Ablation Analysis 1) Can we use end-to-end training scheme to train the Multi-modal Experts Network?: Note that the small gating network has its own feature extractors which should be correlated with the main feature extractors. Training our system with the end-to-end approach is not practical since it fails to learn these correlations, i.e. there is no mechanism to enforce correlation among mentioned feature extractors. Thus we propose a multi-step training method to first train the gating network and then fine-tune the experts based on the behaviors of the gating network. To further support this statement, we performed an experiment where we compare our performance with an end-to-end trained scheme ( Fig. 10(a)) and demonstrate that simple end-to-end training approach does not work well for our setting.
2) The effectiveness of L sparsity and L nentropy : To show the effectiveness of the two losses, we performed controlled experiments. We set both α and β to zero when training the network with the gating mechanism in sub-steps 1.1 and 2.1. We show the test results obtained from the step 1.1 in Fig. 6(b). When comparing this figure with Fig. 6(a), we can see that without L sparsity , the network is less confident when selecting experts and without L nentropy the selection is limited to a group of the following experts: left, center, and rightmost expert (see Fig. 2 for explanation of the experts). This limitation therefore causes the LiDAR Gating Network to ignore the leftmost part of the LiDAR input. Furthermore, training without the sparsity loss (shown in Fig. 10(b)) demonstrate that the outputs of the gating network for each expert are closer to each other than when the loss is introduced (see Fig. 7). In other words, the Main Gating Network has less confidence in making the selection. The intuition is that without the sparsity loss, the experts tend to be cooperative instead of competitive with each other when predicting the steering angles.

V. CONCLUSION
This paper focuses on the autonomous driving problem with multiple sensing modalities, such as cameras and Li-DAR. We consider the problem of efficient usage of multimodal sensory resources with the goal of building a network that activates the processing of the data from only the relevant sensors. We show how to combine the discrete and continuous selection of relevant information. Our approach shows superiority over existing common baselines: it handles sensor failures, has high predictive power, and is efficient (runs in real time). In addition, by visualizing the output of the gating network, it becomes easy to see which part of the input is the most important for the driving task at any given moment. LiDAR full expert in LiDAR only network: The first four conv layers are followed by a 2x2 maxpooling with a stride of 2. An 1x2 maxpooling with a stride of (1,2) follows the conv5 layer.