{'fraction_non_alphanumeric': 0.028524513253577292, 'fraction_numerical': 0.009570724841660803, 'mean_word_length': 4.914539400665927, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adversarial Learning-Based On-Line Anomaly Monitoring for Assured Autonomy

The paper proposes an on-line monitoring framework for continuous real-time safety/security in learning-based control systems (specifically application to a unmanned ground vehicle). We monitor validity of mappings from sensor inputs to actuator commands, controller-focused anomaly detection (CFAM), and from actuator commands to sensor inputs, system-focused anomaly detection (SFAM). CFAM is an image conditioned energy based generative adversarial network (EBGAN) in which the energy based discriminator distinguishes between proper and anomalous actuator commands. SFAM is based on an action condition video prediction framework to detect anomalies between predicted and observed temporal evolution of sensor data. We demonstrate the effectiveness of the approach on our autonomous ground vehicle for indoor environments and on Udacity dataset for outdoor environments.


I. INTRODUCTION
The development of algorithms for autonomous navigation of robotic systems in complex unstructured environments is attracting significant research interest due to emergence of more accurate and cheaper sensors, faster embedded computing devices (e.g., GPUs), and vast amounts of data. Particularly, learning-based algorithms are widely used due to the large sets of available data to train either the perception subsystems or the end-to-end control policy for robust autonomous operation of the robotic system [1], [2].
These learning-based systems cannot be trained in all possible environments and environmental conditions. Such systems are also vulnerable to adversarial attacks [3], [4]. Recently, approaches have been proposed to address this challenge based on generative modeling [5], [6] and gradient based search [7]. Formal verification [8] and Bayesian optimization [9] are promising but are computationally complex.
For real-time detection of anomalies/malfunctions in a cyber-physical system [11], [12], we propose the framework in Figure 1 which combines complementary anomaly monitoring methods [10], [13], namely controller-focused anomaly monitor (CFAM) and system-focused anomaly monitor (SFAM). CFAM uses an image-conditioned EBGAN to validate the mapping from the sensor data to the actuator command. SFAM uses an action conditioned video prediction system to validate the mapping from the system action to the sensor data. The proposed learning-based anomaly detection framework continuously validates and verifies autonomous operation during run-time. Novel aspects of this study are: • A learning-based on-line framework to monitor the controller and the system behavior. • A conditional EBGAN architecture to detect anomalies in controller outputs. • An action conditioned video prediction framework to detect anomalies in system behavior. • A methodology to train a robust video prediction architecture to detect anomalies. This paper is organized as follows. The background and the problem are discussed in Section II. The architecture, training methodology, and anomaly detection for CFAM and SFAM are described in Section III. Section IV reports results on indoor (collected from our experimental unmanned ground vehicle) and outdoor (Udacity [14]) datasets. Section V concludes the paper.

II. PROBLEM FORMULATION
The presented anomaly detection applies to any closedloop system such as an autonomous vehicle. In this paper, we consider an unmanned ground vehicle (UGV) instrumented with camera(s) and LIDAR and controlled by an end-to-end learning system. The input command to the vehicle is u t at time step t with the corresponding sensor output, x t .
The CFAM operates on the stream of sensor data and actuator commands to monitor validity of the mapping x → u and detects anomalies related to the controller. These anomalies can be due to distributional shift (e.g., when the testing and training environments are different) or  due to sensor malfunctions/failures/attacks. The approach for CFAM is based on learning sensor data (image) conditioned energy based generative adversarial network (CEBGAN) in Figure 2 which outputs low energy for valid actuator (steering) commands and high energy for anomalous commands.
The SFAM monitors the validity of the (dynamic) mapping u → x from system action to sensor data (over a time horizon into future) and detects anomalies generated due to malfunctions in the system. Malfunctions/anomalies can occur due to environmental perturbation (e.g., slippery road) or partial/full subsystem failure (e.g., electrical, mechanical) which causes the system to not act according to the given actuator command. The framework addresses this problem by learning the dynamics of the overall system based on action conditioned video prediction as shown in Figure 4. In the implementation, the framework takes as input x t−3:t and actuator commands u t−2:t+1 to predict x t+1 , which is used to validate the system action to system data mapping. The video prediction and anomaly detection architectures for the proposed framework are shown in Figures 3 -5.

III. PROPOSED FRAMEWORK
A. Image conditioned EBGAN 1) Model architecture: Motivated by the success of GANs [15] in a variety of applications, we propose to use discriminator in GAN for CFAM. We combine conditional-GAN [16] with EBGAN [17], to generate CEBGAN and explore its potential to detect controller generated anomalies.
Inputs to the generator are noise vector z sampled uniformly from (0, 1) and condition image x. Discriminator inputs are actuator command u and condition image x. The discriminator learns to map inputs u and x to a scalar value (Energy), while the generator learns to predict steering command from the condition image.The discriminator output is computed as mean square error between input steering command u and the condition x driven steering command prediction. For CFAM, the discriminator is the key component and the generator is merely trained to produce contrastive samples. We train them both simultaneously and use only the discriminator in CFAM.
In CEBGAN, the generator and the discriminator are conditioned on camera input images. The conditional image x feature vector is computed by passing the image through a series of convolutional layers. In our experiments with indoor (UGV) and outdoor (Udacity) datasets, the images are all first rescaled to size 3x128x128 and the best performer discriminator used 6 convolution layers for image feature extraction in both generator and discriminator. The number of kernels in the first, second, third, fourth, fifth and sixth convolution layers are 8, 16, 32, 64, 128, 256 respectively and kernels size is fixed to 4x4 and applied with stride 2. Before each convolution layer, spatial batch normalization is performed and LeakyReLU activations with 0.2 negative slope are used for all layers except for the output, which uses Tanh.The obtained feature map is then reshaped as a vector.
In generator, the noise input z is mapped to a fully connected hidden layer of size same as that of the feature vector computed on the condition image to allow their summation. Then, in both, the summation of these vectors is mapped to another fully connected hidden layer before it is reduced to a single output neuron as shown in Figure 2.
2) Training methodology: In CEBGAN, the discriminator is trained with an objective function in order to shape the energy function to attribute low energies to correct steering commands and high energies to the generated (or anomalous) commands. We use different loss functions to train the discriminator and the generator. Given a positive margin m, a condition image x, true steering command u, and generated steering command G(z), the discriminator loss L D and generator loss L G are defined as: where D(u, x) is discriminator output and [.] + = max(0, .). The generator and discriminator parameters are optimized using Adam optimizer.
3) Anomaly detection framework: The real-time CFAM module consists of a discriminator trained as described in Section III-A.2. Given a condition image x t , energies (discriminator outputs) are computed for N linearly spaced equi-distant points in the steering commands values range. The validity of the incoming sensor data is determined by computing the steering command deviation as the magnitude of the difference of the controller output (steering command) and the corresponding minimum energy steering command value. Anomalies are associated with data points with deviation greater than the threshold.
B. Adversarially learned action conditioned video prediction 1) Model architecture: Learning-based video prediction has been used in various control applications [18]- [21]. The SFAM in Figure 3 consists of an action conditioned video prediction architecture shown in Figure 4, dissimilarity computation module, and validation module. The video prediction architecture is based on the four-layer Prednet [22]. Each layer has four sub-modules: (1) convolution, (2) prediction representation, (3) recurrent representation, and (4) error representation.
All convolutions have 3x3 kernels and max-pooling of stride 2x2 with kernel size 2x2. The number of output convolution channels per layer for prediction and target representation sub-modules are 3, 32, 48, and 64, respectively.
The inputs to the network are a sequence of images x t and the steering commands u t+1 at the next time step as shown in Figure 5. Each layer l at time step t has predictionÂ l t and target representations A l t , generated by the prediction and convolution sub-modules. At the first layer, the target representation A l t is the actual frame. In layers 2, 3, 4 the target representation is generated by the convolution submodule with the error representation from the previous layer E l−1 t . The convolution sub-module has a convolution layer with ReLU nonlinearity followed by max-pooling layer.
The prediction representationÂ l t is generated from the recurrent representation of the current layer R l t as input to the prediction sub-module. The prediction module consists of a convolution layer with ReLU non-linearity except for at the first layer where it is followed by a SatLU non-linearity to saturate the values to the maximum pixel intensity values.
The recurrent representation layer that generates R l t is a convolutional LSTM whose inputs are the error representation from the previous time step of the same layer E l t−1 combined with the upsampled output of the recurrent representation of the higher layer R l+1 t . The hidden state is the recurrent representation from prior time step R l t−1 . The error representation E l t is generated by combining the feature maps of the difference between the predictionÂ l t and the target A l t and the target A l t and the predictionÂ l t followed by a ReLU nonlinearity. In the first time step, all error representations are reset to zero. The steering command is introduced at the next time step u t+1 as the input to generate action conditioned frame prediction. It is concatenated with the error representation of the fourth layer by tiling the steering command to the same dimension and used as input to the fourth recurrent representation layer.
2) Training methodology: The video prediction architecture takes as input four RGB images x t−3:t (scaled between 0 and 1) from a monocular camera of resolution 120x160 and steering commands u t−2:t+1 to predict the framex t+1 . The training has two stages: error representation minimization and adversarial optimization. The weights are updated using an Adam optimizer with a base learning rate of 0.001.
The loss function for the error representation minimization consists of the average of all error representation E l t at each time step t and at each layer l, the negative of structural similarity (SSIM) (kernel size=5) between the predicted framex t+1 and the actual frame x t+1 and the SSIM between the predicted framex t+1 and the previous frame x t . The weight for the average of all error representation is 0.1. The weight for structural similarity between actual and predicted frame is 1. The weight for structural similarity between previous and predicted frame is 0.5.
Adversarial optimization is introduced to curtail blurry prediction images [23]. In this stage, the video prediction architecture is the generator and is combined with a spectrally normalized discriminator [24] with N+1 labels where one label is the fake/real label and the other labels are N steering commands as in Figure 5. We use 15 equally spaced labels for steering command actions for both indoor and Udacity dataset with range from −0.24 to 0.28 and from −0.52 to 0.56 respectively. The discriminator consists of nine spectrally normalized convolution layers with LeakyReLU non-linearity with negative slope of 0.1 in between all of them and a kernel size alternating between 3x3 and 4x4 with stride of 1x1 and 2x2, respectively and a spectrally normalized fully connected layer, which outputs a vector of size 16. The GAN is trained based on the technique proposed in [25] with an additional regularization term as described in [6] to make it robust to out-of-distribution samples.
The regularization term added while updating the generator parameters is KL(U (s)||P θ (s|G(z i )) where U (s) is the uniform distribution of the steering command labels s, G is the generator, z i is the video prediction network input, and θ are the parameters of the generator. This term forces the generator to output out-of-distribution samples which are in the low data density region. The generator is updated by minimizing the weighted sum of cross entropy loss (fake/real, action likelihood estimate) and the KL divergence term introduced above with weights randomly generated between 0-1.

3) Anomaly detection framework:
The video prediction framework in Figure 4 generates video predictionŝ x t+1 |u t+1 = s i where i ∈ [1, N ]. These video predictions are input to the dissimilarity computation module as shown in Figure 3 which calculates the dissimilarity ((1-SSIM)/2) of the prediction with the actual future frame. These dissimilarities are input to the validation module which selects the action corresponding to the least dissimilar predicted frame and compares it with the actual future action to validate the mapping from the actuator command (system action) to sensor data. A windowing strategy using multiple prediction frames validates system actions → sensor data mapping.

A. Datasets
We empirically validated CFAM and SFAM on indoor and outdoor datasets. The indoor dataset is collected by a human controller driving a UGV in a corridor with varying lighting conditions and changing obstacle placements [2], [26]. The Udacity dataset [14] has images recorded while driving on highways and residential roads (with and without lane markings) in clear weather during daytime and includes driver's activities such as staying in and switching lanes.

B. Simulated Scenarios
Datasets of RGB camera images and the corresponding steering commands for various anomalous and nonanomalous scenarios were created. To simulate anomalous scenarios due to malfunctions/anomalies in the controller or the external system, the controller output was overridden by a human driver to create anomalous driving conditions. To test CFAM, the human-created anomalous inputs were provided to the CFAM as controller-generated commands.To test SFAM, the actual controller-generated, correct steering commands and the sensor data that was recorded when the robotic vehicle was actually given the overridden steering commands were provided to the SFAM.
Two complementary anomalous datasets were simulated by the human driver for the UGV in the indoor environment resulting in late-right and early-left crash into a wall. In the late-right turn scenario, the driver provided steering commands to the vehicle to go straight when the vehicle has to take a right turn. In the early-left crash case, the driver provided steering commands to move towards left when the vehicle has to go straight. Fig. 6. Real time anomaly detection when UGV takes a late right turn. Left: controller anomaly detected by CFAM; Right: system anomaly detected by SFAM. In each of these plots, the difference between the controller output and the steering command corresponding to minimum dissimilarity score is plotted as a function of time. By thresholding on the magnitude of this difference, instantaneous anomalies are detected (red dots) and an overall anomaly detection trigger signal is generated.
CFAM and SFAM performance for non-anomalous scenarios are shown in Figures 7 and 10, respectively and for anomalous scenarios in Figures 8 and 11, respectively. The predictions generated by the video prediction architecture in SFAM with varying lighting are shown in Figure 9. By using a threshold on the deviation between the actual steering command and the steering command corresponding to minimum dissimilarity score, a boolean value of anomalous/nonanomalous can be generated as shown in Figure 6.
V. CONCLUSIONS Our proposed framework is successfully able to continuously validate the mappings from sensor data to actuator command and actuator command to (future) sensor data to detect anomalies/malfunctions in the overall system. Fig. 7. Examples demonstrating the effectiveness of CFAM for non-anomalous cases in indoor and outdoor environments. In top row, the set of images on the left are from an indoor environment dataset (collected in-house) and the set on the right are from an outdoor environment (Udacity dataset). Each plot in the second row shows the discriminator's outputs plotted as a function of steering command input with corresponding image in top row as condition. It can be observed in the plots that the energy curves have their global minimum close to the ground truth steering command (highlighted as red dots) as expected. Note: These images were taken from the test set and are not used for training the respective discriminators. All three cases of turning left, right, and moving straight are presented. Fig. 8. Examples demonstrating the effectiveness of CFAM for anomalous cases in indoor environment. The set of images on the left are from an instance when the unmanned ground vehicle (UGV) makes a late right turn and on the right are from an instance when the UGV makes an early left turn. Each plot in second row shows the discriminator's outputs plotted as function of steering command input with corresponding image in top row as condition. It can be observed in the plots that third and fourth images in both sets are anomalous cases and the steering command deviations (distance between red dot and the energy curve global minimum) are high, thus signaling anomalies. Fig. 9. Temporal prediction of camera images in varying lighting for indoor and outdoor environments. The set of images in the left are from an indoor environment dataset (collected in-house) and the set on the right are from an outdoor environment (Udacity dataset). In each set, the four frames in the top row are the input image sequence given to the temporal prediction network. The first two frames in the bottom row of each set are the two ground-truth future frames and the next two frames are the predicted future frames. Fig. 10. Examples demonstrating the effectiveness of the SFAM for non-anomalous cases in indoor and outdoor environments. The set of images in the left are from an indoor environment dataset (collected in-house) and the set on the right are from an outdoor environment (Udacity dataset). In each set, the images on the top are the ground truth of the frame predictions shown in the middle row of each set which are generated conditioned on the actual future action. The last row shows the plot of dissimilarity (DSSIM) ((1-SSIM)/2) between the ground truth future frame and frame predictions conditioned on 15 different actions (same number of actions, as used in the discriminator). The steering command corresponding to action conditioned predicted frame with the lowest dissimilarity is selected and compared with the actual steering command to detect an anomaly. It can be inferred from the plots that the given cases are non-anomalous. Fig. 11. Examples demonstrating the effectiveness of SFAM for anomalous cases in indoor environment. The set of images in the left are from an instance when the UGV takes a late right turn and on the right are from an instance when the UGV takes an early left turn. In each set, the images on the top are the ground truth of the frame predictions shown in the middle row of each set which are generated conditioned on the actual future action. The last row shows the plot of dissimilarity (DSSIM) ((1-SSIM)/2) between the ground truth future frame and frame predictions conditioned on 15 different actions (same number of actions, as used in the discriminator). The steering command corresponding to action conditioned predicted frame with the lowest dissimilarity is selected and compared with the actual steering command to detect an anomaly. It is seen that the third and fourth images on both the sets are anomalous since the predicted action conditioned frame does not provide the best match to the actual action conditioned frame.