{'fraction_non_alphanumeric': 0.03551921686411925, 'fraction_numerical': 0.013393403415095389, 'mean_word_length': 4.827941653160454, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Uncertainty-Aware Visual Perception for Safe Motion Planning

For safe operation, a robot must be able to avoid collisions in uncertain environments. Existing approaches for motion planning with uncertainties often make conservative assumptions about Gaussianity and the obstacle geometry. While visual perception can deliver a more accurate representation of the environment, its use for safe motion planning is limited by the inherent miscalibration of neural networks and the challenge of obtaining adequate datasets. In order to address these imitations, we propose to employ ensembles of deep semantic segmentation networks trained with systematically augmented datasets to ensure reliable probabilistic occupancy information. For avoiding conservatism during motion planning, we directly employ the probabilistic perception via a scenario-based path planning approach. A velocity scheduling scheme is applied to the path to ensure a safe motion despite tracking inaccuracies. We demonstrate the effectiveness of the systematic data augmentation in combination with deep ensembles and the proposed scenario-based planning approach in comparisons to state-of-the-art methods and validate our framework in an experiment involving a human hand.


Introduction
When robots operate in close proximity to humans, safety requires the avoidance of unintended collisions. In order to detect any form of obstacles, visual perception using deep learning (DL) has gained growing attention in recent years [1,2]. While DL-based perception systems have achieved impressive results in various tasks [3,4,5], several issues prevent their applicability in many safetycritical systems. Firstly, DL models are typically trained using large datasets [6,7], which are often not available for specific robotics tasks. However, the creation of task specific datasets is generally costly, such that small datasets are desirable. Moreover, DL models are prone to causing prediction errors in previously unseen situations [8,9,10], such that a reliable quantification of their predictive uncertainty is important to enable a cautious behavior of the robotic system. Finally, even when uncertainty information is available, it is often not in the parametric form required for uncertaintyaware planning algorithms, such that additional simplifications are necessary [11,12,13].
Related Work. A common approach in existing work is to address visual perception by semantic segmentation [5,14,15,16]. In recent years, the focus of segmentation approaches has been on DL architectures due to the advances by [7,17]. Even though these approaches have achieved high predictive accuracies, they typically rely on large training datasets. When only small datasets are available, numerous methods for data augmentation have been proposed [18,19,20,21] to improve generalizability and robustness. In semantic segmentation, datasets are typically increased by a low factor, often two, by applying up to three augmentation methods to avoid overfitting [20,22,23]. However, this is insufficient for small task-specific datasets which are often not representative in terms of diversity and therefore require massive, systematic data augmentation. Moreover, DL is inherently miscalibrated and therefore often produces over-confident predictions [8,9], such that research has focused on improving uncertainty quantification of DL [24,25]. While Bayesian neural networks with dropout have attracted interest early on [26,27,28], deep ensembles [10,29,30] have gained increasing attention recently due to their often demonstrated superior performance [9,31,32,33,34],  Figure 1: Our framework combines visual perception with safe motion planning. By applying systematic data augmentation and an ensemble scheme to semantic segmentation, we obtain reliable probabilistic occupancy information in uncertain environments. Based on the perception data, we plan a chance-constrained path and a maximum velocity profile such that safety of the real robot motion is ensured despite possible tracking errors.
reasonable computational cost for inference and parallelizability. These advantages have also been shown when applying them to semantic segmentation [35]. The results of semantic segmentation can directly be used for robotic motion planning, but such approaches ignore uncertainties in the perception. Previous works on motion planning with probabilistic environment representation mostly assume known obstacle geometry and Gaussian distributed object position [12,36,37,38,39,40]. While these assumptions allow for the derivation of analytical chance constraints, the associated over-approximation of obstacles is problematic when performing tasks in complex environments. Other approaches rely on deterministic error bounds for positions estimated using visual perception, such that robust planning methods can be employed [41,42,43,44]. Since these error bounds require dense training data in practice, large datasets must generally be available for such approaches. Hence, there exists neither an approach for uncertainty-aware visual perception using semantic segmentation when merely a small, non-representative dataset is available, nor a method for safe motion planning using the flexible, probabilistic results from such a perception module.
Contribution. We propose a general framework for visual perception and motion planning as illustrated in Fig. 1. In the perception module, we enable the training of deep ensembles for semantic segmentation with small, non-representative datasets by systematically augmenting the data. For this, we artificially add missing diversity to datasets by modifying training images with seven different methods, such that we can massively increase the dataset size by a factor of 20 to ensure reliable probabilistic occupancy information. In the safe motion planning module, we avoid conservatism due to parametric obstacle representations by formulating the path planning problem in an uncertain environment as a scenario optimization problem. This allows us to directly determine the collision probabilities using the results from the probabilistic segmentation. For the resulting path, we propose safe velocity scheduling to ensure safe robot motion despite tracking inaccuracies. We demonstrate the effectiveness of the proposed systematic data augmentation for visual perception and the scenario-based path planning formulation in a comparison to state-of-theart methods. Moreover, we evaluate the safety and robustness of our framework in an experiment with a robotic manipulator, which is inspired by the knife game [45].

Problem Description
For a robotic system to be considered safe, it must reliably avoid collisions with its environment. When employing sensors such as cameras or LiDARs to determine surrounding obstacles, the perceived objects are uncertain, e.g., due to sensor noise or varying lighting conditions. In safety-critical applications, quantifying this uncertainty is important such that additional measurescan be taken in case of high uncertainty. We consider widely-used camera-based perception based on a DL model, which requires labeled data for training. However, due to the lack of suitable training data for many specific robotic applications and the inherent miscalibration of neural networks [8], it is generally difficult to obtain an accurate representation of the environment with reliable information on uncertainty. Therefore, we investigate this problem in this paper as formalized in the following.
Problem 1 (Visual perception with uncertainty representation). Assume a small with low variety is given, which contains images C ∈ R H×W×3 with height/width H/W of marked obstacles with arbitrary pose, geometry and size. Based on the dataset, we consider the problem of training a DL model f : R H×W×3 ×Θ → [0, 1] H×W with parameters θ ∈ Θ which outputs for each point (i, j) in an image C the probability of being occupied by an obstacle O, i.e., P((i, j) ∈ O) = f ij (C, θ).
Based on the probabilities of obstacles in the image space, the robotic system must be capable of planning a safe trajectory of poses p ∈ T in the task space T . Since each pose p implies that the robot occupies some region R(p) ⊂ W of the physical workspace W, a pose p is only collision-free if the set R(p) lies completely in the obstacle-free subset W free ⊂ W. However, this cannot be ensured deterministically in general as merely an uncertain estimate of the obstacles and consequently W free is available from the visual perception module. Therefore, we aim to satisfy the constraints imposed by W free probabilistically via individual chance constraints with a prescribed probability threshold δ ∈ (0, 1). This leads to the following formal definition of a δ-safe motion. Definition 1 (δ-safe motion). The motionσ : [0, T ] → T executed by a robotic system within the time interval [0, T ] is called δ-safe if it satisfies P (x ∈ W free ) ≥ 1 − δ, ∀x ∈ R(σ(t)), ∀t ∈ [0, T ].
In this definition, safety is introduced using an individual condition for each time instance, which is a commonly considered requirement for planning in uncertain environments [12,36,39,46,47,38,11]. However, these conditions are posed on the motionσ(·) realized by the robotic system, which is a significantly stronger notion of safety than merely requiring their satisfaction for the planned trajectory σ : [0, T ] → T . Therefore, this safety notion clearly cannot be ensured without additional information about the dynamics of the robotic system, such that we assume the availability of a control law with guaranteed tracking error bounds as formalized in the following. Assumption 1 (Velocity-dependent tracking error bound). The tracking error of the robotic system is bounded by a non-decreasing function of its reference velocity, i.e., σ(t) − σ(t) 2 ≤ γ( σ(t) ).
This assumption reflects the fact that the tracking error of robotic systems often grows with increasing velocity of the reference due to unmodeled effects in control laws, e.g., friction or imprecise inertia parameters. Note that it does not require exact tracking for zero velocity, such that it can also be employed for underactuated robots, which can exhibit relatively large tracking errors at low velocities. Reducing the conservatism for such systems using more sophisticated controllers based on barrier functions [41,42] or contraction theory [43, 44] is subject to future work.
In addition to safety, other criteria such as the path length often have to be considered during planning. In order to compactly represent these criteria, we assume that they are summarized in a pathcost function c : T → R 0,+ . Therefore, we address the following safe vision-based planning problem. Problem 2 (Safe vision-based motion planning). Given the uncertain estimate of the obstacles obtained from the visual perception system and the velocity-dependent tracking error bound in Assumption 1, we consider the problem of finding a trajectory σ(·), which minimizes the cost c along the path defined through σ(·) and ensures δ-safety of the resulting executed motionσ(·), i.e.,

Visual Perception with Uncertainty Representation
In order to address Problem 1, it is necessary to probabilistically solve a classification problem for each pixel, which is commonly referred to as semantic segmentation. Therefore, we briefly introduce the fundamentals of training DL models for semantic segmentation in Section 3.1. In Section 3.2, we show how standard semantic segmentation approaches can be extended by considering an ensemble of DL models to obtain occupation probabilities for each pixel. Finally, we propose a systematic data augmentation approach for artificially adding missing information to datasets in Section 3.3.

Deep Learning for Semantic Segmentation
For classifying each pixel in an image using semantic segmentation, we employ the commonly used approach of fully convolutional networks (FCNs) with an encoder-decoder structure [7,17,48]. The encoder applies convolutional and pooling layers, capturing contextual information in a feature vector with downsampled spatial dimensions. In the decoder, the spatial dimensions are upsampled back to the input image size H ×W , allowing for pixel-wise dense predictions. We apply atrous convolutional layers [49] with different rates in parallel (Atrous Spatial Pyramid Pooling), which allows to capture objects at multiple scales [50,51]. In the final layer of the model, we employ the softmax activation function for the output channel of every image pixel (i, j). Denoting the parameters of the DL model byθ, this yields a functionf ij (·,θ) for every pixel, which outputs a probability-like value for pixel (i, j) not being occupied by an obstacle, i.e.,f ij (C,θ) ∈ [0, 1]. For the training process, we employ the commonly used cross-entropy loss. It is computed pixel-wise and summed over the spatial dimensions of the final layer, resulting in the total loss L(θ) = ij L (x ij , y ij ,θ), where y ij = 0 if (i, j) ∈ O, i.e., pixel (i, j) lies in an obstacle, and y ij = 1 otherwise.

Probabilistic Semantic Segmentation using Deep Ensembles
Even though DL models for semantic segmentation yield probability-like outputs, these values are generally not well suited as a measure of uncertainty due to the inherent miscalibration and overconfidence of NNs [8,52]. We address this issue by employing an ensemble of models which is known to produce well calibrated uncertainty predictions [9,31,32,34]. An ensemble consists of multiple distinct models, called ensemble members. For training the ensemble, we initialize each ensemble member with random model parameters and randomly shuffle the training data before each epoch [10]. The models are trained independently, such that they capture different features within the data. For inference, the individual member predictions are combined to the final prediction by considering the ensemble as a uniformly-weighted mixture of M models, i.e., f (C, θ) = M m=1f (C,θ m )/M , whereθ m denotes the parameters of the m-th member concatenated into the overall parameter vector θ. Since each function f ij (·, θ) also yields values in the range [0, 1], but is generally well calibrated, we use it to determine the occupation probabilities of pixels in the following.

Systematic Data Augmentation for Dataset Diversification
While deep ensembles can also be used for learning from small datasets in principle, it has been demonstrated that the resulting performance in semantic segmentation strongly depends on the size of the training set [6]. Moreover, due to the low variation in the training dataset considered in Problem 1, the estimated epistemic uncertainty can often be high when actually employing the deep ensemble in applications with more diverse images. Therefore, directly training a deep ensemble for semantic segmentation using a small dataset with low variation would lead to over-conservative performance in safety-critical applications. In order to mitigate this effect, we propose to systematically augment the training data to artificially add the missing variation to the data. The underlying idea of this augmentation is to apply label-preserving transformations to the annotated images, as well as to the corresponding segmentation masks [53,54,55,56,57]. This idea can be exploited, e.g., when merely few images of the different operating environments are available to artificially modify their backgrounds. Moreover, the robustness of the ensemble against partial object occlusions can be increased by randomly erasing parts of the labeled object. In total, we identify seven systematic augmentation methods for straightforwardly adding missing information to segmentation datasets, which are illustrated together with their intended diversification in Fig. 2. This allows us to substantially improve the performance and reduce the conservatism of the uncertainty quantification using deep ensembles when only a small non-representative dataset is available.

Safe Vision-Based Motion Planning
Based on the segmentation results, the goal is to plan and execute a δ-safe motion as introduced in Problem 2. To this end, the planning problem (2) is split into path and trajectory planning. For solving the former, we present a scenario optimization approach in Section 4.1. Subsequently, we determine the maximum velocity profile along the path still admitting a δ-safe motion in Section 4.2.

Path Planning as Scenario Optimization Problem
In order to admit a planning in the task space T of a robotic system, it is generally necessary to augment the probabilistic semantic segmentation result to 3D by using depth information, e.g., from a LiDAR scanner, a stereo camera system or available knowledge about the scene. This allows us to compute the occupancy probability P (x ∈ W free ) for each point x in the 3D workspace W. Moreover, we can determine the possibly occupied region R(σ(t)) ⊕ B(γ( σ(t) )) in the workspace for each pose along a trajectory σ(t) ∈ T , where B(r) is a sphere with radius r, and γ( σ(t) ) represents the tracking error bound defined in Assumption 1. Therefore, the condition for a pose σ(t) ∈ T being δ-safe can be expressed as P (x ∈ W free ) ≥ 1 − δ, ∀x ∈ R(σ(t)) ⊕ B(γ( σ(t) )). As the dependence on the actual velocityσ(·) is not suitable for standard path planning algorithms, we substitute it with a desired minimum velocity v ∈ R + . This allows us to split the safe motion planning problem into a simple path planning problem followed by a velocity scheduling problem.
For planning with uncertainty, strong assumptions are usually made about the shape of obstacles and their probability distribution to derive analytic expressions for the constraint in Problem 2 [12,36,39,40,46,47]. In order to avoid the associated conservatism, we reformulate Problem 2 as a scenario problem [58,59]. For this, the path is discretized into K poses (p 1 , . . . , p K ) and a fixed set R 0 := R(0) is defined, such that for all possible poses p ∈ T , the set R(p) can be described as a rigid motion T p (·) of R 0 , i.e., R(p) = T p (R 0 ). This allows us to approximate the safety constraint using random samples x (n,k) = T p k x (n) 0 , where N x ∈ N vectors x (n) 0 are drawn from a uniform distribution U(R 0 ⊕B(γ(v))), which is a common approach for reformulating robust into scenario constraints [58]. This leads to the scenario optimization problem min (p1,...,p K ) K j=1 c(p k ) such that P x (n,k) ∈ W free ≥ 1−δ, ∀n = 1, . . . , N x , k = 1, . . . , K, (2a) Based on the solution (p * 1 , . . . , p * K ) of this problem, we define the continuous path σ * : [0, 1] → T by linearly interpolating between all p * k such that σ * (k∆s) = p k+1 for ∆s = 1/(K −1). Due to the strong theoretical foundations of scenario optimization [58,59], this provides a well-suited approach for the obstacle representation resulting from the proposed uncertainty-aware visual perception.

Safe Velocity Scheduling
While the path obtained using (2) admits a trajectory executed at the specified minimum velocity v, a faster execution is possible when the obstacles are sufficiently far away. To find the maximum admissible velocity v * (s) for δ-safety at some point σ * (s), we need to determine the distance between the robot and the closest point which is not δ-safe. This distance can be compactly expressed as which can be effectively approximated by sampling multiple positions x o and merely optimizing with respect to x r . Using the distance d o (σ(s)), we can easily determine v * (s) since γ(σ(s)) ≤ (a) 50 Training images (b) 50 Test images Figure 3: We split our available annotated samples into a training and a test set. The former contains only little variety and is thus not representative for the test set, making the training task a good example for Problem 1.
d o (σ(s)) must hold for δ-safety. Since the chain ruleσ(s) = dσ(s) /dsṡ = σ (s)ṡ admits a parameterization of the velocity in terms ofṡ, we can obtain v * (s) by solving the optimization problem v * (s) = max wherev denotes the maximum executable velocity of the robot. We discretize the range of s ∈ [0, 1] into l steps and use a line-search to efficiently compute (4) at the discretization points. Finally, numerical integration of the obtained velocity profile yields a smooth δ-safe trajectory σ * (t) [60,61].

Evaluation
In order to evaluate the proposed framework, we apply it to planning a safe motion in proximity to a human hand. We first examine the effect of data augmentation and the ensemble scheme in Section 5.1. As a segmentation model, we employ the state-of-the art semantic segmentation architecture DeepLabv3 [62] with the ResNet50 CNN [63] as the backbone. In Section 5.2, we show the high flexibility of the proposed scenario formulation for planning in uncertain environments. Finally, in Section 5.3, the effectiveness of the safe vision-based motion planning framework is demonstrated by playing a variant of the popular knife game [45] with a 7 DOF robotic manipulator.

Uncertainty Quantification in Visual Perception with Small Datasets
Dataset. While there exist some datasets [64,65] for hand segmentation, they are mainly targeted for hand-based activity recognition. This leads to the problem that they contain many images with closed hands, but only very few with separately labeled fingers. Therefore, these datasets are not well-suited for applications which require the highly accurate segmentation of hands, which is necessary for the envisioned problem of controlling a robot in close proximity to the hand. In order to circumvent this issue, we create our own dataset. Since the labelling procedure is time consuming, only 100 images are created and divided into a training and a testing set as depicted in Figures 3a and 3b. It can clearly be seen that the training set is chosen such that it exhibits only limited diversity, e.g., all images have monotonous and very similar backgrounds. In contrast, the test set includes highly diverse images with hands from multiple people in various environments. Therefore, the training set is not representative of the test images, such that the resulting segmentation task is a good instance for Problem 1, and thus well-suited to demonstrate the effectiveness of the proposed systematic augmentation procedure in adding missing variability to a dataset.  Augmentation. For an individual comparison of the augmentation methods, each one is used to create an augmented version of every annotated sample, yielding seven training sets with a size of 100 images. For background replacement, we choose randomly from 15 backgrounds. Each dataset is used to train 20 models. Their performance on the test set is evaluated using the average pixel accuracy (PA) and mean intersection over the union (mIoU) obtained, shown in Table 1. Background replacement, color modification and random erasing perform best due to the lack of diversity of our training set in terms of colors and missing occlusions. Generally, it depends entirely on the structure of the data which augmentation works best. Thus, our results differ from [21], where geometric methods perform best.  We also evaluate the combination of all seven methods within our augmentation scheme. For this purpose, we apply our scheme and the methods proposed in [18,19,20,23] to create two augmented datasets each: A small one containing 200 images and a large one containing 1000 images. In Table 2, the results for our scheme are presented together with the respectively better results of the comparison methods. It can be seen that our scheme provides significantly stronger improvements in model performance for both degrees of dataset inflation. The best performance is obtained when applying our scheme to increase the dataset size by a factor of 20 to 1000 images. Our results show the importance of systematically addressing the lacking variety within the training set by combining different augmentation methods.  Table 3: Predictive uncertainty improves with increasing ensemble size M , which is shown by decreasing Brier score (BS) and negative log-likelihood (NLL).

Uncertainty Quantification.
Our safe planning framework builds on the assumption that for each pixel, the segmentation output can be used as an occupation probability. To justify this assumption, we first examine calibration via the reliability diagram [66]. For this, the confidence interval [0.5, 1) is partitioned into ten equally sized bins. For each bin, the prediction accuracy for the pixels contained in the bin is calculated and plotted against the average confidence value of the pixels within the bin [25]. We evaluate three differently sized ensembles, M ∈ {2, 5, 10}, and the individual members of the medium-sized ensemble. The results depicted in Figure 4 show that calibration improves with increasing ensemble size. Moreover, we measure predictive uncertainty via two common metrics, the Brier score (BS) [67] and negative log-likelihood (NLL) [10,33]. Their average values listed in Table 3 significantly decrease with increasing ensemble size, which is consistent with the results reported in [10]. The reliability curve as well as BS and NLL indicate only small improvements for M > 5. Therefore, we conclude this ensemble size to maintain a good balance between computational demand and calibration suitable for the considered visual perception task.

Comparison of Uncertainty Representations for Collision Avoidance
Performance Comparison. We first compare the performance of path planning with our perceptionbased collision avoidance with three popular methods [12,39,40] that are based on the common assumption of uncertain position x o ∼ N (x o , σ 2 I 3 ), and known geometry and orientation of the obstacles. We consider a 4D task space T composed of the robot x-, yand z-position and its orientation around the z-axis. The set R(p) is described as an ellipse parallel to the x-y-plane, .01 m and we consider spheres and cuboids as obstacles. Our cost function, which is described in the appendix, penalizes the area between the planned path and the straight line connecting x init and x goal . For applying our approach, we consider that the occupancy probability decreases linearly with the distance d to the obstacle surface 1 , becoming zero at d = d stop . Since we employ the chance constraint with probability δ = 0.05, this leads tod = 0.95d stop representing the extended boundaries of the uncertain object. Due to the Gaussian distribution of the object positions, these boundaries are equivalently parameterized using 2σ in the existing approaches [12,39,40]. For comparing the different methods, we create a simple scene containing 3 large obstacles and a more cluttered scenario with 8 small obstacles. We exemplarily consider a scenario chance constrained (SCC) RRT* algorithm for solving (2), which employs the random samples from scenario optimization for performing the necessary collision checks. Running the SCC-RRT* algorithm with N x = 100 for 100 times per method and scene over a range of different values ofd   Figure 5: We compare different uncertainty-aware collision avoidance strategies for path planning within the CC-RRT* framework. Our perception-based approach leads to less conservative paths, with the advantage becoming stronger for more complex environments. Figure 6: Approximating the hand with many simple geometries and assuming zero uncertainty leads to roughly similar paths for the different methods. However, the procedure is tedious and being less accurate worsens path quality. and 2σ results in the normalized average costs illustrated in Fig. 5. While the growth rate of the cost for the simple scenario is almost identical for our method and the approach proposed in [12], the limited flexibility due to the parametric object representation in existing methods [12,39,40] leads to a significantly stronger increase of the costs in cluttered environments compared to our approach.
Efficiency Comparison. Additionally, we compare the computational efficiency of the proposed scenario approach with existing parametric approaches on real perception data of a human hand. Due to the lack of a principled approach for automatically determining spheres and cuboids using the probabilistic hand representation from deep ensembles, we manually specify them as illustrated in Fig. 6. Since these objects practically cover the estimated hand already, we run the existing approaches with σ = 0. While this results in similar costs for all methods, the representation of the hand using many simple objects causes a significant computational overhead, e.g., an average of 24.9 s for the chance constraint approach [12] in comparison to 9.9 s for our proposed scenario approach. While representations using fewer objects also admit a faster computation, the cost generally increases, leading to a cost-efficiency trade-off. Since the scenario approach directly employs the probabilistic semantic segmentation, it does not suffer from this issue. Therefore, the proposed scenario approach simultaneously achieves flexibility and efficiency. In order to show the real-world applicability and safety of our approach, we play a version of the popular knife game [45] with a KUKA iiwa robotic manipulator and human hands. A camera positioned 0.5 m above the working area takes images of the workspace. We apply the trained deep ensemble and augment the 2D probabilistic semantic segmentation result to 3D by assuming fixed hand height. We plan a δ-safe path with the SCC-RRT* algorithm and execute the motion with a maximum velocityv = 0.2 m /s, see Figure 7. As shown in our supplementary video, the experiment is successfully repeated with different individuals, varying lighting conditions and backgrounds, which demonstrates the practicability and robustness of our safe perception-based planning framework.

Limitations and Outlook
While our approach can handle non-static scenes via regular replanning in principle, this is time consuming due to the computationally expensive RRT* planner used to solve (2). Therefore, our current implementation is limited to static scenes in practice. Moreover, although DL models can distinguish between more than 20 semantic classes in principle [7], the obstacle classes considered during training might not cover all possible objects, potentially causing collisions. Similarly, systematic data augmentation can only increase diversity in aspects that are considered during the augmentation design. Therefore, our visual perception approach puts a higher responsibility on its human designer compared to purely data driven approaches [5,23]. In the future, we plan to use smaller semantic segmentation models such as [68], extend our method to estimate obstacle dynamics and allow adaptive online planning based on a scenario MPC approach, so that it can be safely employed in highly dynamic environments. [37] B. D. Luders, S. Karaman, and J. P. How. Robust sampling-based motion planning with asymptotic optimality guarantees. In Proceedings of the AIAA Guidance, Navigation, and Control Conference, pages 1-25, 2013.
[38] P. Wu, L. Li, J. Xie, and J. Chen. Probabilistically guaranteed path planning for safe urban air mobility using chance constrained RRT*. In Proceedings of the AIAA Aviation Forum, pages 1-12, 2020.
[ An autonomous system has to obtain accurate and reliable information about its environment to safely interact with the world. Due to advances in deep learning (DL), specifically the development of convolutional neural networks (CNNs) [54], and the availability of cameras at low cost, visual perception has become increasingly popular. CNNs are widely used for bounding box obstacle detection [69], with applications ranging from autonomous driving [70] to service robotics [71]. Since the bounding box approach can lead to a significant over-approximation of obstacles, it has been proposed to obtain a more detailed representation of the environment using semantic segmentation, which refers to the classification of each pixel in an image [7]. The development of fully convolutional networks (FCNs) [7,48] has greatly increased the performance of semantic segmentation and led to its application in various visual perception tasks. For example, semantic segmentation is applied to detect roads for autonomous driving in [23]. The models are trained with the KITTI Road dataset [72] and an ensemble scheme is employed to increase the prediction accuracy. In [16], data by a monocular camera is combined with stereo features to generate obstacle maps in cluttered underwater environments. Other applications of semantic segmentation in robotics include manipulation [73], mapping [74] and localization [75,76]. Semantic segmentation has also been directly applied to 3D data in the form of point clouds, which is referred to as point cloud semantic segmentation (PCSS) [77]. Point cloud data can be directly obtained with an RGB-D camera that acquires both RGB and depth information. PCSS has for example been applied to construct highresolution maps for autonomous driving [78] or to perform simultaneous localization and mapping (SLAM) [79]. While all of these semantic segmentation approaches achieve high accuracy, they rely on large-scale datasets for training. Even though there exist several large public standard benchmark datasets such as Semantic3D.net [80] with a large collection of urban scenes and the Stanford Large-Scale 3D Indoor Spaces Dataset (S3DIS) [81], this need for large datasets prevents the application of existing methods to highly task-specific problems. For example, for robot-assisted dressing [13] or highly accurate hand segmentation as considered in our experiment, only small datasets can be obtained due to the required laborous manual data labelling [77]. Moreover, existing approaches do not consider the uncertainty associated with the predictions, which can lead to potentially unsafe predictions rendering them unsuitable for safety-critical applications.

A.2 Motion Planning with Uncertainties
A fundamental and widely studied problem in robotics is the planning of a collision-free motion between a start state and a desired goal state. Graph-based algorithms like A* [82] discretize the state space into a grid and are guaranteed to find the optimal solution. However, these algorithms do not scale well with the problem size. Stochastic sampling-based methods such as probabilistic roadmaps (PRMs) [83] for multi-query and rapidly-exploring random trees (RRTs) [84] for single-query planning do not require a discretization of the state space. These methods scale more effectively with problem size and have been very popular for the last 25 years. However, they do not provide optimality guarantees and it has even been shown that the solution returned by RRT is suboptimal with probability one [85]. Asymptotically optimal variants of these algorithms, named PRM* and RRT*, have been proposed in [85].
In real-world applications, information about the robot state and the environment is typically subject to uncertainties, e.g., due to sensor noise. Therefore, several works have investigated motion planning under uncertainty. A chance-constrained variant of the RRT* algorithm (CC-RRT*) for convex polytopic obstacles is proposed in [37]. Several other works [11,46,86,87] also adapt the RRT* algorithm for planning with uncertainties. An alternative approach is stochastic model predictive control (SMPC) [40,36]. Therein, the collision conditions are formulated as soft/chance constraints for an optimization problem that is iteratively solved online. Partially observable Markov decision processes (POMDPs) have also been studied for planning under uncertainty [47,88], but their use is mostly limited to low-dimensional problems due to the high computational demand.
The safety guarantees ensured by such planning approaches can be enforced using different types of constraints. Some approaches rely on joint chance constraints [89,90], i.e., the probability that a collision happens at any time step must be below a prescribed threshold. In recent years, individual chance constraints for single time steps have become very popular [11,12,36,38,39,46,47]. While the certifiable probability for the safety of the overall path decreases quickly with the path length for individual chance constraints, they do admit guarantees for the overall probability of path safety in principle [46]. Moreover, individual chance constraints offer many advantages in practice, such as leading to less conservative results or yielding computationally more efficient optimization problems [91], while simultaneously ensuring suitable constraint satisfaction in practice as demonstrated by their well-established application in model predictive control [92]. Due to these reasons, we consider individual chance constraints for our approach. To allow for a more accurate representation of the environment, polytopic robot and obstacles have also been considered. It is shown in [95] that safe motion planning in 3D with uncertain obstacles modeled as polytopes with Gaussian-distributed faces is N P -hard w.r.t. to the number of obstacles. Even when restricted to a graph, N P -hardness of the motion planning problem remains. The assumption of Gaussianity is dropped in [46,21], where polytopic obstacles with known shape and orientation and uncertain position are considered. The position is modeled as a random vector with unknown probability distribution that is assumed to belong to a moment-based ambiguity set of distributions. Chance constraints are formulated to enforce collision avoidance under the worst-case distribution in the ambiguity set. These constraints are used by [46] in a risk-averse RRT* algorithm for uncertain dynamic environments. The gain in flexibility with this approach comes at the price of a very high computational cost. Moreover, concave objects like hands often cannot be described satisfactorily by polytopes.

A.3 Visual Perception with Semantic Segmentation for Motion Planning
While semantic segmentation and motion planning have attracted a lot of interest in recent years, only a few works have combined both. In [97], semantic segmentation is performed on stereo camera images taken by an unmanned aerial vehicle (UAV) in order to detect texture-less areas where visionbased localization fails. Trajectories avoiding these regions are planned with the A* algorithm and B-spline optimization. Road following for a mobile robot based on semantic segmentation is investigated in [15]. A target point in the moveable area extracted online with a camera is used as reference to determine the desired steering angle. When a robot is in motion, its perceived scene is subject to change. Therefore, [98] interpolate past surrounding-view images to predict future semantic maps of the scene. Trajectories for a self-driving vehicle are planned by sampling a set of random trajectories and choosing the one with minimum cost. A similar planning strategy is employed in [99] to compute semantically-aware trajectories for autonomous off-road driving. In previous work on motion planning with semantic segmentation, the uncertainty arising from perception has not been taken into account. Moreover, for the tasks considered, large representative datasets could be used. There exists neither an approach for uncertainty-aware visual perception using semantic segmentation when merely a small, non-representative dataset is available, nor a method for safe motion planning using the flexible, probabilistic results from such a perception module.

A.4 Planning with Visual Perception in Feedback Control Loops
In recent years, the integration of visual perception using machine learning into control loops has gained significant research interest, which has led to a variety of approaches combining perceptionbased control and planning. Starting with the derivation of efficient Lipschitz constants for neural networks [100], Lipschitz-based error bounds for deep learning predictions have become very pop-ular in control. These error bounds can be used to robustly ensure safety of control with DL based visual perception as state sensor, e.g., using control barrier functions [41,42] or contraction metric approaches [43,44]. These safety guarantees for control can in turn be directly employed for sufficiently robust planning, such that the overall safety of the perception, planning and control approach can be ensured [42,44]. Due to the strong theoretical requirements of Lipschitz based error bounds, these approaches generally require high training data densities to admit reasonable error certificates and often depend on noise-free training samples. Therefore, approaches relying on Lipschitz continuity of DL are not suited for the considered robotics applications, where only a small, non-representative dataset is available. In addition to these deterministic error bound approaches for DL, a variety of other methods as been proposed. For example, PAC-Bayes methods as proposed in [101] can provide probabilistic guarantees on the safety of control and consequently for planning, but require i.i.d. training samples from the true distribution of environment images, which is a challenging condition in practice. Finally, there exist approaches which merely use visual perception to define goal positions for planning trajectories, which can subsequently be tracked with safe controllers [102]. Since such methods do not consider possible errors due to visual perception, they do not provide guarantees for the executed trajectory. Therefore, existing methods originating from control-theoretic considerations cannot be applied to our considered problem.

B.1 Background on Semantic Segmentation using Deep Learning
The goal of semantic segmentation is to assign a label to each pixel in an image corresponding to its semantic class, i.e., to obtain a pixel-wise classification map. The technique can be used not only to detect the presence of certain objects in the work space, but also to accurately determine their boundaries. Such information can be used in robotics, e.g., for vision-based obstacle avoidance, manipulation or localization [23,5]. In the following, we briefly describe the technical background and our training process.
Standard image classification does not take into account the spatial relations of objects to each other. In contrast, semantic segmentation needs to incorporate knowledge about the scene context into the prediction. While global information is crucial for correct classification, local information is required to make accurate predictions on a pixel-level. Hence, semantic segmentation models must balance semantics (what) and location (where) [48]. This trade-off is reflected in the network architectures, as described below. The currently used models for semantic segmentation are based on the idea of fully convolutional networks (FCNs) [7,48]. A FCN mainly consists of three types of layers: Convolutional layers that extract features by convolving a kernel of learnable weights, pooling layers for reducing or increasing the spatial resolution and nonlinear layers applying activation functions to the feature maps. In contrast to CNNs as used for image classification, the last layer of an FCN has the same spatial dimensions H × W as the input image. Moreover, its channel dimension is equal to the number of semantic classes K, which allows for pixel-wise dense predictions. Many popular network architectures for semantic segmentation developed in recent years, such as U-Net [17] and SegNet [103], have some kind of encoder-decoder structure. The encoder creates a feature vector with downsampled spatial dimensions that captures context information. In the decoder, upsampling with deconvolution and unpooling layers is used to recover the spatial dimensions. The loss of fine-grained information during the encoding process can be addressed, e.g., by applying skipconnections [48,17] between the encoding and decoding path. In this way, fine predictions can be obtained while still taking deep context information into account. The popular U-Net [17] consists of a contracting path for capturing context and a symmetric expansive path for localization. In the decoder, the upsampled output is concatenated with the respective cropped high-resolution feature maps from the encoder step, followed by two convolutional layers and another upsampling step. For more information on the many different architectures developed for semantic segmentation, the reader is referred to [104].
In this work, we use the DeepLabv3 network [62], a state-of-the art semantic segmentation model developed by Google, with the ResNet50 CNN architecture [63] as the backbone. DeepLabv3 employs atrous (dilated) convolution, a special type of convolution whose filter window size is determined by the atrous rate. By leaving holes (trous in French) in the kernel, the field of view is increased while maintaining the same computational demand as standard convolution. In DeepLabV3, atrous convolution is applied both in cascaded and parallel modules to recover the spatial resolution after downsampling with max-pooling and striding operations. Combining this with atrous spatial pyramid pooling [50] allows the model to capture objects at multiple scales. In the following, we denote the model parameters, i.e., the kernel weights and biases, by θ, and aggregate the color channels of all pixels in a tensor C ∈ R H×W ×3 . For every pixel (i, j), i = 1, . . . , H, j = 1, . . . , W , the softmax function is applied to the respective output channel, yielding K values f ij,k (C, θ), k = 1, . . . , K adding up to one. These softmax outputs are often interpreted as the predicted class probabilities. However, due to the inherent miscalibration of NNs [8], the values do generally not represent true class probabilities. A classification mask can be obtained by selecting the channel with the largest softmax output as the predicted class, i.e.,ŷ ij = arg max k f ij,k (C, θ). For the training process, we apply the softmax cross-entropy loss, which is a common choice for semantic segmentation. It is computed pixel-wise and summed over the spatial dimensions of the final layer, i.e., the total loss is given by L(θ) = ij L (x ij , y ij , θ), where y ij ∈ (0, 1) denotes the true semantic class of pixel (i, j).

B.2 Additional Information on the Systematic Data Augmentation
As our dataset is very small and non-representative, we employ systematic, massive data augmentation with several methods, each of which is motivated by a diversification goal.
• Robustness against varying object orientations: Since our dataset contains mainly images with a very similar orientation, a model trained directly with it could potentially fail when faced with images with strongly varying object rotations. Therefore, we aim to add additional diversity in this aspect by artificially flipping and rotating the training images. Flipping mirrors an image across either its horizontal or vertical axis. Rotating an image by a random angle can be done in two ways. Either, the scale is kept unchanged, which leads to some image parts being cut, or the image is downsized to completely fit into the original frame. We use both variants and apply zero-padding to the image borders.
• Robustness against object shape variations: The training images are all taken from a very similar view angle and contain only hands from two different people. Therefore, the training hand shapes are very similar, although the trained DL model must be capable of handling a variety of different hands in the experiment. Hence, we apply grid distortion to simulate different view angles or hand postures. An image is gridded into 5x5 cells and warped randomly with specified distortion limits. Constant padding is applied to the image borders.
• Independence of object color and lighting: As images from merely two different hands in similar environments are used in the training dataset, the hands have an almost identical color in the images. Moreover, the color of the hands does not appear in the background of the training images. In order to avoid an overfitting to this color in the training process, we apply color modifications to the hands such that the segmentation model yields good results independent of the color of the hand and the lighting conditions.
• Robustness against object occlusions: All images in the training set show complete hands. However, the end-effector of the robotic manipulator can occlude parts of the hand in the experiment, and images in the test set contain object occlussions, too. Therefore, we randomly erase a rectangle from the image and mask it with a random color. This is done such that the model learns to detect and segment partially occluded objects. Our implementation differs from [18] in that we mask each image with a new randomly selected color instead of using the same color for all images.
• Robustness against random camera noise: Since we consider very small training datasets, the DL model can potentially overfit to the camera noise in principle. In order to avoid this behavior, we add Gaussian-distributed noise to the HSV color channels to increase robustness against camera noise.
• Invariance to different operating environments: Since the images in the training set have extremely similar backgrounds, the performance of a trained DL model in different operating environments is possibly poor. Moreover, an overfitting to the background colors is likely. Therefore, we artificially change the image background by replacing the pixels not belonging to the hand mask with those from another image. We use the "fake" backgrounds shown in Figure 8, which are all taken from (around) the lab.
Note that these augmentation methods have been previously proposed in identical or similar forms, e.g., flipping in [54], rotating in [105], color modifications in [55], random erasure in [18], Gaussian noise in [56], and artificial backgrounds in [57]. However, these augmentation methods are typically used to generally increase the training set size, but not to specifically add missing diversity to it. Therefore, generally applicable methods such as flipping, rotating and random noise can be often found in literature, while other approaches are used far less frequently. Moreover, usually only a few methods are combined to mildly increase the dataset size, while we aim for a massive dataset augmentation to add missing information to the data. Note that we do not just simply combine all existing approaches for data augmentation, but only use techniques relevant for the considered robotics application. For example, we do not make use of the two popular methods cropping and scaling [23,21]. Since images of whole hands need to be segmented in our application, cropping is not useful as it cuts certain parts of the image away, including the hands. Furthermore, we refrain from applying scaling because the hands in our training set already have appropriate size and there is already sufficient variation in terms of size. Therefore, although the augmentation methods considered in this work are a suitable choice for many tasks in robotics, they are systematically chosen to address the lack of diversity in our training dataset.
To create as diverse training samples as possible, we can combine the augmentation methods by successively applying them to the same image. The key idea of our augmentation scheme is to replace the background and subsequently apply all other augmentation methods with a probability of 0.5 each. In that way, we not only apply multiple augmentation methods to the same image, but also the set of methods used varies from image to image. As a result, our augmentation scheme introduces much more variety than if certain methods were always applied in the exact same way and order. The order of the augmentation methods should nevertheless be considered. For example, we apply random erasing before rotation so that the masked rectangles are not always aligned with the image coordinates, but also rotated randomly sometimes.
We can use our scheme to create augmented datasets of desired size. If the number of available backgrounds N bg exceeds the number of augmented samples N aug we wish to create from an annotated image, then we choose randomly without replacement from the set of background images when creating an augmented sample. If N bg = N aug , then every background is used once per annotated image. If N bg < N aug , then we also create augmented samples without background replacement to prevent the "fake" backgrounds from dominating the dataset too much compared to the backgrounds in the original training set. For these augmented images, we apply all other augmentation methods except from background replacement. Our augmentation scheme is illustrated in Figure 9 for an image in the training set augmented by a factor of 4 and 20, respectively.

Theoretical Background on Scenario Optimization
Scenario optimization is an approach originating from control theory with the goal of reducing the conservatism of robust control by modulating robustness [58,106]. Assuming a set ∆ of uncertain variables ξ, robust optimization problems have the general form min where l : R d θ → R is the optimization objective, θ ∈ Θ ⊂ R d θ are the optimization variables and h : Θ × ∆ → R are constraint functions. Solving this optimization problem directly can lead to very conservative results, in particular if the realizations of ξ yielding the high costs are very unlikely. Therefore, the idea of scenario optimization is to relax the problem to a chance constrained optimization problem [106] min where ∈ (0, 1) is the probability of constraint violation. While this formulation allows to trade-off conservatism and robustness by choosing an appropriate value of , the constraint can often not be evaluated directly due to a lack of closed-form expressions for the probability. In order to mitigate this issue, the chance constraint is approximated using random samples of ξ, which are typically straightforward to obtain. This leads to the following formulation of a scenario optimization problem [58] θ * = arg min where X denotes some distribution over ∆. While this approximation cannot ensure that the chance constraint is always satisfied, a sufficiently large number of random samples can ensure that a violation of the chance constraint is unlikely. For example, when considering linear cost functions l(·) and convex constraints, the following result has been shown.

Theorem 1 ([58]
). Assume l is linear and h(·, ξ) is convex. If for β ∈ (0, 1) and ∈ (0, 1), then In this theorem, β is the probability that the chance constraint is violated due to poor realizations ξ (n) . Due to the logarithmic dependency of N on β, β can be made arbitrarily small while merely slightly increasing the required number of random samples. In contrast, is the probability of the original chance constraint (6b), which has a significantly stronger impact on the required number of random samples. Since we can choose β arbitrarily small, e.g., β = 10 −10 , the result of scenario optimization always satisfies the chance constraint in practice. Similar results under different assumptions have been shown for scenario optimization. Recently, it has even be shown that general, non-convex constraints and arbitrary cost functions also admit guarantees of this form [59]. Note that the complexity of the optimization problem directly impacts the certifiable probability for a given number of samples N in this case.
In order to see the relationship to the originally considered problem in the main paper, note that after the path discretization, we can express the search for an optimal path as This problem clearly constitutes a robust optimization problem of the form (5) . Therefore, we can directly transform (10) into the scenario optimization problem (2) in the main paper. Since the complexity of the robot shape directly influences the complexity of the constraints, it follows that the required number of samples N for a given probability depends on the particular planning problem. Even though this can make a large number of samples necessary, this issue can be mitigated by considering simplifications of the robot shape. This allows an intuitive trade-off between complexity and flexibility.
Note that is in principle also possible to integrate the individual ensemble members of the visual perception module into the scenario approach. However, there is no obvious practical advantage of considering the members instead of working only with the condensed information provided by the ensemble. This is due to the fact that rather few models are sufficient to achieve well-calibrated ensemble probabilities as shown in Fig. 4. Therefore, the additional computational complexity of computing all ensemble predictions is rather small, in particular since it can be parallelized if sufficient computational resources are available.

C.2 Safety of the Continuous-Path
In the scenario optimization problem (10), the safety constraint is only evaluated at discrete points p = p 1 , . . . , p K on the path. Thus, there may exist poses on the obtained path for which δ-safety is violated. In order to address this issue, we enlarge the set R 0 ⊕ B(γ(v)) from which the independent and identically distributed samples are drawn. The enlarged setR must be chosen such that for all poses p = λp k + (1 − λ)p k+1 , λ ∈ [0, 1], on the line segment between p k and p k+1 , the condition is satisfied. Here, by a slight abuse of notation, T p (R) := {x| x = T p (x 0 ), x 0 ∈R} denotes the rigid motion of setR into pose p. Condition (12) demands that the two sets T p k (R) and T p k+1 (R) from which the samples are drawn cover all points that the robot can possibly occupy on the linear transition from pose p k to p k+1 .

C.3 Scenario-Based Planning using a Chance Constrained RRT* Algorithm
For solving the scenario optimization problem, we exemplarily employ a chance-constrained version of the popular RRT* algorithm [85]. In this section, we briefly explain the algorithm and go into some implementation details.
The RRT* algorithm is a sampling-based method and therefore avoids a discretization of the task space 2 similar to the classic RRT algorithm [84]. The RRT algorithm builds a search tree, parametrized by a vertex set V and an edge set E, with the initial pose p init as the root node. In each iteration, a random sample p rand is drawn from the task space and the closest node p nearest in the tree is determined. If the distance between p rand and p nearest does not exceed a certain value and the connection from p nearest to p rand is collision-free, then p new = p rand can be added to the search tree. Otherwise, a pose within a range of p nearest in the direction of p rand is selected as the new node p new . In both cases, the vertex and edge set are updated as V = V ∪ {p new } and E = E ∪ (p nearest , p new ). After N iter iterations of the algorithm, the goal pose p goal is connected to the closest node in the tree. The final path is obtained by starting from p goal and iteratively going to the parent node until the root p init is reached.
The RRT* algorithm introduces several modifications to this process. A line cost function c line : T × T → R + 0 is defined to evaluate the quality of a line segment between two poses. Moreover, for each node p ∈ V, the cost of the optimal path from p init to p is stored as c(p). For selecting the parent node, the set T near containing all nodes within a certain Euclidean distance η of p new is computed. Then, p new is connected along a collision-free minimum-cost path, i.e., to the node p min ∈ T near for which c(p min ) + c line (p min , p new ) is minimal. Subsequently, an attempt is made to rewire the search tree with the newly added node p new . For all p near ∈ T near , it is determined whether they can be reached over p new with lower cost than for their current parent node p parent . If so, then the edge (p parent , p near ) is removed from the edge set E and replaced by (p new , p near ).
It can be shown that the probability that the RRT algorithm finds an optimal path is zero [85]. In contrast, the RRT* algorithm asymptotically converges to the optimal path if the cost function is monotonous and bounded [85]. For a detailed description of the RRT* algorithm, the reader is referred to [85].

C.3.1 Collision Checking
The key difference between our SCC-RRT* algorithm and previous approaches [37, 46, 38] is the collision checking. In order to evaluate whether a given pose p ∈ T is in collision with an obstacle, we follow the ideas of scenario optimization and approximate the constraints using random sampling. A fixed set R 0 := R(0) is defined. For all possible poses p ∈ T , the corresponding set occupied by the robot R(p) can be described as a rigid motion T p of R 0 , i.e., R(p) = T p (R 0 ). We approximate the safety constraint (11) by using N x random samples as introduced in Section 4.1. For this purpose, we draw N x ∈ N vectors x (n) 0 from a uniform distribution U(R 0 ⊕B(γ(v))). Then, we can determine the occupancy probability for each pose p ∈ T by applying the transformation T p (·) to the random samples. Therefore, we can check the admissability of a pose p by verifying that P (T p (x (n) 0 ) ∈ W free ) ≥ 1 − δ holds for all n = 1, . . . , N x , which directly defines the set of δ-safe poses

C.3.2 Algorithm
Before presenting the overall structure of our algorithm, some important functions are defined.
• Sample(T δ ): Returns a random sample p from the task space that satisfies P (x ∈ W free ) ≥ δ ∀x ∈ R(p). The condition is evaluated via sampling as described in Appendix C.3.1. • Nearest(G = (V, E), p): Returns the vertex in V that has the smallest Euclidean distance to p, i.e., Nearest(G = (V, E), p) := arg min p ∈V p − p 2 .
determines how finely the line segment is sampled. Thereby, i is increased until the end of the line segment is reached. • Near(G = (V, E), p, η): Returns all vertices p ∈ V that fulfill p − p 2 ≤ η.
With the defined functions, the general structure of our SCC-RRT* algorithm is described in Algorithm 1.

C.4 A Possible Approach for Enabling Online Adaptation: Scenario MPC
Due to the high computational complexity of the RRT* planner, solving the scenario-based planning problem can only be applied to static scenes at the moment. However, this limitation is not inherent to the scenario optimization problem (2) but mostly a result of the solution approach. Therefore, different options that have made it possible to plan in dynamically changing environments with deterministic information are intuitive candidates for solving (2) in non-static situations.
A very common approach for online planning is model predictive control (MPC) [107,108]. The idea of MPC is to only plan for a comparatively small time horizon, but to replan every time step. Thereby, the whole path is obtained online in a receding horizon fashion. Since only a short segment of the path is typically planned at each time step, the complexity of solving the optimal planning problem is typically low enough to allow online planning.
In order to apply MPC to the scenario path planning problem (2), several aspects need to be considered. Since MPC requires dynamics, simple dynamics of the form p k+1 = p k + u k can be defined, such that the opimization is performed with respect to u k instead of p k directly. Since the enlarged sets discussed in Appendix C.2 can become conservative when u k is allowed to be arbitrarily large, we can additionally consider constraints of the form u k ≤ū for some small constantū ∈ R + in scenario MPC. This effectively allows to trade-off conservatism due to discretization and computational complexity of the optimization problem. In order to incorporate chance constraints in (10), several scenario MPC approaches have been proposed and theoretically analyzed in recent years [109,110,111]. Scenario MPC has been applied to a variety of applications. While some of them  such as water resource management [112], building temperature control [113,114] or energy management in autonomous vehicles [115] require merely a small adaptation rate, other applications of scenario MPC such as autonomous driving [116] and collision avoidance with ships [117] and UAVs [118] can require much higher rates for replanning. Therefore, it can be expected that scenario MPC allows to solve (2) sufficiently fast for non-static environments. While scenario MPC is theoretically appealing, its practical implementation can be challenging. Therefore, realizing fast replanning by applying scenario MPC to the optimization problem (2) is subject to future work.

D Details on the Numerical Comparisons and Experimental Evaluations
D.1 Vision module

D.1.1 Dataset
Our goal is to determine the contour of a single hand lying flat with the fingers spread so precisely that a robot can be safely controlled to move into the spaces between the fingers. While there exist some datasets [64,65] for hand segmentation, they are mainly targeted for hand-based activity recognition. This leads to the problem that they contain many images with closed hands, but only very few with separately labeled fingers. Therefore, these datasets are not well-suited for applications which require the highly accurate segmentation of hands. Hence, we create a set of annotated samples ourselves from 100 images of the hands of three different individuals in the lab, referred to as A, B and C in the following. The ground-truth segmentation masks are created by manually circumscribing the hands' boundaries and automatically filling the resulting contour, which takes around five minutes per sample. We have attempted to speed up the procedure through the help of thresholding [119], but have been unsuccessful in doing so.
The set of annotated images is divided into two equally sized parts, one for training and one for testing. To challenge our augmentation procedure and segmentation model as much as possible, we try to introduce only little diversity into the training set, but much into the test set. Hence, the images in the training set are all from the lab with relatively even background (desk, floor), while the test set contains images with a wide variety of colors and backgrounds and also from outside. Furthermore, some test images include partially occluded hands to test robustness against object occlusions. We also put all images of person C into the test set to evaluate the effect of segmenting images of a previously "unseen" person.

D.1.2 Training
We set the dimensions of the first layer of the employed DeepLabv3 semantic segmentation model to H = 288 and W = 384 to balance resolution and computational demand. The training process is performed with the hyperparameters listed in Table 4 on an Intel Core i5-10500 CPU. Optimizing the model parameters via backpropagation takes around 4 s per batch. Therefore, it takes around 1 h 40 min to train a model with a dataset of 1000 images.

D.1.3 Evaluation Metrics for Semantic Segmentation
In the following, we present two popular metrics for evaluating the quality of a semantic segmentation result. We consider the case of two semantic classes, i.e. y ij ∈ {0, 1}, and denote the number of true positive, true negative, false positive, and false negative pixels, by TP, TN, FP, and FN, respectively. The simplest way to evaluate the quality of a segmentation mask is via the pixel accuracy Ground truth Predicted mask Figure 10: This simple example illustrates the differences between the evaluation metrics, pixel accuracy (PA) and mean intersection over the union (mIoU). The former is relatively high with PA = 81.3%, but the much lower value mIoU = 59.3% reflects the weak prediction better due to class imbalance.
(PA), which is given by For imbalanced classes, i.e., if the number of pixels belonging to the classes differs significantly, the pixel accuracy can be quite high even with poor predictions. This is illustrated by the simple example in Figure 10. There, the predicted mask for class 1 (white) deviates strongly from its ground truth, but the PA still has a relatively high value of PA = 13 16 = 81.3%. An alternative metric, the mean intersection over the union (mIoU), is better suited for cases of class imbalance. It is used for example in [7] as the main evaluation metric. For a bi-class problem, the mIoU is defined as Graphically speaking, for each class, the intersection of the ground truth and the predicted mask is divided by their union, and the mean of these values is calculated. For the example in Figure 10, we obtain mIoU = 1 2 2 2+2+1 + 11 11+1+2 = 59.3%, which is much lower than the PA and better captures the unsatisfying prediction.

D.1.4 Detailed Data Augmentation Evaluation
We evaluate our data augmentation scheme in comparison to the augmentation used in related work [18,19,20,22,23]. The data augmentation methods proposed in [18,19,20] are popular approaches for increasing the generalization ability and robustness of semantic segmentation models. These two aspects are especially important when working with a small, non-representative dataset, as in our work. In [22,23], two respectively three data augmentation methods are combined for training an ensemble of semantic models. For our comparison, we use each approach to create augmented datasets containing 100, 200 and 1000 images, respectively. Each dataset is used to train 20 models. Table 5 compares their performance on the test set in terms of PA and mIoU. In addition to their mean values, the standard deviation between the different models is also shown. Our augmentation scheme outperforms the other methods for every considered augmentation factor. Moreover, while the performance of some approaches decreases with increasing dataset size, our scheme yields better results for larger datasets. This demonstrates that our augmentation introduces sufficient variety for the models not to overfit to the training data even for 1000 images. We conclude that systematic data augmentation can enable the training of deep ensembles when only very limited data is available in contrast to the combination of merely one to three generic methods.
As shown in Figure 3, our dataset for hand segmentation already contains some variety in terms of flipping and rotation, which might raise concerns about the fairness of the comparison with the approach [22]. Therefore, we additionally evaluate all methods on the important task of detecting and removing beach litter with a robot, which also demonstrates the generalizability of our data augmentation scheme. For this, we use the dataset [120] consisting of 3500 annotated images, an example of which is shown in Figure 11. Similar to the hand segmentation task, we employ our augmentation scheme to create 1000 training images from only 50 labeled images. Thereby, we only make two minor adaptations compared to the hand segmentation case. Rotations are restricted to angles between −30 • and 30 • and color modification uses arbitrary colors. For comparison, we apply the data augmentation approaches [18,19,20,22,23] to also create datasets of 1000 images. The average performance obtained is shown in Table 6. The deviations in PA are relatively small across the different approaches at 0.8%. In terms of mIoU, our scheme and Mixup perform   Figure 11: An example image from the beach litter dataset [120] and the corresponding ground-truth mask for bi-class semantic segmentation.
superior to the other approaches. Note that the relatively low mIoU values compared to PA can be explained by the substantial class imbalance within the dataset, which is also evident in Figure  11. The evaluation on this second task demonstrates that our proposed data augmentation scheme is applicable to various robotics tasks. Moreover, the task highlights why semantic segmentation is often beneficial for motion planning. The robot must first identify the pieces of litter on camera images and subsequently plan a path to collect them. For this, it is critical to distinguish obstacles, e.g., wood or stones, from flat terrain where motion is possible, which can be realized with semantic segmentation. Put more general, the semantic class of an object often determines how it should be treated in motion planning. For example, the required safety distance to humans could be set larger than that to non-living objects. Therefore, combining the results of semantic segmentation with depth information allows for semantically-aware planning with uncertainties in 3D.

D.1.5 Detailed Uncertainty Quantification Evaluation
We employ a deep ensemble to improve the quantification of predictive uncertainty. In order to evaluate the resulting effect, we compute the reliability curve [33], which is a common way of measuring model calibration. For this, the confidence interval [0.5, 1) is partitioned into ten equally sized bins. For each bin, the prediction accuracy of the contained pixels is plotted over the average confidence value in the bin. We evaluate two ensemble sizes, M = 5 and M = 10, for small (200 images) and large (1000 images) data augmentation with our scheme and the approaches from [23,18,22,19,20]. The reliability curves are depicted in Fig. 12. While the ensemble size has an impact on calibration, the effect of data augmentation on calibration is also evident. For the considered non-representative labeled training set, the use of a deep ensemble alone is not sufficient to obtain well-calibrated uncertainty estimates. In comparison to the augmentation used in other works, only our augmentation scheme adds sufficient variety to the training set so that the interpretation of the softmax outputs as occupancy probabilities is justified. We also compute two common scores to quantify predictive uncertainty, the Brier score (BS) [67] and the negative log-likelihood (NLL) [10], which are depicted in Table 7. Our data augmentation scheme yields much better predictive uncertainty estimates than other approaches, including mixup, which is a popular method to increase calibration [19,25].  Flipping + 90 • Rotation [22] Cutout [18] Mixup [19,20] 200 images, M = 10 Flipping + Rotation + Cropping [23] Our scheme Perfectly calibrated In addition to the ensemble size M , data augmentation also has a large impact on uncertainty quantification. The best results are obtained with our augmentation scheme, 1000 images and ten ensemble members.

D.2.1 Simulation Setup
We compare our scenario-based planning approach with several state-of-the-art methods for path planning in uncertain environments [40,39,12]. For this, we create two scenes containing spherical and cuboidal obstacles. The more simple scene contains three larger obstacles, while the more cluttered scene contains eight smaller obstacles. Figure 13 depicts the two scenes. For the cuboids, the minimal spherical and ellipsoidal over-approximations used by [40] and [39,12], respectively, are also shown in the figure.
The task space T is composed of the x-, yand z-position of the end-effector and its orientation α around the z-axis. Hence, a pose can be expressed with four components p = [p 1 , p 2 , p 3 , p 4 ] T . The set R(p) occupied by the robot is modeled as an ellipse that is parallel to the x-y-plane, centered at the end-effector position x = [x, y, z] T and rotated by α around the zaxis. The line cost function c line of the SCC-RRT* algorithm consists of two additive terms, i.e., c line (p, p ) = c line,1 (p, p ) + c line,2 (p, p ). The first term is defined as c line,1 (p, p ) = area (T (p), p * , T (p )) + area (T (p), p * , T (p )) ,  Table 7: To evaluate uncertainty quantification, we compute the Brier score (BS) and negative log-likelihood (NLL) for different augmentation methods and ensemble sizes. Data augmentation and the ensemble size have a strong impact on uncertainty quantification. The impact of increasing the size of the training set depends on the approach used for data augmentation. where area( is the area of the triangle with vertices x 1 , x 2 , x 3 ∈ R 3 and p * ∈ W is defined as the point on the straight line between T (p init ) and T (p goal ) with smallest Euclidean distance to T (p). With (16), the area in the work space W between the path and the straight line connecting T (p init ) and T (p goal ) is penalized. The second term c line,2 (p, p ) = r(p 4 − p 4 ) 2 (17) with scalar weight r penalizes changes of the orientation around the z-axis. Hence, the term c line (p, p ) is zero if both p and p are on the line between T (p init ) and T (p goal ) and p 4 = p 4 .
The comparison methods are based on a different way of representing uncertainty than our approach. They make the very common assumption that the obstacle's geometry and orientation is known, while their center 3 ] T is uncertain and normally distributed according to 3 ] T and covariance matrix Σ. In order to compare these approaches to ours, we assume that the occupancy probability obtained from the visual perception system decreases linearly with increasing distance d to the obstacle's surface. Although this assumption may not seem intuitive, many predictions we obtain when perceiving real objects show an approximately linear decrease. We could just as well assume other profiles for the occupancy probability around the object boundaries, but choose this profile to enable a comparison to the other methods. The occupancy probability becomes zero at a distance d = d stop , which is equal for all obstacles. We set the chance constraint violation probability to δ = 0.05, i.e., all positions with an occupancy probability larger than 5% are considered unsafe. Hence, our method practically considers d = 0.95d stop =:d as the object boundary. The covariance matrix for the other methods is set to Σ = σ 2 I 3 , where σ is the standard deviation along each coordinate and I 3 ∈ R 3×3 is the identity matrix. From the cumulative distribution function of the normal distribution, we can derive the well-known expression Hence, for a desired maximum chance constraint violation probability of 5%, d = 2σ can similarly be considered as the safety distance to obstacles with center x o ∼ N (x o , σ 2 I 3 ). This serves as  Figure 14: We compare different uncertainty-aware collision avoidance strategies for path planning within the CC-RRT* framework. Our perception-based approach leads to less conservative paths, with the advantage becoming stronger for more complex environments.
a motivation and justification for comparing the different methods for the same value ofd and 2σ, respectively.

D.2.2 Detailed Simulation Results
For the two scenes, we determine the optimal paths using different method for a range of different values ofd/2σ 4 . The * algorithm is applied 100 times per method and value ofd/2σ, respectively. We set the number of search tree iterations to N iter = 2000. In our simulation, the success rate of finding an admissible path is 100% for the simple scene and 99.3% for the cluttered scene across the four methods. This can be explained by the large iteration number N iter and the large parameter values = 0.2 and η = 0.3 chosen in our implementation. Moreover, the initial and goal positions are set far enough away from the nearest obstacles so that even for the highest uncertainty considered, there is a region around the positions where motion is admissible. The average cost with one standard deviation for the different methods and scenes are shown in Fig. 14. Our approach yields the lowest average path cost for almost all considered scenarios. For the simple environment, the variations in path cost are relatively low for all approaches. This can be explained by the fact that the obtained paths simply go around the three obstacles. If the uncertainty increases, then the distance to the obstacles becomes larger, but the shape of the paths remains relatively similar between iterations. In contrast, the standard deviations are much larger for the cluttered environment. Here, there exist many different routes to the goal pose on which the robot navigates between the obstacles. In this complicated environment, our maximum number of iterations is not sufficient to find a path close to the optimum in every run. Hence, the paths obtained in different iterations differ significantly in terms of shape and cost, leading to a generally larger standard deviation. For the comparison methods, more gaps between the obstacles "close" when uncertainty gets higher than for our method. Hence, the relative standard deviation decreases for the other approaches while it stays approximately the same for our method. However, their performance significantly deteriorates in comparison to our proposed scenario approach, which illustrates the practical advantages of avoiding a parametric uncertainty representation.

D.3.1 Evaluation with KUKA iiwa manipulator
For evaluating the proposed method for safe planning based on visual perception, we perform an experiment with a KUKA LBR iiwa 14 R820 robotic manipulator, which is based on the popular knife game [45]. Therefore, the considered application follows in the long tradition of playing games for humans with robots, e.g., soccer [121,122], basketball [123,124], table tennis [125,126], juggling [127,128] and billiards [129,130]. For playing the knife game, a human places one hand on an even surface, while holding a knife in the other hand. In the basic version of the game, the human then tries to stab the knife into the space between the fingers as fast as possible. We play this game with a real human hand, but a robotic manipulator executing the motion. Due to obvious safety reasons, we replaced the knife by a pointy endeffector. Moreover, the cost function with summands (16) and (17) does not encourage stabbing into the surface in order to protect our hardware. For perceiving the hand, a Logitech C270 USB webcam is mounted 0.5 m above the table and takes an image of the scene. An ensemble of M = 5 DeepLabv3 models, trained with our augmentation scheme and 1000 images, performs probabilistic semantic segmentation. We augment the 2D information to 3D by assuming a fixed hand height of 2 cm.
With the obtained probabilistic occupancy information, we plan a δ-safe path connecting two points on the table on either side of the hand. We employ the end-effector depicted in Fig. 7 in the main paper, whose tip can be approximated as an ellipse with semi-axes 8 mm and 3 mm. The task space, robot geometry and cost function are defined as described in Section D.2.1. Modeling the end-effector as a 2D ellipse is justified as there are no overhangs in the scene. We run the SCC-RRT* algorithm for 2000 iterations to compute a δ-safe path, which takes around 10 s. For safe velocity scheduling, we set the maximum executable velocity tov = 0.2 m s . The tracking error bound function is given by a linear function γ(v) = v/v · 0.01m. Hence, the tracking error is no greater than γ(v) = 10 mm for any admissible velocity.
The desired trajectory p(t),ṗ(t) is used as an input to a lower-level Cartesian impedance controller. The stiffness parameters are set to K trans = 500 N /m and K rot = 100 Nm /rad for the translational and rotational stiffness, respectively. In that way, even if a contact between robot and human occurred, the resulting forces would not lead to injuries.
The experiment is repeated with five different scenes and five individuals, respectively. As shown in Fig. 15, the scenarios vary strongly in terms of background colors and lighting conditions. Due to our systematic data augmentation, the probabilistic semantic segmentation is robust to varying lighting conditions, backgrounds and skin tones. Due to employing the ensemble, confidence is lower in areas with high uncertainty. This is shown exemplarily in Figure 16, which visualizes the individual predictions of the ensemble members as well as their combination for the second right scene in Figure 15. Here, some ensemble members do not fully recognize two fingers. However, the combined prediction of the ensemble assigns these areas an occupancy probability sufficiently high such that they are not considered δ-safe. As a result, all trajectories that the robot executes in the experiment are safe. Fig. 17 depicts the desired and real trajectories for the end-effector position. The tracking error does never exceed the bound of 10 mm.

D.3.2 Extension to 3D Perception
While the experiments in Section 5.3 are performed using the assumption that the hand has a constant height and is at a fixed, known distance, this is an assumption that is not satisfied in many settings.  For example, cameras can be mounted on the end-effector of the manipulator or a mobile platform, such that distances are not known a priori. However, the assumption of known, fixed object height and distance is not required for our approach for visual perception based uncertainty-aware planning.
In order to see this, consider the case that a depth sensor, e.g., a depth camera, is available. By keeping the visual perception module unchanged, we merely have to synchronize the depth information with the result from semantic segmentation. When accuracy information about the depth data is available, e.g., standard deviation of measurement errors, this can be straightforwardly combined with the probability information of probabilistic semantic segmentation by considering, e.g., a Gaussian distribution. Therefore, the restriction to 2D data is not necessary in principle.
We demonstrate the straightforwardness of this approach using an Azure Kinect DK depth camera, which results in the 3D occupation information shown in Fig. 18. It can be clearly seen that the information obtained after synchronizing the result from semantic segmentation and the depth information is structurally very similar to the information with assumed hand height. Therefore, planning a safe path using the proposed SSC-RRT* approach can be directly applied leading to the path illustrated in Fig. 18. This clearly demonstrates the effectiveness of the proposed method for planning based on visual perception data, which is not limited to pure 2D image data.

E Extended Discussion of Limitations and Future Extensions
Despite its demonstrated applicability in real-world scenarios, our proposed framework for visual perception and motion planning framework currently has several limitations, which we aim to overcome in future work. Our approach can handle non-static scenes via regular replanning. Unfortunately, this is computationally expensive, such that it is not suited for fast moving objects or robotic systems with low computational resources such as mobile platforms or UAVs. In our experiment, we only employ a 2D webcam and provide depth information. A more accurate 3D representation of the scene can easily be obtained with suitable hardware, e.g., an RGB-D camera, and straightfor-wardly be used with semantic segmentation [131] , as demonstrated in Figure 18. We only consider a single obstacle, a human hand, in the scene. Detecting multiple objects can easily be performed with instance segmentation [104], e.g., using the Mask R-CNN model [132], but this requires a more involved labeling process. Moreover, although DL models can distinguish between more than 20 semantic classes in principle [7], the obstacles classes considered during training might not cover all possible objects, potentially causing collisions. In our experiments, we encountered such a situation, when the robot collided with scissors lying next to the hand shown in Fig. 15 on the right. Similarly, systematic data augmentation can only increase diversity in aspects that are considered during the augmentation design. Therefore, our visual perception approach puts a higher responsibility on its human designer compared to purely data driven approaches [23,5]. This requires detailed and thoughtful considerations on the topic of diversity. With our augmentation scheme, we can add missing information and variety to the dataset and therefore solve visual perception tasks even if only a small and non-representative dataset is available. However, no data augmentation is equivalent to the availability of a diverse annotated training set. Therefore, as much diversity as possible should already be introduced when creating the training dataset. This is especially important for tasks involving humans to avoid any bias in the perception system.
In the future, we plan to extend our approach to estimate obstacle dynamics and allow adaptive online replanning, e.g., by employing the path obtained from RRT* for a warm start scenario MPC implementation, such that it can be safely employed in highly dynamic environments. A promising approach could be to predict future semantic maps from past perception data, as proposed by [98]. This will most likely require a strong GPU due to the high computational demand, which has not been available in the current experiments. The computational cost of semantic segmentation could be decreased, e.g., by employing the recently proposed SegFormer framework [68], which we aim to investigate in future work. Another interesting direction for future work is to adapt certain aspects in motion planning, such as safety distance, to the predicted semantic labels. For example, the distance to fragile objects should be higher than to robust objects which would also withstand a slight collision in the worst case.