{'fraction_non_alphanumeric': 0.026891598463337232, 'fraction_numerical': 0.012582818328600857, 'mean_word_length': 4.469549330085262, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
3D Reconstruction of Multiple Objects by mmWave Radar on UAV

In this paper, we explore the feasibility of utilizing a mmWave radar sensor installed on a UAV to reconstruct the 3D shapes of multiple objects in a space. The UAV hovers at various locations in the space, and its onboard radar senor collects raw radar data via scanning the space with Synthetic Aperture Radar (SAR) operation. The radar data is sent to a deep neural network model, which outputs the point cloud reconstruction of the multiple objects in the space. We evaluate two different models. Model 1 is our recently proposed 3DRIMR/R2P model, and Model 2 is formed by adding a segmentation stage in the processing pipeline of Model 1. Our experiments have demonstrated that both models are promising in solving the multiple object reconstruction problem. We also show that Model 2, despite producing denser and smoother point clouds, can lead to higher reconstruction loss or even loss of objects. In addition, we find that both models are robust to the highly noisy radar data obtained by unstable SAR operation due to the instability or vibration of a small UAV hovering at its intended scanning point. Our exploratory study has shown a promising direction of applying mmWave radar sensing in 3D object reconstruction.


I. INTRODUCTION
Frequency Modulated Continuous Wave (FMCW) Millimeter Wave (mmWave) radar sensing recently has been shown as an effective sensing tool in low visibility environment, thus making it a promising sensing technique in autonomous vehicles [2] and search/rescue scenarios [3]. The capability of 3D object reconstruction is important in a search and rescue scenario, e.g., firefighting scenes, where heavy smoke makes optical sensing not practical. However it is quite challenging to reconstruct 3D object shapes based on mmWave radar data because the data is usually of low resolution, sparsity, specularity, and large noise due to multi-path effects. Recent work on 3D reconstruction has made some progress in this direction [1]- [4], focusing on single object reconstruction.
In this paper, we go one step further to explore the feasibility of reconstructing 3D shapes of multiple objects in a space (which is more challenging than single object reconstruction), based on mmWave radar data collected from a sensor mounted on a UAV. We let the UAV fly in the space and hover at various locations to scan/collect radar signals in the space. Then the UAV can obtain a collection of heatmaps or energy intensity maps of the space after FFT processing of the received raw radar signals. We take the heatmaps as input to a deep neural network model to generate the smooth and dense point clouds of the multiple objects in the space. We investigate two different deep neural network models for point cloud generation.
(1) Model 1 is our recently proposed 3DRIMR/R2P model [1]. It consists of two stages. In stage 1 it generates the 2D depth images of the space based radar signals. In stage 2 it generates 3D point clouds of those objects based on their depth images obtained from stage 1. Model 1 is used to reconstruct single objects in our previous work [1]. (2) Model 2 is formed by adding an image segmentation stage between stage 1 and stage 2 of Model 1. The segmentation stage separates the objects in the depth image representation of a scene (result of stage 1), so that they can be reconstructed separately in the next stage. The reason of introducing a segmentation stage is that working on single objects separately is easier than on multiple objects together when generating final point clouds.
In addition, the sensing/reconstruction system we consider only utilizes low-cost commodity mmWave radar sensors (e.g., [5]) which have low resolution. Therefore in order to obtain high resolution radar scan of a space, we mount a light-weight slider mechanism with radar sensor on a UAV so that the UAV can conduct Synthetic Aperture Radar (SAR) operation by sliding the sensor horizontally and vertically while hovering in the air. Due to the hovering instability of a small UAV, the data collected is highly noisy and the heatmaps of FFT processing is quite different from those of a stable precise SAR operation. Thus we are interested in whether such a vibrating UAV SAR operation can result in any meaningful reconstruction in practice.
Our major contributions are as follows. We demonstrate that it is promising to utilize a multi-stage deep neural network model to reconstruct multiple objects in a space based on mmWave radar data. In addition, we find that Model 1 has better performance than Model 2 even though Model 2 has an extra segmentation stage and can result in denser/smoother point clouds. This is because any segmentation error can cause cascading failures in the following reconstruction stage, hence making the model less robust and more error-prone. Furthermore, we find that both models are fairly robust to the highly distorted and noisy radar data collected by unstable SAR operation due to the vibration/instability of a commodity UAV when hovering. This finding shows that the inherent intricate characteristics of radar energy signature of a space is still retained in the imperfect SAR data, and shows that it is feasible to use low-cost small UAV in an environment sensing/reconstruction mission under low visibility.
In the rest of the paper, we briefly discuss related work in Section II. Then we discuss the two models in Section III, and our experiment results in Section IV. Finally the paper concludes in Section V.

II. RELATED WORK
Recently the application of mmWave radar sensing and imaging has been investigated in various areas [2]- [4], [6]- [9]. Different from person identification [6], [7] and 3D human mesh estimation [9], our work aims to reconstruct detailed 3D shapes of various objects in a space. Instead of focusing on a specific object (e.g., human body [7], [9]) we would like to develop a generic system that can reconstruct the 3D shape of any object. In addition, we choose to work on the raw radar energy heatmaps of a space instead of the point clouds generated by commodity radar sensors as we find them highly sparse and missing lots of information.
Our work is also closely related to a large body of literature in 3D reconstruction in computer vision [10]- [15]. In particular the design of neural networks used in our architecture is inspired by PointNet [13] and PCN [15]. Our work in this paper is developed based on our recent work on 3D reconstruction of a single object from mmWave radar signals [1], [4]. In addition, this work also deals with the input radar energy heatmaps of highly noisy SAR data due to the hovering instability of a small quadcopter UAV.

A. Overview
We investigate two different models to address the problem of 3D reconstruction of multiple objects based on mmWave radar data. Model 1 is from our recent work 3DRIMR/R2P [1], and Model 2 is formed by adding a segmentation stage into Model 1. The input to each model is a set of n mmWave radar intensity maps or heatmaps collected by an UAV's SAR operation while hovering at different locations surrounding a scene of interest. We will discuss the design details of these two models in the following.

B. Model 1
Model 1 is our recent work 3DRIMR [4] with R2P [1] as its second stage generator. For completeness we show the model in Fig. 1. It consists of two stages. In stage 1, generator network G r2i generates depth images based on the radar scans of a scene from multiple view points. Those depth images can be directly converted to a rough point cloud. In stage 2, the rough point cloud is processed by another generator network to produce final dense and smooth point clouds of the multiple objects in the scene. The training of the two generators are based on conditional Generative Adversarial Network (GAN) architecture. For details, see [1], [4].

C. Model 2
We add a segmentation stage into Model 1 to from Model 2. The idea is to separate the objects in the depth images (obtained from stage 1), and then feed those individual objects' depth images into the next stage to produce final 3D point clouds of individual objects, and then finally combine them back into the original space. This model's processing pipeline is illustrated in Fig. 2. Conditional GAN architecture is also used to train the model. 1) Stage 1: Each one of n mmWave radar intensity map captured from n view points is fed into generator G r2i , which can output the corresponding depth image from each viewpoint. 2) Stage 2: An image segmentation network (e.g., Pix2Pix [16]) takes those generated depth images as input, and then does semantic segmentation of each pixel. Based on the output annotation of each pixel, it can separate a depth image with multiple objects into m depth images, each containing only one object. 3) Stage 3: For each object, the model projects its corresponding n depth images from n viewpoints into a 3D space to form a coarse point cloud of the object. Then for each object, a G p2p network takes its coarse point cloud as input, and outputs an accurate, complete and smoother point cloud of this object. Finally, the model combines all point clouds of those m objects in the scene to get the final reconstructed point clouds of multiple objects.

D. Image Segmentation Network of Model 2
The segmentation stage of Model 2 is to label every pixel of a depth image as either a target object or background. We use the popular Pix2Pix [16] network, which is a image-toimage translation conditional generative adversarial network (cGAN), in the segmentation stage. The generator is a 2D encoder-decoder convolutional neural network (CNN), which can map a grayscale depth image into its annotated image. The discriminator is a simple 2D CNN, which can output a score to indicate whether the generator's output is good or not. Remarks. Since 3DRIMR/R2P [1] can reconstruct a single object well, we would like to explore its effectiveness of reconstructing multiple objects, thus we investigate Model 1. On the other hand, since image segmentation may be able to separate multiple objects from each other in a scene and hopefully reconstructing single objects might be easier, we come up with the design of Model 2. However, introducing the segmentation stage will inevitably add more uncertain errors in the formation of those coarse point clouds, and then those additional errors may make G p2p not be able to extract useful features and fail to reconstruct the 3D shapes of objects. For example, if some pixels of a small-sized object Each one of n mmWave radar heatmaps (each containing m objects) captured from n view points are fed into G r2i , which outputs the corresponding depth image of m objects. Then each depth image is processed by an image segmentation network which separates the depth image into m depth images and each of them contains only one object. Then the n depth images of the same object will be projected into 3D space and form a point cloud, which is then fed into G p2p to output an accurate, complete and smooth point cloud of the object. Finally, all m point clouds of those m objects in the scene will be combined together.
are annotated as background pixels during the segmentation stage, then the generated coarse point cloud may totally lose the shape characteristics of the object.

A. Datasets
Our experiments are mainly based on synthetic datasets, as collecting data via our current UAV platform is very time consuming. We first use OptiTrack [17] system to measure the deviation distances of a hovering UAV along x-, y-, and z-axis from its intended stable hovering position for SAR operation (scanning a space), and then based on the collected statistics we add noise (caused by hovering vibration) into the synthetic data generating process [2], [4]. We next show via an example that the radar data collected by a vibrating UAV's SAR operation visually looks quite different from the data of a normal stable SAR operation. Fig. 3 shows a scene where a car and a desk are placed, and Fig. 4 shows the analysis of the radar data of the scene. As shown in Fig. 4, a normal SAR operation gives cleaner and more distinctly clustered radar energy maps and FFT heatmaps (Fig. 4 (a) and (b)), when compared with the maps obtained from a vibrating UAV's SAR operation. Our experiments reported in this paper are all based on the vibrating UAV's SAR data.
In addition, we follow a procedure that is similar to 3DRIMR [4] to generate ground truth depth images and point clouds. To generate the ground truth annotated images that are used to train the segmentation network, we manually set the background pixels to black color, and the pixels of the same object to the same RGB color (except black color).

B. Model Training and Testing
The training and testing processes of G r2i and G p2p in Model 1 are the same as those in [1] except that in this  paper both the training and testing data are point clouds of multiple objects rather than individual ones. Similarly, the training processes of G r2i and G p2p in Model 2 are the same as those in [1]. As for the segmentation stage of Model 2, we use 2400 × 4 views, totally 9600 pairs of images to train G seg for 200 epochs, and the learning rate is 2 × 10 −4 for the first  Table I: Quantitative Results under different settings. We conduct our experiments on 2 different scenes: a scene containing 2 objects (a car and a desk) and a scene containing 3 objects (a car, a desk and a robot arm). The loss type CD or EMD in the parenthesis of method names indicates the distance functions used in training. For example, Model 1 (CD) means the Model 1 is used and CD is used as the distance functions during training.
100 epochs and linearly reduced to 0 for the rest 100 epochs.

C. Evaluation Results
In our experiments, G p2p takes an input point cloud with 1024 points and generates an output point cloud with 4096 points. If we combine m objects' generated point clouds of Model 2, the final output point clouds will contain 4096 × m points, which is m times denser than the output point clouds of Model 1. For the sake of fairness, when comparing the Chamfer Distance (CD) and Earth Mover's Distance (EMD) of both models, we randomly sample 4096 points from the output point clouds of Model 2 so that the output point clouds using these two methods have the same number of points. Then we calculate the two models' CD/EMD with the ground truth point clouds containing 4096 points respectively. Fig. 6 shows the visual comparison of the two models in reconstructing two objects. We see that both models can give reasonably good reconstruction performance visually. However, there might be extra points between two objects in the point clouds generated by Model 1. This is due to the fact that Model 1 attempts to reconstruct multiple objects together. We also see that the point clouds generated by Model 2 is smoother and denser. We have similar observations of the experiments with three objects. Fig. 6 also shows that both models are fairly robust to the highly distorted/noisy SAR data (Fig. 4) caused by unstable UAV hovering. Table I shows the test results of both Model 1 and Model 2 on two different scenes: one with two objects and the other one with three objects. We can see that the two models' test results of CDs are almost the same for the two scenes. However their performance are different in terms of EMD. For the 2-object scene, if the loss functions used in training are CDs, then the average test result EMD between generated point clouds and ground truth point clouds of Model 2 is around 1.7 times of Model 1. If the loss functions in training are EMDs, then the average test result EMD of Model 2 is more than 5.6 times larger than Model 1. The scene with 3 objects also shows the similar results.
The superior performance of Model 1 over Model 2 in terms of CD/EMD can be explained as follows. During the training of G p2p in Model 1, the goal is to reduce the losses (CD/EMD) of all m objects as a whole between the generated point clouds and the ground truth point clouds. On the other hand, the G p2p of each object in Model 2 is trained separately, so the network is only trained to reduce the losses (CD/EMD) of a single object. Therefore Model 1 will have much lower overall losses than Model 2 during test or inference. Furthermore, we notice that the segmentation network in Model 2 can introduce additional reconstruction errors. One example is shown in Fig.  5. Due to poor performance of G seg , the projected point cloud of the m objects will contain lots of errors ( Fig. 5(a)), which further causes G p2p 's failure. As shown in Fig. 5(c)-(d), the object with too many wrong points (desk in this example) is totally missing in the output point clouds, and the other object is wrongly reconstructed (e.g., the orientation of the reconstructed car is inversed).
However, Model 2 can generate denser point clouds than Model 1. Besides, since Model 2 reconstructs each object separately, there will be no extra points between objects in the final point cloud, a problem associated with Model 1. For example, as we can see in Fig. 6, the output point clouds of Model 1 (EMD) have some extra points between two objects.

V. CONCLUSIONS AND FUTURE WORK
We have shown via an exploratory study that it is feasible to utilize the mmWave radar data collected by a vibrating small UAV's unstable SAR operation to reconstruct 3D shapes of multiple objects in a space. We have studied two deep neural network models, and our experiments results have shown that they achieve promising results and they are robust to the vibrating UAV's SAR operation. For future work, we will develop novel modules into the architectures of these models to significantly improve their performance, and we will conduct large scale real-world experiments to develop an extensive dataset to test our models.