{'fraction_non_alphanumeric': 0.025676996981677507, 'fraction_numerical': 0.015006589295583047, 'mean_word_length': 4.853197312764369, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images

3D model generation from single 2D RGB images is a challenging and actively researched computer vision task. Various techniques using conventional network architectures have been proposed for the same. However, the body of research work is limited and there are various issues like using inefficient 3D representation formats, weak 3D model generation backbones, inability to generate dense point clouds, dependence of post-processing for generation of dense point clouds, and dependence on silhouettes in RGB images. In this paper, a novel 2D RGB image to point cloud conversion technique is proposed, which improves the state of art in the field due to its efficient, robust and simple model by using the concept of parallelization in network architecture. It not only uses the efficient and rich 3D representation of point clouds, but also uses a novel and robust point cloud generation backbone in order to address the prevalent issues. This involves using a single-encoder multiple-decoder deep network architecture wherein each decoder generates certain fixed viewpoints. This is followed by fusing all the viewpoints to generate a dense point cloud. Various experiments are conducted on the technique and its performance is compared with those of other state of the art techniques and impressive gains in performance are demonstrated. Code is available at https://github.com/mueedhafiz1982/


Introduction
Three-dimensional model generation from a single RGB image [4,6,29,37] has been around for some time and is quite a challenge in computer vision [13]. Its importance lies in the fact that this task represents one of the fundamental goals of computer vision (i.e., interpretation or understanding of scenic vision) [19]. Human vision system is an expert in the interpretation of stereo vison based images and the above mentioned task represents this aspect of the former [10,18]. In spite of the fact that some progress has been made recently using deep network based models and some largescale databases are available for research in this field, generation of fine 3D geometry for numerous categories of objects with different topologies is still challenging [9]. One popular 3D representation for tackling this challenge is the point cloud due to a number of important advantages [14,35]. For example, it conveniently represents varying geometry unlike a mesh. It does not have problems with cubic complexity unlike a 3D voxel-grid. Also, it can be used for shape reconstruction of implicit functions with the help of single evaluations in neural networks [21,26]. In spite of their success, 3D convolutional networks have the intrinsic drawback when they model a volumetric shape representation [22]. A 2D image has pixels with rich spatial and texture information which is not the case with a volumetric representation which has sparse information. Also when a voxel-grid is used to express a 3D object, a voxel which is outside or inside the object is not important and is not much useful. Alternately, the most useful data for 3D object representation is the surface-based data which is present scantily in the voxel occupancy grid. As a result, 3D convolutional networks are very wasteful computationally and memory-wise for prediction of data, because they use very complex 3D convolution mechanisms, which in turn have severe limitations on 3D volumetric shape granularity generated using high-end GPUs.
There are various existing techniques for 2D image to 3D model conversion having different strategies and different backbone networks. However, many of these techniques rely on inefficient and wasteful representative formats like meshes [23,36] and voxel-grids [5]. Some techniques have weak backbones which generate sparse point clouds [25]. This issue in turn necessiates complex and computationally expensive post-processing for generation of dense point clouds. Some techniques rely on silhouttes of objects found in RGB images [44] which is not how expert vision systems like that of humans work, the realization of the latter being the goal of this computer vision task. Some techniques use the groundtruth (GT) point clouds of the training data as co-input in addition to RGB images [24]. Again this procedure is unfound in the human vision system. In our opinion, staying on the main path is the best approach and any detours may be unsuccessful. The need of the hour is a 3D model generation system which addresses these issues. Accordingly, a 2D RGB image to point cloud conversion technique is proposed in this paper which uses point clouds for 3D representation, takes only 2D RGB images as input, has an efficient parallelized backbone network with slight computational overhead in comparison to single stream networks, does not have a post-processing pipeline,... etc. Further, this architecture can be extended to other computer vision tasks which can benefit from its advantages [12].
In this paper, a novel technique is proposed for 3D point cloud generation from single RGB images using a singleencoder multiple-decoder deep network. It is based on the parallelization concept [11] introduced into the backbone network of Lin et al. [22], which leads to impressive gains in performance. The proposed technique is capable of being trained on multiple categories of objects. The main focus of the proposed technique is to address the problem of 2D RGB image to point cloud conversion by introducing a new deep network architecture, which is efficient, robust, convenient to train, and is relatively convenient to implement. It has two stages; in the first stage, N viewpoint images are obtained from a single RGB image using the proposed network, while in the next stage, these images are fused to reconstruct the point cloud. Since the proposed technique uses 2D convolution, it conveniently achieves high resolution point clouds without having the issues of 3D convolution. Also, the proposed deep network does not use point clouds as co-input hence it is a 'pure' 2D image to point cloud conversion technique as is the human vision system which does not use point clouds and depends only on stereo images for visual environment understanding. Hence, it goes in the natural direction and does not suffer from additional complexity. Evaluation of the proposed technique has been done in different ways. Both single-category as well as multiple-category data have been used for training the model and testing. It is empirically demonstrated that the proposed technique generalizes better than other competing techniques. Additionally, the proposed technique does not need 2.5D or 3D data and does not have the issues related to processing such data. To summarize, the main contributions of the proposed technique are given below: -A new technique for generation of a 3D shape in the form of a point cloud from a single 2D RGB image using a two-stage technique with first generation of intermediate fixed coordinate images and second reprojection by fusion to form the point cloud based 3D representation of the object. -To the best of our knowledge, it is the first work to use a parallelization in point cloud generation backbones featuring single-encoder and multiple-decoder deep network for a computer vision task and demonstrates the efficacy of the model, which can be extended to other computer vision tasks. -Our technique reconstructs 3D representations better than previous techniques on datasets like ShapeNet [2] and impressive gains in performance are witnessed.
The rest of the paper is organized as follows. Section 2 discusses the related work regarding the focus of this paper (i.e., 3D model generation using single 2D image). Section 3 provides details of the proposed technique for generating 3D representations with fine grained point clouds. Section 4 presents the experimental results and evaluation analysis of the proposed technique. Finally, Section 5 concludes this paper.

Related work
With the development of deep learning models, single 2D image to 3D model generation has become an active area of research with substantial progress. Due to regularity, initial works chiefly used to learn the reconstruction of voxel grids using a 3D supervised technique [5] or a 2D supervised technique [32] with the help of a differentiable renderer [34,38]. In spite of this, these techniques are able to reconstruct low resolution shapes e.g. 32 or 64, because of the complex cubic nature of the voxel grids. Additionally, many techniques [15,33] have been developed for increasing the model resolution, however these techniques are too complicated to be followed. Mesh-based techniques [23,36] are an alternative for increasing the 3D model resolution. In spite of this, these techniques face difficulty in handling of arbitrary topologies because the vertex-based topology for the generated 3D shapes chiefly inherits from the template. Point cloud representation techniques [8,24,28,41] provide a viable means for single 2D RGB image to 3D model conversion. There have been issues with resolution. However the proposed technique is able to address these issues.
There are also techniques that map the texture of the object from the 2D image to the 3D representation using meshes [17] or point clouds [14,17,30,32,[41][42][43]. However, texture based 3D model generation research is still in early phase and has been applied to artificial images and is not capable of texture generation on real images e.g. like those found in databases like Pix3D [31] . They also have issues like complex pipelines, difficult training, need for large computational resources. Hence, instead of focusing on improving the crucial area of 3D model generation they currently tend to take the focus away from it.
Shape completion is another upcoming area in image to 3D model generation e.g. in view of occlusion [27]. It involves inferring the complete 3D geometrical model from partial observations. Various techniques use voxel grids [7] or point clouds [1,39,40] for shape completion with backbones like the PointNet architecture [3]. Although these techniques demonstrate decent shape completion, they have limited resolution. The body of work is small and the shape completion task can be considered as a post-processing in 2D image to 3D model conversion tasks which are not efficient enough.
We focus on the backbone of the above computer vision task for making the main task efficient, robust and simple by introducing parallelization in decoder portion. Of course, post processing techniques may be added to the 3D models generated by the proposed technique. One more important aspect of the proposed technique is that it falls within the category of 'pure' 2D image based 3D model generation techniques as found in works of Lin et al. [22], which do not depend on point cloud data as co-input. The human vision process also depends only on stereo-vision based captured images and is very efficient for real-time depth estimation. It should be noted that this novel architecture which is a first in a computer vision task, can be extended in other tasks.

The proposed technique
The aim of the proposed technique is generating 3D representations with fine grained point clouds. Inspired by the model used in Lin et al. [22], which consists of a conventional single-encoder single-decoder deep network, we propose a single-encoder multiple-decoder model as shown in Fig. 1. The 2D image is fed to the encoder that maps the former to a rich representative space. From the representative space, rich point clouds are generated using N structure generators based 2D convolution with a criterion for 2D projection. It should be noted that our technique is different than that in Lin et al. [22] because the latter uses a conventional architecture (single-encoder single-decoder network), while the proposed technique for the first time uses a singleencoder multiple-decoder network. Also, the computational overhead introduced due to parallelization is small.

N Structure generators
Each structure generator gives a 3D structure prediction of an object present in a 2D RGB image for a single viewpoint, i.e. 3D coordinates as inx i = [x iŷiẑi ] T for every pixel location. Values of pixels in 2D images may be created by generative architectures using convolution chiefly because of major dependencies in spatial domains. Such phenomena may be also shown by point clouds if they are treated as multi channel images in a 2D grid having coordinates in the form of (x, y, z). Due to this fact, each structure generator uses 2D convolution for prediction of images having (x, y, z) form for representation of 3D surfaces. The above approach prevents issues like heavy time consumption and need for heavy computation hardware for 3D convolution for predicting volumes.
Considering 3D matrices for transformation for the N viewpoints given by (R 1 , t 1 ) . . . (R N , t N ), every 3D point represented byx i at the n th viewpoint can be used for transformation to its 3D coordinate representation p i using Where K denotes a camera predefined matrix. This is the relationship definition among the generated 3D points and the fusion of all point clouds inside the 3D coordinates and it is the result of the proposed architecture.

Optimization of the 2D joint projection
Pseudo-rendering defined depth imagesẐ = f PR {x i } are used alongside masksM for different viewpoints in order to optimize the system. Loss is defined as a combination of mask loss L mask and depth loss L depth , with their respective definitions as Optimization is simultaneously done over different viewpoints. For the k th viewpoint, let M k and Z k represent the ground truth (GT) image and the depth image respectively. Loss L 1 is calculated for every depth element. Cross-entropy loss is used for the mask. The combined/holistic loss with the weight factor λ is represented as Optimizing each structure generator over its own n th projection followed by combining all of the projections leads to enforcement of combined 3D reasoning for geometry among the generated point clouds obtained from the N viewpoints. The same optimization also ensures uniform distribution of the error among different viewpoints as against concentrating on predefined N viewpoints. Algorithm 1 summarizes main steps of the proposed technique to generate 3D object point clouds.

Network architecture
The encoder and the decoder architectures are similar to those used in Lin et al. [22]. However, N identical decoders are used in parallel each feeding on the common feature map generated by the encoder. Feed image i to single encoder 5: Extract feature map f i from encoder 6: for j in range: (1, N) do 7: Feed feature map f i to decoder j 8: Extract K/N viewpoints from decoder j 9: end for 10: Fuse K viewpoints to form point cloud for image i 11: end while structure generator has a single encoder and multiple decoders, each using 2D convolutional layers with 3 × 3 kernels. All feature map dimensions are halved during each convolutional encoding and are doubled during each convolutional decoding. Details and dimensions of the encoder and decoder are listed in Table 1. Batch normalization layers [16] and ReLU layers are added between different layers of the network. Each decoder generates 8/N images of dimensions 128 × 128 × 4 (x, y, z, binary mask), where the unique viewpoints are those belong to 8 corners of a centered cube.

Experimental Results
The proposed technique is evaluated by studying its performance for single 2D image to 3D point cloud conversion followed by comparing it to state of the art techniques. The ShapeNet database [2] is mainly used because the latter has been used in many point cloud based techniques, and as explained earlier, point clouds have many advantages over other representation models like meshes and voxel-grids. Databases like Pix3D [31] have been scantily used and benchmarking is limited and as such it offers limited scope for pixel cloud based research. Also, techniques using Pix3D such 3D-LMNet and others (See [24,25]) sample limited number pixels from the surface of the ground truth (GT) models for subsequent point cloud representation and experimentation. Although some works like those given in [44] use Pix3D because of it contains real-world images, however they simultaneously depend on its image silhouettes (additional information) for 3D model generation, which is not an appropriate approach in our opinion because this is different from the natural process of 2D image to 3D model conversion which does not use silhouetting.
The popular ShapeNet database [2] has been used for training and evaluation of all models. It consists of a large number of 3D model categories (about 3 million shapes from online 3D model repositories), where semantic annotations for each 3D model is provided including geometric attributes, consistent rigid alignments, parts and bilateral symmetry planes, and physical sizes. For each model, we use random prerendering on 100 depth as well as mask images in pairs for a size of 128 × 128 for different viewpoints. The inputs to the proposed technique are object instances generated by prerendering with fixed elevation and with 24 unique azimuth angles. Fig. 2 shows sample images for four 3D aligned models found in the ShapeNet database.

Network training
Adam optimizer [20] is used for network training. Training has two stages viz. pretraining of the structure generator for prediction of N viewpoint depth images. A constant learning rate of 5e-3 is used for the proposed technique respectively for obtaining optimum results. A smaller constant learning rate is used as it was found empirically that multistream networks learn better using smaller learning rates as compared to their single stream counterparts. Subsequently, the full network is optimized by fine-tuning it on joint 2D projection. A constant learning rate of 5e-6 is used for the single-stream model. Also λ = 1.0 and K= 5 are used.

Evaluation metrics
The following metrics have been used for performance evaluation of the proposed technique. 3D Euclidean distance is measured for every point p i between source 3D model and target 3D model with the distance S to the target 3D model, where The 3D Euclidean distance metric is bidirectional and it is important to report both its forward and backward values as they represent different quality aspects viz. the forward value gives similarity of 3D shapes and the backward value gives surface coverage. The GT 3D models are represented by 100K densified point clouds. Chamfer distance(CD) between two point-setsX P and X P is a loss function, which is the nearest-neighbor distance metric and is given by Earth mover distance (EMD) between two point-setsX P and X P is given by where φ :X P → X P is the bijection. This definition enforces point-wise mapping among two sets and hence ensures uniform point predictions.

Single object category
The evaluation begins with that of point clouds on 3D generation for single object category viz. chair category of ShapeNet, consisting of 6,778 models. The proposed network is pretrained for 200K iterations and fine-tuned end-to-end for 100K iterations and 80% -20% training/testing split is considered. The 3D Euclidean distances (both forward and backward) for the single category experiment were measured for the proposed deep model with single-encoder, and 2-, 4-and 8-decoders. The proposed network variants with 2, 4 and 8 decoders generate 4, 2 and 1 viewpoints per decoder respectively. The related results are shown in Table 2. It is clear that as the parallelization increases (i.e. the number of decoders used in the proposed network increases), the performance of the latter increases. For the proposed single-encoder 8decoder network, both the metrics are better than that of [22]. Fig. 3 shows pre-training error plots for [22] and the proposed technique for first 100K iterations. As can be observed from the figure, our pre-training is much more efficient, and also with more parallelization (using 8 decoders instead of 4 decoders in the proposed approach) the pretraining efficiency increases further.
The best performing proposed network variant (i.e., singleencoder 8-decoder proposed model) is used for further experimentation. Next, performance comparison is done against Tatarchenko et al. [32], which uses mixed emdedding 3D representation, and Perspective Transformer Networks (PTN) proposed by Yan et al. [38], which uses projection error minimization for prediction of volumetric data. The chosen proposed network is pretrained for 200K iterations and finetuned end-to-end for 100K iterations. The quantitative results on the testing data are shown in Table 3 and in Table 4 using different metrics. It is observed from the results in these tables that the proposed technique outperforms all baselines in the metrics used. The results demonstrate that the proposed technique is capable of predicting more accurate 3D representations with higher density and better granularity. The results also show the efficacy of the proposed technique using 2D convolution based model as compared to other 3D convolution based models like PTN [32]. 2D convolution based architecture has been proved to be more efficient for geometric prediction using a suitable number of viewpoints and combination of the same using geometrical transform functions. Visualization is done for point clouds generated by the proposed model. The same is shown in Figure 4 and Figure 5. Compared to Lin et al. [22], the proposed technique predicts more accurate point clouds with comparable density.

General object categories
Evaluation is also done via single RGB image to 3D model conversion task using training on multiple ShapeNet categories. Our technique is compared against 3D-R2N2 [5] using recurrent network based volumetric prediction, Fan et al. [8] , and Lin et al. [22]. The results of this comparison are re-ported in Table 5. As it is clear, the proposed technique outperforms all other techniques by impressive margins with better predictions in most cases. Also, the CD and EMD metrics on the multiple-category experiments are listed in Table 6. The proposed technique outperforms others demonstrating its efficacy.

Conclusion
In this work, a new deep network architecture was proposed for generation of 3D point clouds which are powerful 3D model representation modelss. The 2D convolution based deep network aims to introduce parallelization by using a single-encoder multiple-decoder structure. The proposed net-work is tested on the popular ShapeNet database used for benchmarking of state of the art 2D RGB image to 3D model conversion techniques. The database has been used by many researchers. Compared to state of the art techniques, the proposed technique is experimentally proved to be more efficient and robust. The same is illustrated in the form of various tabular experimental results and graphical point cloud  comparisons. Our proposed network can be used in other computer vision tasks as it is simple, robust and efficient due to its parallelized architecture as found in expert image processing systems like those of humans. In the future, the parallelization concept will be investigated in other strategies for 3D model generation with more complex datasets.

Conflict of interest
The authors declare no conflict of interest.