{'fraction_non_alphanumeric': 0.0341416167975371, 'fraction_numerical': 0.012314504133660798, 'mean_word_length': 4.872435506804794, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Regularization Strategy for Point Cloud via Rigidly Mixed Sample

Data augmentation is an effective regularization strategy to alleviate the overfitting, which is an inherent drawback of the deep neural networks. However, data augmentation is rarely considered for point cloud processing despite many studies proposing various augmentation methods for image data. Actually, regularization is essential for point clouds since lack of generality is more likely to occur in point cloud due to small datasets. This paper proposes a Rigid Subset Mix (RSMix), a novel data augmentation method for point clouds that generates a virtual mixed sample by replacing part of the sample with shape-preserved subsets from another sample. RSMix preserves structural information of the point cloud sample by extracting subsets from each sample without deformation using a neighboring function. The neighboring function was carefully designed considering unique properties of point cloud, unordered structure and non-grid. Experiments verified that RSMix successfully regularized the deep neural networks with remarkable improvement for shape classification. We also analyzed various combinations of data augmentations including RSMix with single and multi-view evaluations, based on abundant ablation studies.


Introduction
Deep neural networks have achieved outstanding performances in various fields regardless of the data domains, such as image, video, speech, and point cloud. In particular, three-dimensional (3D) point cloud processing is attracting considerable interest following the pioneering network PointNet [20] development, since point clouds can be applied directly to deep learning without preprocessing. Although various tasks have been successfully addressed using point clouds, such as 3D object shape classification and part segmentation, inherent drawback of deep learning is still less considered in the point cloud domain. Due to the typical nature of deep neural networks (DNNs) that ap- proximates the model from the given data distribution, the trained model tends to be overfitted regardless of the data domain. This lack of generality is a fundamental deep learning problem. One way to alleviate overfitting and generalize the model is data augmentation, which improves diversity of the training data.
Various data augmentation methods have been recently proposed in the image domain as network regularization strategies, but data augmentation for point clouds has only rarely been considered. Actually, regularization is essential for point clouds since it is easier to be biased to the distribution of training samples than that of image. That is largely because point cloud datasets [29,2,6] are typically considerably smaller and less diverse than image datasets, such as ImageNet [7] and MSCOCO [18], which have millions of training data. For example, ModelNet40 [29], one of the most widely used point cloud dataset, includes only 12,311 models with 40 categories. Therefore, it is essential to improve the generality of models for point cloud.
In the image domain, regional dropout [36,23,9,8] and mixup-based methods [32,26,31,13,10] have been proposed as data augmentation strategies, which are different to conventional methods, to generate virtual training samples. These methods are designed to improve generality of the neural network, preventing the model from being significantly affected by only small part of the sample that has discriminative characteristics by eliminating or mixing the part of the data. However, it is difficult to apply this intuition directly to the point clouds due to two inherent properties of point cloud: non-grid and order-invariance. Although Chen et al. [4] applied the concept of Mixup [32] to point clouds handling the properties of point cloud with linear interpolation based on optimal assignment, generated samples lost the structural information of the original sample due to distortion. This paper proposes Rigid Subset Mix (RSMix), the shape-preserving data augmentation method for point clouds that can partially mix two samples, preserving the partial shapes of the original samples. We redefine the concept of mask region from image analysis and adapt it to 3D space to extract parts from each sample while preserving structural information of the point cloud. We also define a Rigid Subset (RS) derived from the redefined mask region, a group of adjacent points within a certain distance from a specific query point using a neighboring function to address unique characteristics: unordered structure and non-grid. In contrast to PointMixup [4], we can utilize structural information of the original point cloud sample intactly by using RS. In addition, we designed RS scale to vary, to improve diversity of the training sample, and hence increase regularization effects. Furthermore, RSMix can be used in conjunction with the existing data augmentation since it utilizes the part of the given data intactly. In the end, by introducing RS, we can improve generality of DNNs and give attention them to recognize parts of the object. In Section 3.2, we describe in detail how to generate the virtual sample preserving shape of the source sample by extracting RS. In advance, we provide visualized RS samples to be extracted and resultant mixed samples in Figure 1.
We provide the experimental results for shape classification on ModelNet40 [29] and ModelNet10 [29] with the most representative DNN approaches [21,27] for point clouds. RSMix successfully improved the network performance, outperforming the existing data augmentation methods. Moreover, abundant ablation studies for various combinations of existing data augmentation and RSMix verified that RSMix improved the model regardless of which conventional data augmentation method was employed.
Meanwhile, we analyzed the experimental results with respect to two evaluation mechanisms to ensure fair com-parisons. In fact, although the evaluation methods of shape classification on point cloud are divided into two ways: single and multi-view, many studies present their experimental results without clearly specifying their mechanism. This makes hard to quantitatively compare results among studies. Our experiments show that the results evaluated by single and multi-view approaches have significant differences. Therefore, it is essential to analyze experimental results along the evaluation methods. Sections 4.2 presents analysis with single and multi-view approaches based on ablation studies.
To summarize, this paper provides the following major contributions.
• Shape preserving augmentation. We propose new data augmentation method for point clouds that mixes training samples with preserved structures by using Rigid Subset (RS).
• Significant improvement. The proposed method remarkably improves DNN performances and robustness for shape classification and outperforms existing data augmentation strategies.
• Complementary method. RSMix can be used in conjunction with other data augmentation approaches. Abundant ablation studies verify that RSMix can be combined well with other augmentations and further improves the target model.

Related Work
Data Augmentation for Images. Data augmentation is a regularization methods that expands the knowledge range that can be learned from training data by transforming data while retaining the essential sample meaning. Thus, the model becomes less dependent on the specific given data. Various methods have been proposed in the image domain in addition to conventional methods, such as random rotation, flip and crop. Some works have enabled the model to learn spatially distributed representation by removing the part of the data on pixel [36,23,8] or feature map [9] basis. Furthermore, several mixup-based methods [32,26,31,13,10] have been proposed that generate virtual samples by combining the two samples. Mixup [32] generates virtual training samples by linearly interpolating two images and defining the mixed area ratio as a corresponding label. By introducing the combination between data, Mixup brings out the regularization effect and shows improved performance for several tasks. After Mixup, Verma et al. [26] extended Mixup by applying the concept to the feature map, and Yun et al. [31] fusioned the concept of [32] and [8] to improve localization and classification ability of the model. In addition, Kim et al. [13] and Harris et al. [10] utilized saliency maps [13] and Fourier Mixup [32] (1 − λ)x α + λx β Manifold Mixup [26] (1 − λ)f (x α ) + λf (x β ) CutMix [31] (1 − M) x α + M x β Puzzle Mix [13] ( transform [10], respectively, to use semantically representative parts of the data when generating virtual samples. However, these approaches can only be applied to image based models rather than point clouds, because they have different data structures. Therefore, we propose the RSMix, novel mixup-based augmentation strategy for the point cloud that generates virtual samples considering the unique properties of point clouds . Point Cloud Structural Properties. In contrast with images, point clouds have 3D coordinate information, including implicit geometric feature, which is essential to understand them. Due to the unique properties of point cloud: non-grid and unordered structures, it is difficult to extract the local and geometric feature from point clouds. Various networks have been proposed with different structures, such as point-wise multi layer perceptron [21,12,35,30], convolution [19,24,28,1,11,15,17,14], and graph [27,33,22,34] based networks, to extract local and geometric features. For example, [21,24,27] extract local and geometric features by applying point-wise grouping, radiusbased kernel, and graph-structure, respectively. All these networks demonstrated that considering structural information is significantly important for the DNN model to understand point clouds. Therefore, regardless of how data augmentation occurs, structural information of point cloud should be regarded as core characteristics since it is a critical component for the model. Data Augmentation on Point Cloud. Data augmentation has not been extensively explored in the point cloud domain, aside from general conventional methods, such as randomly scaling, rotation, and jittering. Few studies [16,5,4] dealt with data augmentation in the point cloud domain. Liu et al. [16] proposed auto-augmentation network for point clouds to find an optimal combination of conventional data augmentation methods corresponding to each sample. Choi et al. [5] divided the sample into specific partitions and transformed or mixed each part independently. However, there is a limit to diversity of virtual sample because mixing is performed on inter-classes and specified grids are used for partitioning. PointMixup [4], which is the closest method to our proposed approach, extends the concept of Mixup [32] to point clouds through linear interpolation with optimal assignments between two samples. However, the generated samples have distorted structures which lead to the loss of structural information. Structural information is a core point cloud feature since they have no textural information. Therefore, we propose a more general data augmentation method for point clouds that can preserve shape of the original data .

Preliminary
Neural networks aim to model function f that describes the true distribution P for given data where samples x ∈ X have corresponding labels y ∈ Y. It has been proved through Empirical Risk Minimization [25] that f can be approximated by minimizing empirical risk R ξ (f ) of the model by computational optimization using loss L and empirical distribution P ξ for given data distribution as (1) In data augmentation, P ξ can be expanded to P ψ with additional augmented data through vicinal risk minimization [3], where ψ is a vicinity distribution, i.e., the probability that virtual sample and label pair (x,ỹ), are sampled from the vicinity of given sample and label pair (x i , y i ). For image data, Zhang et al. [32] designed a vicinal distribution ψ that generated a virtual mixed sample-label pair (x,ỹ) from two paired data (x α , y α ) and (x β , y β ) using mix function f M d , for sample and f M l , for label, as where λ ∼ beta distribution Beta(θ, θ), for θ ∈ (0, ∞). Table 1 shows the deformations of f M d in various ways using features from model f [26] or masking approaches such as binary mask M [31], salient data included mask z [13], or thresholding mask H [10] with filtered data G in the frequency domain. However, these mask-based approaches cannot be applied directly to point cloud, since point clouds have no grid and points can exist anywhere in 3D real space. Though Chen et al. [4] solved this problem by linear interpolation between two point clouds, introducing optimal assignment J φ * , they could not generate virtual samples preserving shape of the original sample. Therefore, our goal is to generate a shape-preserved virtual sample that has combined information from both samples as well as proposing an adapted spatial mask for 3D data. We are inspired by Figure 2: Overall pipeline of RSMix. Three steps to synthesize the mixed samples(P α mix ) using Rigid Subset (RS).
concept of the mask region from image analysis, which preserves the part of the original data intactly.
Mask as Region of Neighboring Data. CutMix [31] defined the mix function as where represents the element-wise multiplication; mask M denotes a d u × d w binary rectangular region represented is a randomly chosen pixel of the image. To utilize inherent definition of the mask region, we redefine mask M as a group of successive adjacent pixels within distances d u /2 and d w /2 in image. Hence, mask M can be denoted as where (u i , w j ) is the (i, j)th pixel for the given image; i = {1, 2, ..., W }; j = {1, 2, ..., H}; and (u c , w c ) is the center of the mask (u M + du 2 , w M + dw 2 ). Thus, the mask can be regarded as an adjacent group of data from a particular point (u c , w c ). We adapt this definition of mask to point clouds.

Rigid Subset Mix
Rigid Subset Mix (RSMix) mixes parts of two point cloud samples by extracting the Rigid Subset (RS), which preserves each samples's shape. RSMix is divided in three steps: neighboring, extraction, and insertion. First, we utilize the redefined mask region concept from Section 3.1 with neighboring function A to prevent deformation of original data. Then we extract RSs from each sample to mix the samples. Finally, we mix two RSs in the insertion step. Figure 2 shows the RSMix algorithm pipeline. Neighboring Rigid Subset. We define two n sampled point sets normalized in the unit sphere as P t = {p t i | i = 1, 2, ..., n}, where t∈ {α, β}. p t i ∈ R 3 is its Euclidean coordinates, which represents location of the point. We only consider coordinate information since RSMix operates on point-wise coordinates.
We adapt the regional mask for image to spatial subset of each point sets from given point sets P α and P β , by grouping adjacent points from a certain query point, q t , randomly chosen from P t . These subsets are denoted as S α and S β according to below Equation (6).
which are grouped using the specific neighboring function A. We define these subsets as Rigid Subset (RS) since they preserve the sample shape rigidly. We introduce two instantiations for A to retain the original point set shape: K-Nearest Neighbor(KNN) to given q t and Ball-query method that neighboring points in certain distance r rigid from q t as respectively, where r rigid is sampled from beta distribution Beta(θ, θ), with parameter θ = 1.0 as default, i.e., the uniform distribution, since P t is normalized in the unit sphere. Both A are based on Euclidean distance in 3D space considering the point cloud's unordered structure and free space around them. Each method has different characteristics on neighboring subsets with respect to the density or directional bias of given point sets.
Meanwhile, we limit |S β | ≤ n max , where n max and | · | denote the upper bound number of points in RS and cardinality for the point set, respectively. We usually set n max = |P t |/2 to preserve at least half of the original point sets. In addition, when using A ball , we randomly sample points in S β along the difference between |S α | and |S β | to maintain the |P α mix |, where P α mix denotes a mixed sample described in below Insertion part. We compare and analyze the two methods for quantitative and visualized results in Section 4.2. Further, we also provide experiments with various θ values in Section 4.2. Figure 2(a) shows neighboring the RS from the each sample. Extraction. Neighbored RSs, S α and S β , are used to a generate mixture sample P α mix . To mix two samples, we remove the S α from P α and replace the empty space with S β . Hence, extracted RSs from each point cloud sample to generate mixture samples are denoted as P α − S α and S β as shown in Figure 2(b). Insertion. However, q α and q β are usually different because they are randomly chosen from P α and P β , respectively. Hence, before insertion, S β should be translated by the difference between the two query points. We introduce translation function T β→α to translate S β by q α − q β as where p β is a point in S β . Applying T β→α to S β , the translated subset S β→α is denoted as Therefore, mixed sample P α mix is defined as and Figure 2(c) describes the inserted mixture sample.
Thus, mix function f M d (x α , x β ) for RSMix can be expressed using as follow Equation (11) using P α and P β instead of x α and x β .

Mixture Ratio λ for Training
In this Section, we define the mixture ratio λ, the ratio of |S β→α | w.r.t. |P α mix |, to train the network for shape classification. In contrast to A knn or some previous image masks, |P α −S α | and |S β | are often different when using the A ball , since we apply same r rigid to P α and P β , despite of their different densities. Hence, we define To explicitly consider the relation between |P α − S α | and |S β |. Finally, we define the label mix function as which is same as in CutMix [31], to generate virtual label y for classification training. Detailed implementations are available in the supplementary material with pseudo-code.

Experiments
Datasets. We evaluate RSMix on ModelNet40 [29] and ModelNet10 [29], which are widely used point cloud classification benchmark datasets. ModelNet40 is small dataset which comprises 12,311 CAD models from 40 man-made object categories, and ModelNet10 is subset of ModelNet40 that includes only 4899 CAD models from 10 categories. We utilized the preprocessed data provided by PointNet [20] for ModelNet40 with same train-test split, which were 1024 uniformly sampled points on mesh faces according to face area and normalized onto the unit sphere, and preprocessed ModelNet10 similarly. In particular, we ignored normals of samples since they are not available for real-data. Backbone Networks. We considered three representative point-wise DNNs for point cloud: PointNet++ [21], DGCNN [27], and PointNet [20] as our backbone network architecture. We applied RSMix to several neural networks to emphasize RSMix is model agnostic.
Single and Multi-view Evaluations. Single and multiview evaluations are separated depending on whether objects were evaluated from different angles or not. These approaches can be separated into two cases: with or without voting strategy to predict an object multiple times by rotating about an axis. The experiments adopted voting strategy of evaluating an object 12 times, rotating it 30 • on its vertical (y) axis between evaluations. Meanwhile, Mod-elNet40 [29] has 10 classes with aligned poses/headings. Thus, it is trivial to separate the 10 classes with the remaining 30 classes if we don't do random rotation on test samples when evaluating the model. Hence, there are obvious differences between the results from single and multiview evaluations. Appropriate combinations of augmentation strategies also vary depending on evaluation type. We investigated results from both evaluation strategies and explored optimal combinations of augmentations for different models by abundant ablation studies (Sections 4.2). Implementation details. We implemented RSMix using PointNet++ [21] and DGCNN [27] with conventional data augmentation, ConvDA (comprising jittering(σ 2 =0.01); scaling(0.8∼1.25); rotation along the y-axis i.e., gravity axis; and shifting (range=0.1) for the training dataset. Further details are included in the supplementary material.

Shape Classification
Evaluations. We evaluate RSMix for shape classification using three backbone networks on ModelNet40 and Mod-elNet10. All experiments were implemented using official codes and results are shown in Table 2. "Multi" indicates the evaluation with multi-view in Table 2. To ensure fair comparison given the rotational bias in ModelNet40, results related to point-wise networks [20,21] in Table 2    plied. Section 4.2 presents a rotation-related ablation study.
All the results reveal that RSMix improved the network accuracies regardless of network type or evaluation methods, verifying the effectiveness of our shape-preserved mixture approach with significant improvements for Point-Net++ [21] and DGCNN [27], which encode local or geometric features of object through hierarchical grouping or graph structure, respectively. Comparison against PointMixup [4]. We demonstrate results of RSMix for two evaluation methods against Point-Mixup [4], the closest work to us, in Table 3. We compared pre-aligned and unaligned settings to for single and multi-view accuracies, respectively, since PointMixup [4] do not specify their evaluation method but each are similar. They follow the PointCNN [17] setting discriminating pre-aligned and unaligned with horizontal rotation on point cloud samples. They randomly rotate the training point cloud along the y-axis for unaligned settings. For natural evaluation, we do not preprocess the dataset as pre-aligned or unaligned (denoted as Raw in Table 3). RSMix achieves more competitive performance than PointMixup [4] for networks that use local information [21,27], and further enhances the network's ability to recognize local information.

Ablation Study
Neighboring Function. This Section describes the two types of neighboring function, A ball and A knn , employed to extract RSs from point cloud samples. Figure 3 Table 5: Ablation studies on evaluation accuracy with single(ACC S (%)) and multi-view(ACC M (%)) for Point-Net++ [21] on ModelNet40.
serving shape of the sample based on Euclidean distance d knn , it is prone to be overlapped with the other extracted part, e.g. Figure 3(b), if there is directional bias on the sample. On the other hand, A ball alleviates overlapping by selecting points within the distance, r rigid e.g. Figure 3(a). However, in contrast with A knn , if the density of sample around the query point is high or r rigid is too large, the number of points in RS from A ball must be controlled to maintain the number of points in a mixed sample. Therefore, we randomly sample the extracted points in S β to preserve the overall shape of the extracted part. Figure 4 illustrates the difference between processed RS with A ball and A knn depending on density of point cloud. We also compare two methods with quantitative results of single-view evaluation on DGCNN [27] in Table 4, using scaling and shift augmentations as ConvDA. Both approaches achieved improved results over existing networks and A ball achieves superior results since it is more robust to directional bias as well as considers the density of point cloud. Therefore, we used A ball as our main neighboring function. Single and multi-view evaluations with various combinations of augmentations. RSMix can be applied in conjunction with existing ConvDA methods to further increase the diversity of mixed data since they are independent approaches. However, some combinations of augmentations can cause excessive deformation on the data sample, reducing the network's ability to recognize objects itself. Therefore, it is essential to analyze various combinations of augmentation strategies. We provide quantitative results in Table 5 with single and multi-view evaluations for Point-Net++ [21] on ModelNet40. RandDrop is the data augmentation method proposed in [21] that randomly drops the points from sample so that network can extract the global features better. The results show that models with RSMix alone achieved the highest accuracy for single-view evaluation. In addition, overall experiments show better results  In addition, we also provide the results for DGCNN [27] in Table 6 with single-view evaluation on ModelNet40 and ModelNet10 with scaling augmentation as ConvDA, since single-view evaluation shows better results without rotational augmentation. We also obtained remarkable improvements with RSMix for all presented combinations. Therefore, we can notice three things as follows.
• Single and multi-view evaluation performances differ significantly depending on the presence of rotational biases in the dataset.
• Rotational augmentation reduces single-view evaluation performance, but must be included when training if evaluation is performed with multi-view evaluation.
• RSMix successfully improved model generality by appropriate combination with other augmentation strategies regardless of evaluation type.
Robustness Test. We tested the robustness of RSMix with PointNet++ [21] to four noisy environments: jitter, rotation, scaling, and DropPoint, in order to verify that our method makes the model robust to noise.  cially, multi-view evaluation for rotational noisy environment along the y-axis was performed by rotating the sample along the x-axis for fair comparison. ConvDA includes jittering, shifting, scaling, and rotational augmentations with default settings as same as PointNet++ [21]. The results in Table 7 reveal that RSMix improves the robustness of model whether or not ConvDA was applied for rotation and Drop-Point noise, since shape and scale of original point cloud were preserved. However, we achieved lower results with multi-view evaluation when only RSMix was applied for scaling noise. The reason is that if scaling noise is large, it is difficult for the model to interpret the data when viewed from different angles since shape is preserved but scale compared with the original data. Meanwhile, results differed greatly depending on the level of noise for jittering noise, where the shape of an object was not preserved. Although RSMix alone cannot improve robustness for multiview evaluation, RSMix provided improvements when jitter noise was small for single-view evaluation regardless of ConvDA usage. However, robustness was reduced for both evaluation methods for large jitter noise when RSMix was applied because it was difficult for subsets extracted from each sample by RSMix to have sufficient shape information since the original sample shape was greatly distorted prior to applying RSMix.  Various θ values. We introduced beta function Beta(θ, θ), to sample r rigid from the beta distribution when using A ball in Section 3.2. We demonstrate the experimental results for various θ for DGCNN [27] on ModelNet40 [29] in Figure 5 to figure out the effect of θ to our model using same experimental settings. Due to a property of beta function, larger θ implies higher probability that r rigid was sampled close to 0.5. However, since we set the n max = half of the number of points in sample, more frequent sampling of large r rigid does not have much effect. Best accuracy was achieved for θ = 1.0 for all cases. Therefore, we set θ = 1.0 as default for all experiments.

Conclusion
This paper proposes RSMix, a novel data augmentation method for point clouds, that generates virtual mixed samples from extracted subsets from each point cloud without additional learnable parameters. We extracted the subsets from samples without shape distortion by redefining the rectangular mask for images as a subset of adjacent points from a query point in 3D space. Various experiments verified that RSMix improved deep neural networks to extract discriminative feature effectively by increasing diversity of data. In addition, extensive tests demonstrated that RSMix also improved robustness of the model to various types of noise. We further analyze the two types of evaluation method for shape classification: single and multi-view, which are utilized as evaluation metrics for the overall experiments. Experiments verified explicit differences between two methods and necessity of selecting appropriate combination with various data augmentation strategies. Extensive ablation studies also verified generic effectiveness of RSMix with various combinations with existing data augmentations.