{'fraction_non_alphanumeric': 0.043869338803850404, 'fraction_numerical': 0.01507022250276156, 'mean_word_length': 4.489389346037245, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data Interpolating Prediction: Alternative Interpretation of Mixup

Data augmentation by mixing samples, such as Mixup, has widely been used typically for classification tasks. However, this strategy is not always effective due to the gap between augmented samples for training and original samples for testing. This gap may prevent a classifier from learning the optimal decision boundary and increase the generalization error. To overcome this problem, we propose an alternative framework called Data Interpolating Prediction (DIP). Unlike common data augmentations, we encapsulate the sample-mixing process in the hypothesis class of a classifier so that train and test samples are treated equally. We derive the generalization bound and show that DIP helps to reduce the original Rademacher complexity. Also, we empirically demonstrate that DIP can outperform existing Mixup.


Introduction
Data augmentation [Simard et al., 1998] has played an important role in training deep neural networks for the purpose of preventing overfitting and improving the generalization performance. Recently, sample-mixed data augmentation [Zhang et al., 2018, Tokozume et al., 2018a,b, Verma et al., 2018, Guo et al., 2019, Inoue, 2018 has attracted attention, where we combine two samples linearly to generate augmented samples. The effectiveness of this approach is shown especially for image classification and sound recognition tasks.
Many traditional data augmentations (e.g., slight deformations for image data [Taylor and Nitschke, 2017]) rely on the specific properties of the target domain such as invariances to some transformations. On the other hand, sample-mixed augmentation can be applied to any dataset due to its simplicity. However, its effectiveness depends on the specified data structure. There is basically a difference between original clean samples and augmented samples in that augmented samples are not drawn from an underlying distribution directly. Thus a classifier trained with sample-mix augmentation may learn the biased decision boundary. In fact, we can easily create a distribution where sample-mix deteriorates the classification performance (See Fig. 1).
To overcome this problem, we propose a alternative framework called Data Interpolating Prediction (DIP), where the sample-mixing process is encapsulated in a classifier. More specifically, we consider sample-mix as a stochastic perturbation in a function and obtain the prediction by computing the expected value over the random variable. Note that we apply sample-mix to both train and test samples in our framework. This procedure is similar to existing studies such as monte carlo dropout [Gal and Ghahramani, 2016] and Augmented PAttern Classification [Sato et al., 2015]. Furthermore, Figure 1: Test accuracy and visualization of decision area on 2d spirals data. The neural networks are trained either (Left) without sample-mix, (Center) with the existing sample-mix method, or (Right) with the proposed sample-mix training + output approximation with monte carlo sampling. Although the standard sample-mix training deteriorates classification performance due to the biased decision boundary caused by sample-mix training, our method mitigates this problem. See section 2 for the details of a hyper-parameter of beta distribution α and the number of sampling S.
we derive the generalization error bound for our algorithm via Rademacher complexity and find that sample-mix helps to reduce the Rademacher complexity of a hypothesis class. Through experiments on benchmark image datasets, we confirm the generalization gap can be reduced by sample-mix and demonstrate the effectiveness of the proposed method.
2 Proposed Method

Data Interpolating Prediction
Let X ⊂ R d be a d-dimensional input space and Y ⊂ {0, 1} K be a K-dimensional one-hot label space. Denote a classifier by f : R d → R K . The standard goal of classification problems is to obtain a classifier that minimizes the classification risk defined as: where p(x, y) is the joint density of an underlying distribution and is a loss function. Here, we consider a function where the sample-mixing process is encapsulated as a random variable. We describe sample-mix between x and x with a function ψ : where λ ∈ [0, 1] is a parameter that controls mixing ratio between two samples.
, and h be a specified classifier. Using the function ψ, we redefine the deterministic function f : where p(λ) is some density function over [0,1]. Note that the function f is equivalent to the base function h when we set p(λ) = δ(λ = 1).

Practical Optimization
Since the expected value E λ∼p(λ),x ∼p(x ) [·] is usually intractable, we train the classifier by minimizing upper the bound of an empirical version of L(f ), By applying Jensen's inequality, we have where S is a positive integer which represents the number of sampling to estimate the expectation. We denote the RHS in equation 5 by L upper,S (f ). The tightness of the above bound is related to the value of S as We can prove this in a similar manner to Burda et al. [2016]. Since lim S→∞ L upper,S (f ) = L(f ), larger S gives a more precise risk estimation.

Label-Mixing or Label-Preserving
There are two types of sample-mix data augmentation, namely, label-mixing approach and labelpreserving approach. We can show that the objective functions of both approaches are consistent under some conditions.
Proposition 1. Suppose that is a linear function with respect to the second argument and p(λ) = Beta(α, α) for some constant α > 0. Then we have the following equation.
The proof of this theorem can be found in the blog post 1 . For many label-mixing approaches [Zhang et al., 2018, Verma et al., 2018, they use beta distribution for a prior of λ. Thus, the optimization of such approaches can be considered as a special case of our framework because an empirical version of RHS in equation 7 corresponds to L upper,1 where p(λ) is set to Beta(α + 1, α). We experimentally investigate behaviors of both label-mixing and label-preserving training in Sec. 3.

Generalization Bound via Rademacher Complexity
In this section, we present a generalization bound for a function equipped with sample-mix. Let F be a function class of the specified model and R n (F) be the empirical Rademacher complexity of F. Then we have the following inequality.
random variables drawn from an underlying distribution with the density p(x, y) and • F := { • f | f ∈ F}. Suppose that is bounded by some constant B > 0. For any δ > 0, with the probability at least 1 − δ, the following holds for all f ∈ F.
The proof of this theorem can be found in the textbook such as Mohri et al. [2012]. Now we analyze a Rademacher complexity of a proposed function class. Let H be a specified function class | h ∈ H} as defined in equation 3. Suppose that the empirical Rademacher complexity of H can be bounded with some constant C H as follows.
We can prove this assumption holds for neural network models in a similar manner to Gao and Zhou [2016]. Then we have the following theorem.
Theorem 1. Suppose that is a ρ-Lipschitz function with respect to the first argument (0 < ρ < ∞) and H satisfies the assumption in equation 9. Let F = {f : X → R} be a function class of f defined in equation 3. Then we have the following inequality. where always holds from Jensen's inequality and C λ ∈ [0, 1]. Thus, sample-mix can reduce the empirical Rademacher complexity of the function class, which reduces the generalization gap (i.e., L(f ) − L(f )). For example, when p(λ) = Beta(α + 1, α) in equation 3, we have C λ = α+1 2α+1 , which is a monotonically decreasing function with respect to α. Hence, we claim that larger α can be effective for the smaller generalization gap. We experimentally analyze the behavior with respect to α in Sec. 3.

Experiments on CIFAR Datasets
In this section, we analyze the behavior of our proposed framework through experiments on CI-FAR10/100 datasets [Krizhevsky and Hinton, 2009]. We evaluated the classification performances with two neural network architectures, VGG16 [Simonyan and Zisserman, 2015] and PreActResNet18 [He et al., 2016]. The details of the experimental setting are described in Appendix B. For our proposed DIP, output after final fully-connected layer is used as h(x) in equation 3 and the expected output is approximated by 500 times monte carlo sampling in test stage. As we discussed in Sec. 2.3, there are two types of optimization process when S = 1. We evaluated both label-preserving and labelmixing style training. We set p(λ) = Beta(α + 1, α) for label-preserving sample-mix training and p(λ) = Beta(α, α) for label-mixing sample-mix training. Note that the prediction was computed with p(λ) = Beta(α + 1, α) in test stage even when label-mix style was used for training.
For two baseline methods, we trained a classifier with (i) standard training (without sample-mix) and (ii) Mixup [Zhang et al., 2018] training (label-mixing style). To evaluate the performances of these methods, we computed the prediction only from original clean samples. We used p(λ) = Beta(α, α) for Mixup training.
We show the classification performances in Table 1 and generalization gap (i.e., the gap between train and test performances) in Fig. 2. Note that the magnified versions of experimental results are deferred to Appendix C. As can be seen in Table 1, our proposed method is likely to outperform existing Mixup approach.
Remarks: For all approaches including existing Mixup, the larger α leads to the smaller generalization gap, which is consistent with the discussion in Sec. 2.4. In addition, we found that the larger S is likely to enlarge the gap and deteriorate the performance on test samples. It might be because the variance of the empirical loss function computed by S times sampling plays a role of regularization.  [Zhang et al., 2018] 5

Conclusion
In this paper, we proposed a novel framework called DIP, where sample-mix is encapsulated in the hypothesis class of a classifier. We theoretically evaluated the generalization error bound via Rademacher complexity and showed that sample-mix is effective to reduce the generalization gap. Through experiments on CIFAR datasets, we demonstrated that our approach can outperform existing Mixup data augmentation.

A Proofs of Theorem 1
In this section, we give a complete proof of theorem 1. The empirical Rademacher complexity is defined as follows.
Definition 1. Let n be a positive integer, x 1 , ..., x n be i.i.d. random variables drawn from p(x), F = {f : X → R} be a class of measurable functions, and = ( 1 , ..., n ) be Rademacher random variables, namely, random variables taking +1 and −1 with the equal probabilities. Then the empirical Rademacher complexity of F is defined as We assume that is a ρ-lipschitz function with respect to first argument. Here we have the following useful lemma. The proof of this lemma can be found in [Mohri et al., 2012].
Lemma 1 (Talagrand's lemma). Let Φ : R → R be an ρ-lipschitz function. Then for any hypothesis set H of real valued function functions, the following inequality holds: From this lemma, we have In equation 9, we assume that Now we can bound R n (F) as follows.
By combining the above result and equation 13, we complete the proof of this theorem.

B Details of Experimental Setting
In this section, we describe the details of training for experiments in Section 3.

B.1 Training
VGG16 [Simonyan and Zisserman, 2015] and PreActResNet18 [He et al., 2016] was used for experiments. We did not apply Dropout similarly to Mixup [Zhang et al., 2018]. For all experiments, we trained a neural network for 200 epoch. Learning rate was set to 0.1 in the beginning and multiplied by 0.1 at 100 and 150 epoch. We applied standard augmentation such as cropping and flipping. The size of mini-batch was set to 128. We set p(λ) = Beta(α + 1, α) for label-preserving sample-mix training and p(λ) = Beta(α, α) for label-mixing sample-mix training. λ was generated for each sample in mini-batch, and x was obtained by permutation of samples in mini-batch.

B.2 Prediction
For standard without sample-mix method and Mixup method, we predicted labels of test samples from original clean samples. For proposed method, we predicted labels from the expectation over mixed samples computed by monte carlo approximation. In the same manner to the training process, we sampled λ and x 500 times and calculated the average to obtain the final output. In evaluation stage, data augmentation except for sample-mix was turned off.

C Magnified Versions of Experimental Results
In this section, we present the magnfied version of experimental results in Section 3.