{'fraction_non_alphanumeric': 0.0389131057303764, 'fraction_numerical': 0.01514524801934519, 'mean_word_length': 4.416164053075995, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards a Deeper Understanding of Adversarial Losses

Recent work has proposed various adversarial losses for training generative adversarial networks. Yet, it remains unclear what certain types of functions are valid adversarial loss functions, and how these loss functions perform against one another. In this paper, we aim to gain a deeper understanding of adversarial losses by decoupling the effects of their component functions and regularization terms. We first derive some necessary and sufficient conditions of the component functions such that the adversarial loss is a divergence-like measure between the data and the model distributions. In order to systematically compare different adversarial losses, we then propose DANTest—a new, simple framework based on discriminative adversarial networks. With this framework, we evaluate an extensive set of adversarial losses by combining different component functions and regularization approaches. This study leads to some new insights into the adversarial losses. For reproducibility, all source code is available at https: //github.com/salu133445/dan.


Introduction
Generative adversarial networks (GANs) (Goodfellow et al., 2014) are a class of unsupervised machine learning algorithms. In essence, a GAN learn a generative model with the guidance of another discriminative model which is trained jointly. However, the idea of adversarial losses is not limited to unsupervised learning. Adversarial losses can also be applied to supervised and semi-supervised scenarios (e.g., (Isola et al., 2017;dos Santos et al., 2017)). Over the past few years, adversarial losses have advanced the state of the art in many fields (Goodfellow, 2016).
Despite the success, there are several open questions that need to be addressed. On one hand, although plenty adversarial losses have been proposed, we have little theoretical understanding of what makes a loss function a valid one.
On the other hand, we note that any two adversarial losses can differ in terms of not only the component functions (e.g., minimax or hinge; see Section 2) used in the main loss function that sets up the two-player adversarial game, but also the regularization approaches (e.g., gradient penalties (Gulrajani et al., 2017)) used to regularize the models. However, it remains unclear how they respectively contribute to the performance of an adversarial loss. In other words, when empirically compare two adversarial losses, we need to decouple the effects of the component functions and the regularization terms, otherwise we cannot tell which one of them makes an adversarial loss better than the other.
Among existing comparative analysis of adversarial losses, to the best of our knowledge, only Lucic et al. (2018) and Kurach et al. (2018) attempted to decouple the effects of the component functions and regularization approaches. But, only few combinations of component functions and regularization approaches were tested in these two prior works, only seven and nine respectively. We attribute this to the high computational cost that may involve to conduct the experiments, and, more importantly, the lack of a framework to systematically evaluate adversarial losses.
These two research questions can be summarized as follows: RQ1 What certain types of component functions are theoretically valid adversarial loss functions? RQ2 How different combinations of the component functions and the regularization approaches perform empirically against one another?
We aim to tackle these two RQs in this paper to advance our understanding of the adversarial losses. Specifically, our contribution to RQ1 is based on the intuition that a favorable adversarial loss should be a divergence-like measure between the distribution of the real data and the distribution of the model output, since in this way we can use the adversarial loss as the training criterion to learn the model parameters. We derive necessary and sufficient conditions such that an adversarial loss has such a favorable property (Sections 3.3 and 3.4). Interestingly, our theoretical analysis leads to a new perspective to understand the underlying game dynamics of adversarial losses (Section 3.6).
sarial networks (DANs) (Mirza & Osindero, 2014), which are essentially conditional GANs with both the generator and the discriminator being discriminative models. Based on DANs, we propose DANTest-a new, simple framework for comparing adversarial losses (Section 4). The main idea is to first train a number of DANs for a supervised learning task (e.g., classification) using different adversarial losses, and then compare their performance using standard evaluation metrics for supervised learning (e.g., classification accuracy). With the DANTest, we systematically evaluate 168 adversarial losses featuring the combination of ten existing component functions, two new component functions we originally propose in this paper in light of our theoretical analysis, and 14 existing regularization approaches (Section 5). Moreover, we use the DANTest to empirically study the effect of the Lipschitz constant , penalty weights (Mescheder et al., 2018), momentum terms (Kingma & Ba, 2014), and others. We discuss the new insights that are gained, and their implications to the design of adversarial losses in future research.

Generative Adversarial Networks
A generative adversarial network (Goodfellow et al., 2014) is a generative latent variable model that aims to learn a mapping from a latent space Z to the data space X , i.e., a generative model G, which we will refer to as the generator. A discriminative model D (i.e., the discriminator) defined on X is trained alongside the G to provide guidance for it. Let p d denote the data distribution and p g be the model distribution implicitly defined by G(z) when z ∼ p z . In general, most GAN loss functions proposed in the literature can be formulated as: where f , g and h are real functions defined on the data space (i.e., X → R) and we will refer to them as the compoenent functions. We summarize in Table 1 the component functions f , g and h used in some existing adversarial losses.
Some prior work has also investigated the so-called IPM-based GANs, where the discriminator is trained to estimate an integral probability metric (IPM) between p d and p g : where D is a set of functions from X to R. For example, the Wasserstein GANs  consider D to be the set of all 1-Lipschitz functions. Other examples include McGAN (Mroueh et al., 2017), MMD GAN  and Fisher GAN (Mroueh & Sercu, 2017). Please note that the main difference between (1) and (3) is that in the latter we constrain D to be in some set of functions D.

Gradient Penalties
As the discriminator is often found to be too strong to provide reliable gradients to the generator, one regularization approach is to use some gradient penalties to constrain the modeling capability of the discriminator. Most gradient penalties proposed in the literature take the following form: where the penality weight λ ∈ R is a pre-defined constant, and R(·) is a real function. The distribution px defines where the gradient penalties are enforced. Table 2 shows the distribution px and function R used in some common gradient penalties. And, Figure 1 illustrates px.
When gradient penalties are enforced, the loss function for training the discriminator contains not only the component functions f and g in (1) but also the regularization term (4).

Spectral Normalization
Another regularization approach we consider is the spectral normalization proposed by Miyato et al. (2018). It normalizes the spectral norm of each layer in a neural network to enforce the Lipschitz constraints. While the gradient penalties introduced in Section 2.2 impose local regularizations, the spectral normalization imposes a global regularization on the discriminator. Therefore, it is possible to combine the spectral normalization with the gradient penalties. We will examine this in Section 5.3.

px R(x)
coupled gradient penalties (Gulrajani et al., 2017) local gradient penalties (Kodali et al., 2017) p d + c N (0, I) (x − k) 2 or max(x, k) R 1 gradient penalties (Mescheder et al., 2018) p d x R 2 gradient penalties (Mescheder et al., 2018) p g x Table 2. Distribution px and function R in (4) for common gradient penalties, where c, k ∈ R are considered hyperparameters (k is the Lipschitz constant). We will refer to the (x − k) 2 and the max(x − k) versions as the two-side and the one-side penalties, respectively.

Theoretical Results
In the following analysis, we follow the notations in (1) and (2). Proofs can be found in Appendix A.

Favorable properties for adversarial losses
Let us first consider the minimax formulation: We can see that if the discriminator is able to reach optimality, the training criterion for the generator is In general, for a valid adversarial loss, the discriminator is responsible for providing a measure of the discrepancy between the data distribution p d and the model distribution p g . In principle, this will then serve as the training criterion for the generator to push p g towards p d . Hence, we would like such an adversarial loss to be a divergence-like measure between p g and p d . From this view, we can now define the following two favorable properties of adversarial losses. Property 1. (Weak favorable property) For any fixed p d , L G has a global minimum at p g = p d . Property 2. (Strong favorable property) For any fixed p d , L G has a unique global minimum at p g = p d .
We can see that Property 2 makes L G − L * G a divergence of p d and p g for any fixed p d , where L * G = L G pg=p d , and Property 1 provides a weaker version when the identity of indiscernibles is not necessary. Note that L G is not a divergence since L G ≥ 0 does not always hold.

Ψ and ψ functions
In order to derive some necessary and sufficient conditions for Properties 1 and 2, we first observe from (6) that Now, if we letγ = Please note thatγ(x) = 1 2 if and only if p d (x) = p g (x). Let us now consider the terms inside the integral and define the following two functions: where γ ∈ [0, 1] and y ∈ R are two variables independent of x. We visualize in Figures 2(a)-(d) the Ψ and ψ functions for different common adversarial losses (see Appendix B for the graphs of the ψ functions alone). These two functions actually reflect some important characteristics of the adversarial losses (see Section 3.6) and will be used intensively in our theoretical analysis. The green lines show the domains of the ψ functions (i.e., the value(s) that y can take for different γ in (11)). The star marks, and any points on the yellow dashed lines, are the minimum points of ψ. The midpoints of the color maps are intentionally set to the minima of ψ (i.e., the values taken at the star marks or the yellow segments). Note that γ ∈ [0, 1] and y ∈ R, so we plot different portions of y where the characteristics of Ψ can be clearly seen.

Necessary conditions for the favorable properties
For the necessary conditions of Properties 1 and 2, we have the following two theorems.
Theorem 1. If Property 1 holds, then for any γ ∈ [0, 1], With Theorems 1 and 2, we can easily check if a pair of component functions f and g form a valid adversarial loss.

Sufficient conditions for the favorable properties
For sufficient conditions, we have two theorems as follows.
We also have the following theorem for a more specific guideline for choosing the component functions f and g.
By Theorems 4 and 5, we now see that any component function pair f and g that satisfies the prerequisites in Theorem 5 makes L G − L * G a divergence between p d and p g for any fixed p d . Interestingly, while such a theoretical analysis has not been done before, it happens that all the adversarial loss functions listed in Table 1 have such favorable properties. We intend to examine in Section 5.2 empirically the cases when the prerequisites of Theorem 5 do not hold.
In practice, the discriminator often cannot reach optimality at each iteration. Therefore, as discussed by Nowozin et al. (2016); Fedus et al. (2018), the objective of the generator is similar to variational divergence minimization (i.e., to minimize a lower bound of some divergence between p d and p g ), where the divergence is estimated by the discriminator.

Loss functions for the generator
Intuitively, the generator should minimize the divergencelike measure estimated by the discriminator. We have accordingly h = g. However, some prior works have investigated setting h different from g. In general, most of these alternative generator losses do not change the solutions of the game and are proposed base on some heuristics. While our theoretical analysis concerns with only f and g, we intend to empirically examine the effects of the generator loss function h in Section 5.4.
3.6. Analyzing the adversarial game by the Ψ functions Figure 2 gives us some new insights regarding the adversarial behaviors of the discriminator and the generator. On one hand, if we follow (9) and considerỹ = D(x) and , then the discriminator can be viewed as maximizing Ψ along theỹ-axis. On the other hand, since the generator is trained to push p g towards p d , it can be viewed as minimizing Ψ along theγ-axis. In this way, we can see why all these Ψ functions are saddle-shaped and have saddle points at γ = 1 2 (i.e., when p d (x) = p g (x)). Ideally, if the discriminator can be trained till optimality, then we will be on the green line, the domain of the ψ function. In this case, the generator can be viewed as minimizing Ψ along the green line (i.e., minizing ψ). Note that as L G is an integral over all possible x, such adversarial game is actually being played in a (usually) high dimensional space.
By designing the landscape of Ψ, we propose and consider two new losses in our empirical study in Section 5.3: • The absolute loss, with f (y) = −h(y) = −|1 − y|, g(y) = −|y|. Its Ψ-landscape is similar to those of the least squares and the hinge losses (see Figure 2(e)). • The asymmetric loss, with f (y) = −|y|, g(y) = h(y) = −y. Its Ψ-landscape is similar to that of the Wasserstein loss, but the positive part of y is blocked (see Figure 2(f)). G is now a discriminative model that aims to predict the label of a real data sample. D takes as input either a "(real data, real label)" or a "(real data, fake label)" pair and aims to examine its authenticity.

DANTest
Discriminative adversarial networks (DANs) (dos Santos et al., 2017) are essentially conditional GANs (Mirza & Osindero, 2014) where both the generator and the discriminator are discriminative models, as shown in Figure 3. Based on DANs, we propose a new, simple framework, dubbed DANTest, for systematically comparing different adversarial losses. Specifically, the DANTest goes as follows: Step 1 Build several DANs. For each of them, the generator G takes as input a real sample and outputs a fake label. The discriminator takes as input a real sample with either its true label, or a fake label made by G, and outputs a scalar indicating if the "sample-label" pair is real.
Step 2 Train the DANs with different component loss functions, regularization approaches or hyperparameters.
Step 3 Predict the labels of test data by the trained models.
Step 4 Compare the performance of different models with standard evaluation metrics used in supervised learning.
Note that the generator is no longer a generative model in this framework, while the discriminator is still trained by the same loss function to measure the discrepancy between p d and p g . This way, we can still gain insight into the performance and stability for different adversarial losses. Moreover, although we take a classification task as an example here, the proposed framework is generic and can be applied to other supervised learning tasks as well, as long as the evaluation metrics for that task are well defined.
An extension of the proposed framework is the imbalanced dataset test, where we examine the ability of different adversarial losses on datasts that feature class imbalance. This can serve as a measure of the mode collapse phenomenon (Che et al., 2017), which is a commonly-encountered failure case in GAN training. By testing on datasets with different levels of imbalance, we can examine how different adversarial losses suffer from the mode collapse problem.  Table 3. Error rates (%) for the -weighted versions of the nonsaturating, the Wasserstein and the hinge losses (see (12)) on the standard dataset. Here, = 1.0 corresponds to the original losses.

Datasets and Implementation Details
All the experiments reported here are done based on the DANTest. If not otherwise specified, we use the MNIST handwritten digits database (LeCun et al., 1998), which we refer to as the standard dataset. As it is class-balanced, we create two imbalanced versions of it. The first one, referred to as the imbalanced dataset, is created by augmenting the training samples for digit '0' by shifting them each by one pixel to the top, bottom, left and right, so that it contains five times more training samples of '0' than the standard dataset. Moreover, we create the very imbalanced dataset, where we have seven times more training samples for digit '0' than the standard dataset. For other digits, we randomly sample from the standard dataset and intentionally make the sizes of the resulting datasets identical to that of the standard dataset. We use the same test set for all the experiments.
We implement G and D as convolutional neural networks (see Appendix C for the network architectures). We use the batch normalization (Ioffe & Szegedy, 2015) in G. If the spectral normalization is used, we only apply it to D, otherwise we use the layer normalization (Ba et al., 2016) in D. We concatenate the label vector to each layer of D. For the gradient penalties, we use Euclidean norms and set λ to 10.0 (see (4)), k to 1.0 and c to 0.01 (see Table 2). We use the Adam optimizers (Kingma & Ba, 2014) with α = 0.001, β 1 = 0.0 and β 2 = 0.9. We alternatively update G and D once in each iteration and train the model for 100,000 generator steps. The batch size is 64. We implement the model in Python and TensorFlow (Abadi et al., 2016). We run each experiment for ten runs and report the mean and the standard deviation of the error rates.

Examining the necessary conditions for favorable adversarial loss functions
As discussed in Section 3.4, we examine here the cases when the prerequisites in Theorem 5 do not hold. We consider the classic nonsaturating, the Wasserstein and the hinge losses and change the training objective for the discriminator into where ∈ R is a constant. The prerequisites in Theorem 5 do not hold when = 1. We illustrate the Ψ functions of these -weighted losses in Appendix B. Table 3 shows the results for = 0.5, 0.9, 1.0, 1.1, 2.0, using the spectral normalization for regularization. We can see that all the original losses (i.e., = 1) result in the lowest error rates. In general, the error rates increase as goes away from 1.0. Notably, the Wasserstein loss turn out failing with error rates over 50% when = 1.

On different discriminator loss functions
In this experiment, we aim to compare different discriminator loss functions. Specifically, we evaluate an comprehensive set (in total 168) of different combinations of component functions and regularization approaches.
For the component functions, we consider the classic minimax and the classic nonsaturating losses (Goodfellow et al., 2014), the Wasserstein loss , the least squares loss (Mao et al., 2017), the hinge loss (Lim & Ye, 2017;Tran et al., 2017), the relativistic average and the relativistic average hinge losses (Jolicoeur-Martineau, 2018), as well as the absolute and the asymmetric losses we propose and describe in Section 3.6.
For the regularization approaches, we consider the coupled, the local, the R 1 and the R 2 gradient penalties (GP) and the spectral normalization (SN). For the coupled and the local gradient penalties, we examine both the two-side and the one-side versions (see Table 2). We will use in the captions OCGP and TCGP as the shorthands for the one-side and the two-side coupled gradient penalties, respectively, and OLCP and TLCP for the one-side and the two-side local gradient penalties, respectively. We also consider the combinations of the SN with different gradient penalties.
We report in Table 4 the results for all the combinations and present in Figure 4 the training progress for the nonsaturating and the hinge losses. We can see that there is no single winning component functions and regularization approach across all different settings. Some observations are: With respect to the component functions- • The classic minimax and nonsaturating losses never get the lowest three error rates for all different settings. • The hinge, the asymmetric and the two relativistic losses are robust to different regularization approaches and tend to achieve lower error rates. • The relativistic average loss outperforms both the classic minimax and nonsaturating losses across all regularization approaches. But, the relativistic average hinge loss does not always outperform the standard hinge loss.
With respect to the regularization approaches- • The coupled and the local GPs outperform the R 1 and the R 2 GPs across nearly all different component functions, no matter whether the SN is used or not. • The coupled and the local GPs stabilize the training (see Figure 4) and tend to have lower error rates. • The R 2 gradient penalties achieve lower error rates than the R 1 gradient penalties. In some cases, they can be too strong and even stop the training early (see Figure 4 (a)). 1 • Combining either the coupled or the local GP with the SN usually leads to higher error rates than using the coupled or the local GP only. • Similarly, combining either the R 1 or the R 2 GP with the SN degrades the result. Moreover, it leads to unstable training (see Figures 4(b) and (d)). This result implies that R 1 and R 2 GPs do not work well with the SN. • Using the one-side GPs instead of their two-side counterparts increase the error rates by 0.1-9.5%. (We report the results for the one-side GPs in Appendix D due to page limit.) We also note that some combinations result in remarkably high error rates, e.g., "least squares loss + R 1 GP", "least squares loss + R 2 GP" and "classic minimax loss + R 1 GP".
In sum, according to the overall performance and the robustness to different settings, for the component functions, we recommend the hinge, the asymmetric and the two relativistic losses. We note that these functions also feature lower computation costs as all their components functions are piecewise linear (see Table 1 and Section 3.6). For the regularization approaches, we recommend the two-side coupled and the two-side local gradient penalties.
We also conduct the imbalanced dataset test (see Section 4) on the two imbalanced datasets described in Section 5.1 to compare the regularization approaches. We use the classic nonsaturating loss. As shown in Table 5, the error rates increase as the level of imbalance increases. The two-side local GP achieve the lowest error rates across all three datasets. The error rates for the R 1 and the R 2 GPs increase significantly when the dataset goes imbalanced.

EFFECTS OF THE LIPSCHITZ CONSTANTS
In this experiment, we examine the effects of the Lipschitz constant (k) used in the coupled and the local GPs (see Table 2). We use the classic nonsaturating loss here. We report in Figure 5 the results for k = 0.01, 0.1, 1, 10, 100.
We can see that the error rate increases as k goes away from 1.0, suggesting that k = 1 is indeed a good default value. Moreover, the two-side GPs are more sensitive to k than their one-side counterparts.  We note that Petzka et al. (2018) suggested that the oneside coupled GP are preferable to the two-side version and showed empirically that the former has more stable behaviors. However, we observe in our experiments that the twoside penalties usually lead to faster convergence to lower error rates compared to the one-side penalties. 2

EFFECTS OF THE PENALTY WEIGHTS
We then examine the effects of the penalty weights (λ) for the R 1 and the R 2 GPs (see (4)). We consider the classic nonsaturating, the Wasserstein and the hinge losses. We present in Figure 6 the results for λ = 0.01, 0.1, 1, 10, 100.
We can see that the R 1 GP tends to outperform the R 2 GP, while they are both sensitive to the value of λ. Hence, future research should run hyperparmeter search for λ to find out its optimal value. When the spectral normalization is not used, the hinge loss is less sensitive to λ than the other two losses. However, when spectral normalization is used, the error rate increases as λ increases, which again implies that the R 1 /R 2 GPs and the SN do not work well together.

On different generator loss functions
As discussed in Section 3.5, we also aim to examine the effects of the generator loss function h(·). We consider the classic and the hinge losses for the discriminator and the  Table 4. For the classic discriminator loss, we see no single winner among the three generator loss functions across all the regularization approaches, which implies that the heuristics behind these alternative losses might not be true. For the hinge discriminator loss, the minimax generator loss is robust to different regularization approaches and achieves three lowest and four lowest-three scores. Hence, we recommend to use hinge loss for the discriminator and minimax loss for the generator as the overall best choice according to our experimental results.

Effects of the momentum terms of the optimizers
We observe a trend towards using smaller momentum (Radford et al., 2016) or even no momentum Gulrajani et al., 2017;Miyato et al., 2018;Brock et al., 2018) in GAN training. Hence, we would also like to examine the effects of momentum terms in the optimizers with the proposed framework. As suggested by Gidel et al. (2018), we also include a negative momentum value of −0.5. We use the classic nonsaturating loss and the SN along with the coupled GPs for regularization. Figure 7 shows the results for all combinations of β 1 = −0.5, 0.0, 0.5, 0.9 for G and D. We can see that for the two-side coupled GP, using larger momenta in both G and D leads to lower error rates, while there is no specific trend for the one-side coupled GP.

Discussions and Conclusions
In this paper, we have shown in theory what certain types of component functions form a valid adversarial loss. We have also introduced a new framework called DANTest for comparing adversarial losses. With DANTest, we systematically compared combinations of different component functions and regularization approaches to decouple their effects. Our empirical results show that there is no single winning component functions or regularization approach across all different settings. Our theoretical and empirical results can together serve as a reference for choosing or designing adversarial training objectives in future research.
As compared to the commonly used metrics for evaluating generative models, such as the Inception Score (Salimans et al., 2016) and Frchet Inception Distance (Heusel et al., 2017) adopted in Lucic et al. (2018) and Kurach et al. (2018), the DANTest is simpler and is easier to control and extend. This allows us to easily evaluate new adversarial losses. However, we note that while the discriminator in a DAN is trained to optimize the same objectives as in a conditional GAN, the generators in the two models actually work in opposite ways (X → Z in a DAN versus Z → X in a GAN). Hence, it is unclear whether the empirical results can be generalized to conditional and unconditional GANs. Nonetheless, recent work has also adapted adversarial losses to plenty discriminative models (e.g., image-to-image translation (Isola et al., 2017) and image super-resolution (Ledig et al., 2017)). Therefore, it is worth investigating the behaviors of adversarial losses in different scenarios.
In addition, our theoretical analysis provides a new perspective on adversarial losses and reveals a large class of component functions valid for adversarial losses. We note that Nowozin et al. (2016) has also shown a certain class of component functions can result in theoretically valid adversarial losses. However, in their formulations, the component functions f and g are not independent of each other as they considered only the f -divergences. A future direction is to investigate the necessary and sufficient conditions for the existence and the uniqueness of a Nash equilibrium.
Theorem 2. If Property 2 holds, then for any γ ∈ Proof. Since Property 2 holds, we have for any fixed p d , Following the proof of Theorem 1, consider for some γ ∈ [0, 1] and some s, t ∈ X , s = t. It can be easily shown that p g = p d if and only if γ = 1 2 . As (20) holds for any fixed p d , by substituting (21) and (22) into (20), we get for any γ ∈ [0, 1] \ { 1 2 }, concluding the proof. Theorem 3. If ψ(γ) has a global minimum at γ = 1 2 , then Property 1 holds.
Proof. First, we see that = max y (f (y) + g(y)) = max y f (y) + g(y) (29) On the other had, we have dx . Since dx . (35) As ψ(γ) has a global minimum at γ = 1 2 , now we have Finally, combining (30) and (38) yields which holds for any p d , thus concluding the proof.
Proof. First, we have by definition By taking the partial derivatives, we get We know that there exists some y * such that f (y * ) = −g (y * ) = 0 .

B. More Graphs of the Ψ and ψ Functions
We show in Figure 1 the graphs of the ψ functions for different adversarial losses. Note that for the Wasserstein loss, the ψ function is only defined at γ = 0.5, where it takes the value of zero, and for the asymmetric loss, the ψ function is only defined when γ > 0.5, where it takes the value of zero. Hence, we do not include them in Figure 1.
We also present in Figure 2 the graphs of the Ψ functions for the -weighted versions of the classic, the Wasserstein and the hinge losses. Moreover, Figures 1(b) and (c) show the graphs of the ψ functions for the -weighted versions of the classic and the hinge losses, respectively.

C. Network Architectures
We present in Table 1 the network architectures for the generator and the discriminator used for all the experiments.

D. More Results
We report in Table 2 the results for the one-side coupled and local gradient penalties.
We also present in Figure 3 the results for the experiment on the momentum terms using the hinge loss.  The midpoints of the color maps are intentionally set to the minima of ψ (i.e., the values taken at the star marks or the yellow segments). Note that γ ∈ [0, 1] and y ∈ R, so we plot different portions of y where the characteristics of Ψ can be clearly seen. Table 2. Error rates (%) for different adversarial losses and regularization approaches, on the standard dataset. OCGP and OLCP stand for the one-side coupled and the one-side local gradient penalties, respectively. SN stands for the spectral normalization. M, N and L stand for the minimax, the nonsaturating and the linear loss functions used in the generator, respectively. Underlined and bold fonts indicate respectively entries with the lowest and lowest-three mean error rates per column.