{'fraction_non_alphanumeric': 0.059714795008912656, 'fraction_numerical': 0.018394204877308758, 'mean_word_length': 4.049794120463468, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Structure-preserving GANs

Generative adversarial networks (GANs), a class of distribution-learning methods based on a two-player game between a generator and a discriminator, can generally be formulated as a minmax problem based on the variational representation of a divergence between the unknown and the generated distributions. We introduce structure-preserving GANs as a data-efficient framework for learning distributions with additional structure such as group symmetry, by developing new variational representations for divergences. Our theory shows that we can reduce the discriminator space to its projection on the invariant discriminator space, using the conditional expectation with respect to the sigma-algebra associated to the underlying structure. In addition, we prove that the discriminator space reduction must be accompanied by a careful design of structured generators, as flawed designs may easily lead to a catastrophic"mode collapse"of the learned distribution. We contextualize our framework by building symmetry-preserving GANs for distributions with intrinsic group symmetry, and demonstrate that both players, namely the equivariant generator and invariant discriminator, play important but distinct roles in the learning process. Empirical experiments and ablation studies across a broad range of data sets, including real-world medical imaging, validate our theory, and show our proposed methods achieve significantly improved sample fidelity and diversity -- almost an order of magnitude measured in Fr\'echet Inception Distance -- especially in the small data regime.


Introduction
Since their introduction by Goodfellow et al. [2014], generative adversarial networks (GANs) have become a burgeoning domain in distribution learning with a diverse range of innovative applications Karras et al. [2019], Mustafa et al. [2019], Yi et al. [2019], Zhu et al. [2019]. Mathematically, the minmax game between a generator and a discriminator in GAN can typically be formulated as minimizing a divergence-or other notions of "distance"-with a variational representation between the unknown and the generated distributions. Such formulations, however, do not make prior arXiv:2202.01129v2 [cs.LG] 17 Jun 2022 structural assumptions on the probability measures, making them sub-optimal in sample efficiency when learning distributions with intrinsic structures, such as the (rotation) group symmetry for medical images without preferred orientation; see Figure 1.  We introduce, in this work, the structure-preserving GANs, a data-efficient framework for learning probability measures with embedded structures, by developing new variational representations for divergences between structured distributions. We demonstrate that efficient adversarial learning can be achieved by reducing the discriminator space to its projection onto its invariant subspace, using the conditional expectation with respect to the σ-algebra associated to the underlying structure; such practice, which is rigorously justified by our theory and generally applicable to a broad range of variational divergences, acts effectively as an unbiased regularization to prevent discriminator overfitting, a common challenge for GAN optimization in the limited data regime Zhao et al. [2020]. Furthermore, our theory suggests that the discriminator space reduction must be accompanied by correctly building generators sharing the same probabilistic structure, as the lack of which may easily lead to "mode collapse" in the trained model, i.e., the generated distribution samples only a subset of the support of the data source [cf. Figure 4a (2nd row)].
As an example, we contextualize our framework by building symmetry-preserving GANs for learning distributions with group symmetry. Unlike prior empirical work, our choice of equivariant generators and invariant discriminators is theoretically founded, and we show (theoretically and empirically) how flawed design of equivariant generators results easily in the aforementioned mode collapse [cf. Figure 4a (4th row)]. Experiments and ablation studies over synthetic and real-world data sets validate our theory, disentangle the contribution of the structural priors on generators and discriminators, and demonstrate the significant outperformance of our framework in terms of both sample quality and diversity-in some cases almost by an order of magnitude measured in Fréchet Inception Distance; see Figure 1 and 2 for a visual illustration.
In Section 2 we will discuss several related approaches to equivariant GANs. We provide background on GANs, variational representations of divergences, and group equivariance in Section 3. Section 4 contains our main theoretical results regarding divergences between structured distributions. Section 5 contains additional theoretical results specific to a primal formulation of (f, Γ)-divergences for structured distributions, building on the inf-convolution formulation of general (f, Γ)-divergences in Birrell et al. [2022]. Finally, our experiments on synthetic and real-world data sets are found in Section 6.

Related Work
Neural generation of group-invariant distributions has mainly been proposed in a flow-based framework Biloš & Günnemann [2021], Boyda et al. [2021], Garcia Satorras et al. [2021], Köhler et al. [2019Köhler et al. [ , 2020, Liu et al. [2019], Rezende et al. [2019]. Such models typically use an equivariant normalizing-flow to push-forward a group-invariant prior distribution to a complex invariant target. In the context of GANs, Dey et al. [2021] intuitively replace the 2D convolutions with group convolutions Cohen & Welling [2016a] to build group-equivariant GANs; however, their empirical study has not been justified by theory, and their incomplete design of the equivariant generator may easily lead to a "mode collapse" of the learned model; see the discussion of Theorem 4.7. The existence of symmetry can often be deduced from prior or domain knowledge of the distribution, e.g., the rotation symmetry for medical images without preferred orientation. Symmetry detection from data has also been studied in recent works such as Dehmamy et al. [2021]. When extended from group symmetry to probability structures induced from other operators, our work is also related to GAN-assisted coarse-graining (CG) for molecular dynamics Durumeric & Voth [2019] and cosmology Feder et al. [2020], Mustafa et al. [2019]; see the end of Section 4.1 for a detailed discussion.

Generative adversarial networks
Generative adversarial networks are a class of methods in learning a probability distribution via a zero-sum game between a generator and a discriminator Arjovsky et al. [2017], Goodfellow et al. [2014], Gulrajani et al. [2017], Nowozin et al. [2016]. Specifically, let (X, M) be a measurable space, and P(X) be the set of probability measures on X; given a target distribution Q ∈ P(X), the original GAN proposed by Goodfellow et al. [2014] learns Q by solving inf g∈G D(Q P g ) = inf g∈G sup γ∈Γ H(γ; Q, P g ), where H(γ; Q, P g ) = E Q [log γ] + E Pg [log(1 − γ)]. The map g : Z → X in Eq. (1) is called a generator, which maps a random vector z ∈ Z to a generated sample g(z) ∈ X, pushing forward the noise distribution P ∈ P(Z) (typically a Gaussian) to a probability measure P g ∈ P(X), i.e., P g := g * P := P • g −1 .
The test function γ : X → R is called a discriminator, which aims to differentiate the source distribution Q and the generated probability measure P g by maximizing H(γ; Q, P g ). The spaces G and Γ, respectively, of generators and discriminators are both parametrized by neural networks (NNs), and the solution of model (1) is the best generator g ∈ G that is able to "fool" all discriminators γ ∈ Γ by achieving the smallest D(Q P g ), which measures the "dissimilarity" between Q and P g .

Variational representations for divergences
Mathematically, most GANs can be formulated as minimizing the "distance" between the probability measures Q and P g according to some divergence or probability metric with a variational representation sup γ∈Γ H(γ; Q, P g ) as in (1). We hereby recast these formulations in a unified but flexible mathematical framework that will prove essential in Section 4.1. Let M(X) be the space of measurable functions on X and M b (X) be the subspace of bounded measurable functions. Given an objective functional H : if and only if Q = P , hence providing a notion of "distance" between probability measures. Variational representations of the form (3) have been widely used, including in GANs Arjovsky et al. [2017], Goodfellow et al. [2014], Nowozin et al. [2016], divergence estimation Birrell et al. [2021], Nguyen et al. [2007Nguyen et al. [ , 2010, Ruderman et al. [2012], determining independence through mutual information estimation Belghazi et al. [2018], uncertainty quantification of stochastic processes Chowdhary & Dupuis [2013], Dupuis et al. [2016], bounding risk in probably approximately correct (PAC) learning Catoni et al. [2008], McAllester [1999, Shawe-Taylor & Williamson [1997], parameter estimation Broniatowski & Keziou [2009], statistical mechanics and interacting particles Kipnis & Landim [1999], and large deviations Dupuis & Ellis [2011]. It is known that formula (3) includes, through suitable choices of functional H(γ; Q, P ) and function space Γ, many divergences and probability metrics. Below we list several classes of examples.
(a) f -divergences. Let f : [0, ∞) → R be convex and lower semi-continuous (LSC), with f (1) = 0 and f strictly convex at x = 1. The f -divergence between Q and P is where f * denotes the Legendre transform of f . Some notable examples of the f -divergences include the Kullback-Leibler (KL) divergence and the family of α-divergences, which are constructed, respectively, from The flexibility of f allows one to tailor the divergence to the data source, e.g., for heavy tailed data. However, the formula (4) becomes D f (Q P ) = ∞ when Q is not absolutely continuous with respect to P , limiting its efficacy in comparing distributions with low-dimensional support.
(b) Γ-Integral Probability Metrics (IPMs). Given Γ ⊂ M b (X), the Γ-IPM between Q and P is defined as Apart from the Wasserstein metric when Γ = Lip 1 (X) (the space of 1-Lipschitz functions), examples of IPMs also include the total variation metric, the Dudley metric, and maximum mean discrepancy (MMD) Müller [1997], Sriperumbudur et al. [2012]. With suitable choices of Γ, IPMs are able to meaningfully compare not-absolutely continuous distributions, but they could potentially fail at comparing distributions with heavy tails Birrell et al. [2022].
(c) (f, Γ)-divergences. This class of divergences was introduced by Birrell et al. [2022] and they subsume both f -divergences and Γ-IPMs. Given a function f satisfying the same condition as in the definition of the f -divergence and Γ ⊂ M b (X), the (f, Γ)-divergence is defined as where One can verify that (7) includes as a special case the f -divergence (4) when Γ = M b (X), and it is demonstrated in Birrell et al. [2022] that under suitable assumptions on Γ we have making D Γ f suitable to compare not-absolutely continuous distributions with heavy tails. An example of the (f, Γ)divergence is the Lipschitz α-divergence, where f = f α as in Eq. (5), and Γ = Lip L b (X) is the space of bounded L-Lipschitz functions. (d) Sinkhorn divergences. The Wasserstein metric associated with a cost function c : X 2 → R + has the variational representation , ∀x, y ∈ X}, and C(X) is the space of continuous functions on X. The Sinkhorn divergence is given by where W Γ c, (Q, P ) is the entropic regularization of the Wasserstein metrics Genevay et al. [2016], where γ 1 ⊕ γ 2 (x, y) := γ 1 (x) + γ 2 (y) and Γ = C b (X) × C b (X) (C b (X) denotes the space of bounded continuous functions on X).
We refer to Appendix A for a more detailed discussion of the variational divergences introduced above. In all the aforementioned examples, the choice of the discriminator space, Γ, is a defining characteristic of the divergence. We will explain, in Section 4.1, a general framework, i.e., the structure-preserving GANs, for incorporating added structural knowledge of the probability distributions or data sets into the choice of Γ, leading to enhanced performance and data efficiency in adversarial learning of structured distributions.

Group invariance and equivariance
We first introduce the structure-preserving GAN framework in the context of learning distributions with group symmetry.
Here we explain the necessary background and notations. We emphasize that the focus of this work is not to discuss the group-invariance properties of probability measures (which can be found in, e.g., Schindler [2003]), but to understand how to incorporate such structural information into the generator/discriminator of GANs such that invariant probability distributions can be learned more efficiently. However, we first require the following background and notations.
Groups and group actions. A group is a set Σ equipped with a binary operator, the group product, satisfying the axioms of associativity, identity, and invertibility. Given a group Σ and a set X, a map T : Σ × X → X is called a group action if, for all σ ∈ Σ, T σ := T (σ, ·) : X → X is an automorphism on X, and T σ1 • T σ2 = T σ1·σ2 , ∀σ 1 , σ 2 ∈ Σ.
Group equivariance and invariance. Let T Z and T X , respectively, be Σ-actions on the spaces Z and X. A map Invariance is thus a special case of equivariance after equipping Y with the action T Y σ y ≡ y, ∀σ ∈ Σ. In the context of NNs, achieving equivariance/invariance via group-equivariant CNNs (G-CNNs) has been well-studied, and we refer the reader to Cohen et al. [2019], Weiler & Cesa [2019] for a complete theory of G-CNNs.
Let G be a collection of measurable maps g : Z → X. We denote its subset of Σ-equivariant maps as Similarly, let Γ be a set of measurable functions γ : The Finally, a probability measure P ∈ P(X) is called Σ-invariant if P = P • (T X σ ) −1 for all σ ∈ Σ. For instance, the distribution of medical images without orientation preference should be SO(2)-invariant; see Figure 1. The set of all Σ-invariant distributions on X is denoted as 3.4 Definition of Haar measure on Σ and the symmetrization operators S Σ and S Σ We will make frequent use of the symmetrization operators, on both functions and probability distributions, that are induced by a group action on X. These are constructed using the unique Haar probability measure, µ Σ , of a compact Hausdorff topological group Σ (see, e.g., Chapter 11 in Folland [2013]). Intuitively the Haar measure is the uniform probability measure on Σ. Mathematically, this is expressed via the invariance of Haar measure under group multiplication, µ Σ (σ · E) = µ Σ (E · σ) = µ Σ (E) for all σ ∈ Σ and all Borel sets E ⊂ Σ. This is a generalization of the invariance of Lebesgue measure under translations and rotations. The Haar measure can be used to define symmetrization operators on both functions and probability measures as follows (going forward, we assume the group action is measurable).
Symmetrization of functions: Symmetrization of probability measures (dual operator): S Σ : P(X) → P(X), defined for γ ∈ M b (X) by Remark 3.1. Sampling from S Σ [P ]: If x i , i = 1, ..., N are samples from P , and σ j , j = 1, ..., M are samples from the Haar probability measure µ Σ (all independent) then T σj (x i ) are samples from S Σ [P ]. If P is Σ-invariant then the use of T σj (x i ) can be viewed as a form of data augmentation.
The following lemma provides several key properties of the symmetrization operators.
in the sense that (b) The symmetrization operator S Σ : P(X) → P(X) is a projection operator onto the subset of Σ-invariant probability measures P Σ (X) := {P ∈ P(X) : P • T −1 σ = P for all σ ∈ Σ} , (23) in the sense that (c) S Σ is the conditional expectation operator with respect to the σ-algebra of Σ-invariant sets. More specifically, for all γ ∈ M b (X), P ∈ P Σ (X) we have Proof. We will need the following invariance property of integrals with respect to Haar measure, which can be proven using the invariance of Haar measure under left and right group multiplication: Furthermore any γ ∈ M inv b,Σ (X) belongs to the range of S Σ since γ • T σ = γ for all σ ∈ Σ implies that γ = S Σ [γ]. This also shows that S Σ • S Σ = S Σ . Finally, for γ ∈ M b (X), σ ∈ Σ, x ∈ X we can compute where we again used the invariance property of integrals with respect to Haar measure (26).
(b) For P ∈ P(X), γ ∈ M b (X), and σ ∈ Σ we can use (22) to compute This holds for all for all σ ∈ Σ and γ ∈ M b (X) and thus, by Fubini's theorem, . Hence S Σ [P ] = P and so P ∈ S Σ [P]. This completes the proof that S Σ [P(X)] = P Σ (X). Combining these calculations it is also clear that (c) Let γ ∈ M b (X) and P ∈ P Σ (X). From part (a) we know that S Σ [γ] ∈ M inv b,Σ (X) and from this it is straightforward to show that S Σ [γ] is M Σ -measurable. Now fix A ∈ M Σ and note that 1 A • T σ = 1 A for all σ ∈ Σ (where 1 A denotes the indicator function for A). Using this fact together with S Σ [P ] = P (see part (b)) we can compute This proves S Σ [γ] = E P [γ|M Σ ] by the definition of conditional expectation.
Lemma 3.2 implies that since S Σ , S Σ are projections onto M inv b,Σ , P Σ (X) respectively, they are necessarily structurepreserving, namely here symmetry-preserving. We discuss a general concept of structure-preserving operators at the end of Section 4.1.

Theory
We present in this section our theory for structure-preserving GANs. The results are first stated for the special case of learning group-invariant distributions. We then extend the theory to a general class of structure-preserving operators.

Invariant discriminator theorem
We demonstrate under assumptions outlined below and for broad classes of divergences and probability metrics that for Σ-invariant probability measures P, Q we can restrict the test function space (16)], without changing the divergence/probability metric, i.e., The space Γ inv Σ is a much "smaller" and more efficient discriminator space to optimize over in the proposed GANs. We rigorously formulate our results in the following theorem, which first considers the (f, Γ) divergence (7), the Γ-IPM (6), and the Sinkhorn divergence (11). where Proof. We first prove the Theorem for (f, Γ)-divergences. Start by using Jensen's inequality and the convexity of the Legendre transform f * to obtain where in the next to last equality we use Lemma 3.2(c) together with the assumptions P, Q ∈ P Σ (X) to conclude Combining this with S Σ [Γ] ⊂ Γ and (7) we obtain D The proof for Γ-IPMs is similar, but does not require Jensen's inequality due to the linearity of the objective functional in γ. Hence the hypothesis S Σ [Γ] ⊂ Γ is not necessary to obtain W Γ (Q, P ) = W SΣ [Γ] (Q, P ). The proof for Sinkhorn divergences follows similar steps as for the (f, Γ)-divergences; see Appendix B.1 for details.
Theorem 4.1 suggests that the discriminator space reduction effectively acts as an unbiased regularization to prevent discriminator overfitting, a common challenge for GAN optimization in the small data regime. Using invariant discriminators can thus improve the data-efficiency of the model; this will be empirically verified in Tables 1 -3.
2. If the group Σ is finite and the function space Γ ⊂ M b (X) is convex and closed under Σ in the sense of (17), then S Σ [Γ] ⊂ Γ , as readily follows from the definition (19). Our implemented examples in Section 6 fall under this category. 3. The space of 1-Lipschitz functions on a metric space (X, d), assuming the action is 1-Lipschitz, i.e., d(T σ (x), T σ (y)) ≤ d(x, y) for all σ ∈ Σ, x, y ∈ X. 4. The unit ball in an appropriate RKHS; see Lemma 4.13. 5. More generally, if Γ is convex and closed in the weak topology on Γ induced by integration against finite signed measures; see Lemma 4.15 for a proof.

Extension to general objective functionals
Next we show how the proof of Theorem 4.1 can be generalized to a wider variety of objective functionals. This result will utilize a certain topology on the space of bounded measurable functions which we describe in the following definition.
T is a separating vector space of linear functionals on V and we equip V with the weak topology from T (i.e., the weakest topology on V for which every τ ∈ T is continuous). This makes V a locally convex topological vector space with dual space V * = T ; see Theorem 3.10 in Rudin [2006]. In the following we will abbreviate this by saying that V has the M (X)-topology. Theorem 4.3. Let V be a subspace of M b (X) n , n ∈ Z + , that is closed under Σ in the sense of (17) and satisfies If H(·; Q, P ) is concave and upper semi-continuous (USC) in the M (X)-topology on V (see Definition 4.2) and for all σ ∈ Σ, γ ∈ V , and Q, P ∈ P(X) then for all Σ-invariant Q, P we have If, in addition, Remark 4.4. See Section 4.3 for conditions implying S Σ [Γ] ⊂ Γ.

Extension to other structure-preserving operators
Let K x (dx ) be a probability kernel from X to X and define S K : In this setting we have the following generalization of Theorem 4.1.
where D Γ is an (f, Γ)-divergence or a Γ-IPM. It also holds for the Sinkhorn divergence if S K [c(·, y)] = c(·, y) and In addition, if S K is a projection (i.e., Remark 4.6. Theorem 4.5 is an instance of the data processing inequality; see Theorem 2.21 in Birrell et al. [2022]. Proof. We prove (34) for (f, Γ)-divergences. The proofs for Γ-IPMs and Sinkhorn divergences are similar. We note that for Γ-IPMs, (34) does not require the assumption S K [Γ] ⊂ Γ.
Fix Q, P ∈ P K (X) and use Jensen's inequality along with the K-invariance of Q and P to compute . Note that this computation is a special case of the proof of the data processing inequality for (f, Γ)-divergences; see Theorem 2.21 in Birrell et al. [2022]. The assumption S K [Γ] ⊂ Γ implies the reverse inequality, hence we conclude D , are a special case of Theorem 4.5 with the kernel being a regular conditional probability, K = P (·|A). Here Γ inv K is the set of A-measurable functions in Γ, which can be significantly "smaller" than Γ. The case where A = σ(ξ) for some random variable ξ has particular importance in coarse graining of molecular dynamics Noid [2013], Pak & Voth [2018], as we will detail below. The result for Σ-invariant measures, Theorem 4.1, is also special case of Theorem 4.5, where the kernel is K . Alternatively, Lemma 3.2 (c) shows S Σ can be written as a conditional expectation.
Coarse-graining and structure-preserving operators Here we show how to apply our structure preserving formalism, Theorem 4.5, in the context of coarse-graining. We refer to the reviews Noid [2013], Pak & Voth [2018] for fundamental concepts in the coarse-graining of molecular systems. Mathematically, a coarse-graining of the state space X is given by a measurable (non-invertible) map where y = ξ(x) are thought of as the coarse variables and Y as a space of significantly less complexity than X. If A = σ(ξ) is the σ-algebra generated by the coarse-graining map ξ then a function is measurable with respect to A if it is constant on every level set ξ −1 (y).
To complete the description of the coarse-graining one selects a kernel K y (dx), which in the coarse-graining literature is called the back-mapping. The kernel K y (dx) describes the conditional distribution of the fully resolved state x ∈ ξ −1 (y), conditioned on the coarse-grained state y = ξ(x), namely K y (dx) = P (dx|y); in particular K y (dx) is supported on the set ξ −1 (y). The kernel induces naturally a projection S K : then it is uniquely determined by its value on A, in other words it is completely specified by a probability measure Q ∈ P(Y ) on the coarse variable y = ξ(x). We refer to such a Q as a "coarse-grained" probability measure. Once a coarse-grained measure is constructed on Y , see Noid [2013], Pak & Voth [2018] for a rich array of such methods, it can be then "reconstructed" as a measure on X by the kernel K y (dx) as P (dx) = K y (dx)Q(dy). For example, if we take X and Y to be discrete sets we can chose the trivial (uniform) reconstruction kernel with density k y (x) = δ x (ξ −1 (y)) 1 |ξ −1 (y)| and any coarse-grained measure with density q(y) on the coarse variables y is reconstructed on X as a probability density on X: Finally, we note that back-mappings K y (dx) = P (dx|y) in coarse-graining-being probabilities conditioned on the coarse variables-can be constructed, to great accuracy, as generative models using conditional GANs, see Li et al. [2020], Stieffenhofer et al. [2021].

Equivariant generator theorem
Theorem 4.1 provides the theoretical justification for reducing the discriminator space Γ to its Σ-invariant subset Γ inv Σ when the source Q and the generated measure P g are both Σ-invariant. Our next theorem, however, shows that such practice could easily lead to "mode collapse" if one of the two distributions is not Σ-invariant, see Figure 4a; the proof is deferred to Appendix B. Theorem 4.7. Let S Σ [Γ] ⊂ Γ and P, Q ∈ P(X), i.e., not necessarily Σ-invariant. We have where D Γ is an (f, Γ)-divergence or a Γ-IPM.
Remark 4.8. The analogous result for the Sinkhorn divergences also holds if the cost is separately Σ-invariant in each variable, i.e., c(T σ (x), y) = c(x, y) and c(x, T σ (y)) = c(x, y) for all σ ∈ Σ, x, y ∈ X. Though this is not satisfied by most commonly used cost functions and actions one can always enforce it by replacing the cost function c with the symmetrized cost Proof. We prove the result for (f, Γ)-divergences; the proof for Γ-IPMs is similar.
where the first equality is due to Theorem 4.1, and the third equality holds as γ − ν and f * (γ − ν) are both Σ-invariant when γ ∈ Γ inv Σ .
Theorem 4.7 has the following implications: If one uses a Σ-invariant GAN (i.e., invariant discriminators and equivariant generators) to learn a non-invariant data source Q then one will in fact learn the symmetrized version On the other hand, if the data source Q is Σ-invariant (i.e., S Σ [Q] = Q, cf. Lemma 3.2) but the GAN generated distribution P g is not then discriminators from Γ inv Σ alone can not differentiate Q and P g , i.e., D Γ inv Σ (Q P g ) = 0, as long as Q = S Σ [P g ]. This suggests that P g can easily suffer from "mode collapse", as it only needs to equal Q after Σ-symmetrization; we refer readers to Figure 4a (2nd and 4th rows) for a visual illustration, where a unimodal P g can be erroneously selected as the "best" fitting model, even though its Σ-symmetrization S Σ [P g ] should be the "correct" one.
To prevent this from happening, one needs to ensure the generator produces a Σ-invariant distribution P g ; this is guaranteed by the following Theorem. Theorem 4.9. If P Z ∈ P(Z) is Σ-invariant and g : Z → X is Σ-equivariant then the push-forward measure P g := P Z • g −1 is Σ-invariant, i.e., P g ∈ P Σ (X).
Proof. The proof is based on the equivalence of the following commutative diagrams: More specifically, where the third and fifth equalities are due to the equivariance and invariance, respectively, of g and P Z .
We note that equivariant flow-based methods have also been proposed based on a similar strategy to Theorem 4.9. We refer readers to Section 2 for a discussion of related works. Remark 4.10. Suppose g = γ 2 • γ 1 is a composition of two maps, γ 1 : Z → W and γ 2 : W → X. Even if γ 1 is not Σ-equivariant (in fact, Z does not even need to be equipped with a Σ-action T Z σ ), as long as P γ1 ∈ P(W ) is Σ-invariant and γ 2 is Σ-equivariant, the push-forward measure P g ∈ P(X) is still Σ-invariant.
To construct the Σ-invariant noise source required in Theorem 4.9 (or Remark 4.10) one can begin with an arbitrary noise source and use a Σ-symmetrization layer, as described by the following theorem. Theorem 4.11. Let W ∼ µ Σ and N be a Z-valued random variable (i.e., an arbitrary noise source). If N and W are independent then the distribution of Therefore This proves P Σ Z is Σ-invariant as claimed. Remark 4.12. Dey et al. [2021] also proposed to use G-CNNs to generate images with C 4 /D 4 -invariant distributions. However, the first step in their model, i.e., the "Project & Reshape" step [cf. Figure 3], uses a fully-connected layer which destroys the group symmetry in the noise source, leading to non-invariant final distribution P g even if the subsequent layers are all Σ-equivariant. This easily leads to "mode collapse" [cf. Theorem 4.7], which we will empirically demonstrate in Section 6; see, e.g., Figure 4a (4th row). An easy remedy for this is to add a Σ-symmetrization layer: let w be the output of "Project & Reshape"; the Σ-symmetrization layer draws a random σ ∼ µ Σ and transforms w into T W σ (w), producing a Σ-invariant distribution on the layer output (see Theorem 4.11). The final distribution P g is thus Σ-invariant if subsequent layers are all Σ-equivariant by Remark 4.10. See Figure 3 for a visual illustration.

Conditions Ensuring S Σ [Γ] ⊂ Γ
In this section we provide conditions under which the test function space Γ is closed under symmetrization, that being a key assumption in our main results in Section 4. First we show that S Σ [Γ] ⊂ Γ when Γ is the unit ball in an appropriate RKHS. Lemma 4.13. Let V ⊂ M b (X) be a separable RKHS with reproducing-kernel k : X × X → R. Let Γ = {γ ∈ V : γ V ≤ 1} be the unit ball in V . Suppose we have a measurable group action T : Σ × X → X and k is Σ-invariant under this action (i.e., k(T σ (x), T σ (y)) = k(x, y) for all σ ∈ Σ, x, y ∈ X). Then S Σ [Γ] ⊂ Γ. Remark 4.14. The proof will use many standard properties of a RKHS. In particular, recall that the assumption X ⊂ M b (X) implies k is bounded and jointly measurable. See Chapter 4 in Steinwart & Christmann [2008] for this and further background. See Sriperumbudur et al. [2011] and references therein for more discussion of characteristic kernels as well as the related topic of universal kernels.
and k(·, T σ (x)), k(·, T σ (y)) V = k(T σ (x), T σ (y)) = k(x, y) = k(·, x), k(·, y) V for all σ ∈ Σ and x, y ∈ X. Next we will show that the map U σ : γ → γ • T σ is an isometry on V for all σ ∈ Σ, γ ∈ V : It is clearly a linear map. To show its range is contained in V , first recall that the span of {k(·, x)} x∈X is dense in V . Therefore, given γ ∈ V there is a sequence γ n → γ having the form γ n = Nn i=1 a n,i k(·, x n,i ) for some a n,i ∈ R, x n,i ∈ X. Equation (41) implies a n,i k(T σ (·), x n,i ) = Nn i=1 a n,i k(·, T σ −1 (x n,i )) .
Combining Eq. (43) with Eq. (42) we can conclude that γ n •T σ V = γ n V and γ n •T σ −γ m •T σ V = γ n −γ m V . γ n converges in V , hence is Cauchy, therefore γ n • T σ is Cauchy as well. We have assumed V is complete, therefore γ n • T σ →γ for someγ ∈ V . V is a RKHS, hence the evaluation maps are continuous and we findγ(x) = lim n γ n (T σ (x)) = γ(T σ (x)) for all x. Therefore γ • T σ =γ ∈ V and This proves U σ is an isometry on V .
Now fix γ ∈ Γ. We will show that the map σ → U σ [γ] is Bochner integrable (see, e.g., Appendix E in Cohn [2013]): It clearly has has separable range since V was assumed to be separable. By the same reasoning as above, givenγ ∈ V we have a sequenceγ n →γ whereγ n = Nn i=1 a n,i k(·, x n,i ) . Hence which is now clearly measurable in σ due to the measurability of the action. Therefore This proves U σ [γ]µ Σ (dσ) ∈ Γ. Finally, V is a RKHS and so the evaluation maps are in V * . Therefore evaluation commutes with the Bochner integral and we find Hence we can conclude S Σ [γ] ∈ Γ for all γ ∈ Γ as claimed.
The next result provides a general framework for proving S Σ [Γ] ⊂ Γ. Lemma 4.15. Let V ⊂ M b (X) n , n ∈ Z + , be a subspace equipped with the M (X)-topology (see Definition 4.2) and Γ ⊂ V . If Γ is convex and closed, the group action T : Proof. Suppose we have γ ∈ Γ with S Σ [γ] ∈ Γ. As noted in Definition 4.2, V is a locally convex topological vector space with V * = {τ ν : ν ∈ M (X) n }, τ ν (γ) := n i=1 γ i dν i . The separating hyperplane theorem (see Theorem 3.4(b) in Rudin [2006]) applied to A = {S Σ [γ]} and B = Γ therefore implies the existence of ν ∈ M (X) n such that for allγ ∈ Γ. We have assumed Γ is closed under Σ and so we can letγ = γ • T σ to get for all σ ∈ Σ. Integrating with respect to µ Σ (dσ) and using Fubini's theorem to change the order of integration we obtain a contradiction. Therefore S Σ [γ] ∈ Γ as claimed.
We end this section with several examples of function spaces, V , that are useful in conjunction with Lemma 4.15: 1. V = M b (X) n , n ∈ Z + , in which case S Σ [V ] ⊂ V follows from measurability of the action.
2. X is a metric space, the action T : Σ × X → X is continuous, and V = C b (X) n , n ∈ Z + . In this case, S Σ [V ] ⊂ V follows from the dominated convergence theorem.
3. X is a metric space, the action T : Σ × X → X is continuous, T σ is 1-Lipschitz for all σ ∈ Σ, and V = Lip 1 b (X) n , n ∈ Z + . In this case, S Σ [V ] ⊂ V follows from the following calculation: for all γ ∈ Lip 1 b (X).

Primal Formulation of Σ-invariant (f, Γ)-Divergences
In this section we will derive a primal formulation of the (f, Γ)-divergence between Σ-invariant distributions when Γ consists of Σ-invariant discriminators; this will take the form of an infimal convolution formula. Our result is reminiscent of the results in Genevay et al. [2016] for Sinkhorn divergences. Under appropriate assumptions, we will show that the primal optimization problem has a unique solution and will prove the divergence property for (f, Γ)-divergences on P Σ (X).
In this section we will assume that X is a complete separable metric space (with metric d). Our analysis will require the following notion of a determining set of functions. Definition 5.1. Given Q ⊂ P(X), a subset Ψ ⊂ M b (X) will be called Q-determining if for all Q, P ∈ Q, We will also need f and Γ to satisfy one of the following admissibility criteria, as introduced in Birrell et al. [2022]. We will call f ∈ F 1 (a, b) admissible if {f * < ∞} = R and lim y→−∞ f * (y) < ∞ (note that this limit always exists by convexity). If f is also strictly convex at 1 then we will call f strictly admissible. We will call Γ ⊂ C b (X) admissible if 0 ∈ Γ, Γ is convex, and Γ is closed in the M (X)-topology on C b (X) (see Definition 4.2). Γ will be called strictly admissible if it also satisfies the following property: There exists a P(X)-determining set Ψ ⊂ C b (X) such that for all ψ ∈ Ψ there exists c ∈ R, > 0 such that c ± ψ ∈ Γ. Finally, an admissible Γ ⊂ C inv b,Σ (X) (the set of Σ-invariant bounded continuous functions) will be called Σ-strictly admissible if there exists a P Σ (X)-determining set Ψ ⊂ C b (X) such that for all ψ ∈ Ψ there exists c ∈ R, > 0 such that c ± ψ ∈ Γ.
One way to construct a Σ-strictly admissible set is to start with an appropriate strictly admissible set and then restrict to the subset of Σ-invariant functions; see Appendix B.2 for a proof. The following result extends the infimal convolution formula and divergence properties from Birrell et al. [2022] to the case where the models and test-function space are Σ-invariant. Theorem 5.4. Suppose f and Γ are admissible and Γ ⊂ C inv b,Σ (X). For Q, P ∈ P Σ (X) we have the following properties: 1. Infimal Convolution Formula on P Σ (X): In particular,

Existence of an Optimizer
If f is strictly convex then there is a unique such η * . Proof.
1. Part 1 of Theorem 2.15 from Birrell et al. [2022] implies an infimal convolution formula on P(X), hence To prove the reverse inequality, we use the bound , the equality S Σ [Γ] = Γ, and then Theorem 4.7 to compute This proves the infimal convolution formula on P Σ (X).
2. Now suppose D Γ f (Q P ) < ∞. Part 2 of Theorem 2.15 from Birrell et al. [2022] implies there exists η * ∈ P(X) such that We need to show that η * can be taken to be Σ-invariant. To do this, first use the infimal convolution formula to bound The Σ-invariance of Q and P together with Theorem 4.7 imply and Therefore Hence with S Σ [η * ] ∈ P Σ (X) as claimed.
If f is strictly convex then uniqueness is a corollary of the corresponding uniqueness result from Part 2 of Theorem 2.15 in Birrell et al. [2022].

Admissibility of
Letting g = c ± ψ as in the definition of Σ-strict admissiblity we see that for all ψ ∈ Ψ. Ψ is a P Σ (X)-determining set and Q, P ∈ P Σ (X), hence we can conclude that Q = P .
4. We know that D f ≥ 0 and W Γ ≥ 0, therefore the infimal convolution formula implies hence D Γ f (Q P ) = 0. Finally, suppose f is strictly admissible, Γ is Σ-strictly admissible, and D Γ f (Q P ) = 0. Then Part 2 of this theorem implies for some η * ∈ P Σ (X). Both terms are non-negative, hence The P Σ (X)-divergence property for W Γ then implies Q = η * . f being strictly admissible implies that D f has the divergence property, hence η * = P . Therefore Q = P as claimed.

Experiments
We now present experiments on both synthetic and real-world data sets with embedded group symmetry to empirically verify our theory for structure-preserving GANs from Section 4.

Algorithmic Feasibility
Theorems 4.1 and 4.9 imply that one can build invariant GANs by using Σinvariant discriminators, Σ-equivariant generators, and a Σ-invariant noise source. Equivariant networks for arbitrary group symmetry (and gauge invariance) have been studied in recent works such as Cohen & Welling [2016b]. Invariant noise sources can be constructed as shown in Theorem 4.11. We note that the symmetrization operators S Σ , S Σ are only used in the proofs of theoretical properties of the proposed GANs and are not needed in practical implementations. The necessary invariance/equivariance is built into the discriminator/generator via the structure of the layers; see Appendix D.4.

Data sets and common experimental setups
Toy example. Following Birrell et al. [2022], this synthetic data source is a mixture of four 2D t-distributions with 0.5 degrees of freedom, embedded in a plane in R 12 . The four centers of the t-distributions are located (in the supporting plane) at coordinates (±10, ±10), exhibiting C 4 -symmetry [cf. Figure 4a].
RotMNIST is built by randomly rotating the original 10-class 28 × 28 MNIST digits LeCun et al. [1998], resulting in an SO(2)-invariant distribution. We use different portions of the 60,000 training images for experiments in Section 6.4.
ANHIR consists of pathology slides stained with 5 distinct dyes for the study of cellular compositions Borovec et al. [2020]. Following Dey et al. [2021], we extract from the original images 28,407 foreground patches of size 64 × 64. The staining dye is used as the class label for conditioned image synthesis. As the images have no preferred orientation/reflection, the distribution is O(2)-invariant.
LYSTO contains 20,000 patches extracted from whole-slide images of breast, colon and prostate cancer stained with immunohistochemical markers Ciompi et al. [2019]. The images are classified into 3 categories based on the organ source, and we downsize the images to 64 × 64. Similar to ANHIR, this data set is also O(2)-invariant.
Common experimental setups. To verify our theory in Section 4, and to quantify and disentangle the contributions of the structure-preserving discriminator (D) and generator (G) (Theorem 4.1 and Theorem 4.7), we replace the baseline G and/or D by their group-equivariant/invariant counterparts, Eqv G and Inv D, while adjusting the number of filters according to the group size to ensure a similar number of trainable parameters. We also consider the incomplete attempt by Dey et al. [2021] at building equivariant generators ((I)Eqv G), wherein the first fully-connected layer destroys the symmetry in the noise source, resulting in non-equivariant G even if subsequent layers are all equivariant [cf. Remark 4.12]. We use the Fréchet Inception Distance (FID) Heusel et al. [2017] to evaluate the quality and diversity of the GAN generated samples after embedding them in the feature space of a pre-trained Inception-v3 network Szegedy et al. [2016]. Due to the simplicity of RotMNIST, we replace the inception-featurization by the encoding feature space of an autoencoder trained on the rotated digits. We note that, compared to classifiers, autoencoders are guaranteed to produce different features for rotated versions of the same digit; they are thus more suitable to measure sample diversity in rotation.
(a) 2D projection of the generated samples.

Toy Example
We test the performance of different GANs (and their equivariant versions) based on 3 types of divergences, namely the Wasserstein-GAN (WGAN) based on the Γ-IPM Eq. (3), the D fα -GAN based on the classifical f -divergence Eq. (4) and (5), and the D L α -GAN based on the (f, Γ)-divergence Eq. (9), in learning the C 4 -invariant mixture Q. We use fully-connected networks with 3 hidden layers for the baseline G and D (Vanilla G&D). The generator pushes forward a 10D Gaussian noise source, which is itself C 4 -invariant after prescribing a proper group action, e.g., π/2-rotations in the first two dimensions. Equivariant G (Eqv G) and invariant D (Inv D) are built by replacing fully-connected layers with C 4 -convolutional layers based on Theorem 4.9 due to the C 4 -invariance of the noise source. We also mimic the incomplete attempt by Dey et al. [2021] in building equivariant generators ((I)Eqv G) by leaving the first fully-connected layer unchanged and replacing only the subsequent layers by C 4 -convolutions. Figure 4a displays the 2D projection of the generated samples learned by the D L α=2 -GAN (and its equivariant versions) on 200 training samples. It is clear that the baseline model without structural prior (Vanilla G&D) has difficulty in learning Q in such small data regime. Using an Inv D alone without an Eqv G (Vanilla G + Inv D) or with an incorrectly imposed Eqv G ((I)Eqv G + Inv D) leads easily to "mode collapse", validating Theorem 4.7. On the other hand, D L α -GAN with an Eqv G (even without an Inv D) is able to learn all 4 modes of Q. We omit the results of (equivariant) D fα -GANs and WGANs from Figure 4a, as both fail to learn the data source Q; this is unsurprising due to the lack of absolute continuity between Q and P g (the former is supported on a plane, while the latter is the entire 12D space) and the fact that Q is heavy-tailed (as the mean does not exist.) This demonstrates the importance of our framework's broad applicability to a variety of variational divergences, as an improper choice of the divergence-even with structural prior-can fail to learn the source distribution. Values concentrated around zero indicate successful learning of the low-dimensional source distribution, i.e., generating high-fidelity samples. Figure 4b indicates that an Inv D in the D L α -GAN helps produce a distribution with sharper support, whereas Eqv G alone without Inv D tends to generate relatively low-quality samples away from the supporting plane. In contrast, Figure 4c indicates that WGAN (even with symmetry prior) fails to learn the support plane due to Q being heavy-tailed. Results with different numbers of training samples and α's are shown in Appendix C, and the conclusions are similar.

RotMNIST
We adopt a similar setup to Dey et al. [2021]. Specifically, in the baseline G, a fully-connected layer first projects and reshapes the concatenated Gaussian noise and class embedding into a 2D feature map (see Figure 3); spectrallynormalized convolutions Miyato et al. [2018], interspersed with pointwise-nonlinearities, class-conditional batchnormalizations, and upsamplings, are subsequently used to increase the spatial dimension. We note again that replacing 2D convolutions with C n -convolutions does not simply lead to Eqv G, as the distribution after the "project and reshape" layer is no longer C n -invariant. This can be fixed by adding a C n -symmetrization layer after the first linear embedding; see Remark 4.12. We consider GANs with the relative average loss (RA-GANs) Jolicoeur-Martineau [2019] in addition to the D L α -GANs for this experiment. All configurations are trained with a batch size of 64 for 20,000 generator iterations. Implementation details are available in Appendix D. Table 1: The median of the FIDs (lower is better), calculated every 1,000 generator update for 20,000 iterations, averaged over three independent trials. The number of the training samples used for experiments varies from 1% (600) to 10% (6,000) of the RotMNIST training set. See Appendix C for further results.

Loss
Architecture 1% 5% 10% 50% 100%   Table 1 shows the median of the FIDs, calculated every 1,000 generator update, averaged over three independent trials. It is clear that our proposed models (Eqv G + Inv D) consistently achieve significantly improved results compared to the baseline CNN G&D and the prior approach ((I)Eqv G + Inv D); the out-performance is even more pronounced when increasing the group size from Σ = C 4 to C 8 . We note that, similar to RotMNIST, one can also use a custom autoencoder featurization for FID evaluation, and the superiority of our model (Eqv G + Inv D) is even more prominent under such metric: for instance, on ANHIR, the median FIDs calculated through autoencoder featurization of the three comparing models are, respectively, 1221 (CNN G&D), 936 (((I)Eqv G + Inv D)), and 329 (Eqv G + Inv D). See Figure 5 for randomly generated samples by RA-GANs trained with 1% training data. More results are available in Appendix C.

RA-GAN
Figure 5: Randomly generated digits by the RA-GANs trained on RotMNIST after 20K generator iterations with 1% (600) training data. More images are available in Appendix C.

ANHIR and LYSTO
Compared to RotMNIST, ResNet and its D 4 -equivariant counterpart are used instead of CNNs for G and D. All models are trained for 40,000 generator iterations with a batch size of 32. Implementation details are available in Appendix D. Table 2 displays the minimum and median of the FIDs, calculated every 2,000 generator update, averaged over three independent trials. The plus sign "+" after the data set, e.g., ANHIR+, denotes the presence of data augmentation (random 90 • rotations and reflection) during training. It is clear that augmentation usually (but not always) has a positive effect on the results evaluated by the FID; however, our proposed model even without data augmentation still consistently and significantly outperforms the baseline model (CNN G&D) and the prior approach ((I)Eqv G + Inv D) Dey et al. [2021] with augmentation. Figure 6 presents a random collection of real and generated LYSTO images, visually verifying the improved sample fidelity of our model over the baseline. More results are available in Appendix C.

Discussion of empirical findings
Consistently across all experiments, our proposed structure-preserving GAN outperforms prior approaches in generating high-fidelity and diverse samples by a significant margin, in some cases almost an order of magnitude measured in FID.
The results also show that, compared to data-augmentation (a common strategy for learning from limited data), building  theoretically-guided structural probabilistic priors directly into the two GAN players achieves substantially improved performance and data efficiency in adversarial learning.
where Γ is the unit ball in an RKHS Müller [1997], Sriperumbudur et al. [2012]. With suitable choices of Γ, IPMs are able to meaningfully compare not-absolutely continuous distributions, but they could potentially fail at comparing distributions with heavy tails Birrell et al. [2022].
(c) (f, Γ)-divergences. This class of divergences were introduced in Birrell et al. [2022] and they subsume both f -divergences and Γ-IPMs. Given a function f satisfying the same condition as in the definition of the f -divergence and Γ ⊂ M b (X), the (f, Γ)-divergence is defined as where Λ P f [γ] is again given by Eq. (62), implying that Eq. (7) includes as a special case the f -divergence (4) when for any Γ ⊂ M b (X). It is demonstrated in Birrell et al. [2022] that one also has Some notable examples of such Γ's can be found in Birrell et al. [2022], for instance the 1-Lipschitz functions Lip 1 (X), the RKHS unit ball, ReLU neural networks, ReLU neural networks with spectral normalizations, etc. The property (67) readily implies that (f, Γ) divergences can be defined for non-absolutely continuous probability distributions. If X is further assumed to be a complete separable metric space then, under stronger assumptions on f and Γ, one has the following Infimal Convolution Formula: which implies, in particular, ,Eq. (66) and Eq. (67). (d) Sinkhorn divergences. The Wasserstein (or "earth-mover") metric associated with a cost function c : X × X → R + has the variational representation where Co(Q, P ) is the set of all couplings of P and Q and Γ = {γ = (γ 1 , γ 2 ) ∈ C(X) × C(X) : γ 1 (x) + γ 2 (y) ≤ c(x, y) , x, y ∈ X}, with C(X) being the space of continuous functions on X (C b (X) will denote the subspace of bounded continuous functions). The Sinkhorn divergence is given by with W Γ c, (Q, P ) being the entropic regularization of the Wasserstein metrics Genevay et al. [2016], where now Γ = C b (X) × C b (X) and γ 1 ⊕ γ 2 (x, y) := γ 1 (x) + γ 2 (y).

B Proofs
In this appendix we provide proofs of several results that were stated without proof in the main text.
, and c(T σ (x), T σ (y)) = c(x, y) for all σ ∈ Σ, x, y ∈ X then Remark B.2. Note that the classical Sinkhorn divergence is obtained when Γ = C b (X) × C b (X) but the proof of this theorem applies to any Proof. Equation (70) =W SΣ [Γ] c, Using Jensen's inequality followed by Fubini's theorem on the third term we obtain Finally, the Σ-invariance of Q, P , and c imply S Σ [P ] = P , S Σ [Q] = Q, and The reverse inequality follows from Γ inv Σ ⊂ Γ and so the proof is complete.

B.2 Admissibility Lemmas
In this appendix we prove several lemmas regarding admissible test function spaces. First we prove the admissibility properties of Γ inv Σ from Lemma 5.3. Proof.

The zero function is
We have assumed Γ is admissible, hence it is closed. The maps τ ν , ν ∈ M (X) are continuous on C b (X), hence the sets {γ ∈ C b (X) : τ δ Tσ (x) [γ] = τ δx [γ]} are also closed. Therefore Γ inv Σ is closed. This proves Γ inv Σ is admissible.     (200) training data. Our model Eqv G + Inv D, Σ = 8 is the only one that can generate high-fidelity images in this setting. We note that the repetitively generated digits are inevitable in such a small data regime, as the models are forced to learn the empirical distribution of the limited training data (20 images per class). Figure 13: Randomly generated digits by the RA-GANs trained on RotMNIST after 20K generator iterations with 0.33% (200) training data. Our model Eqv G + Inv D, Σ = 8 is the only one that can generate high-fidelity images in this setting. We note that the repetitively generated digits are inevitable in such a small data regime, as the models are forced to learn the empirical distribution of the limited training data (20 images per class).

D.2 RotMNIST
For RA-GAN, the training is stabilized by regularizing the discriminator γ ∈ Γ with a zero-centered gradient panelty (GP) on the real distribution Q in the following form We set the GP weight λ 1 = 0.1 according to Dey et al. [2021]. For the D L α -GAN, we use the one-sided GP as a soft constraint on the Lipschitz constant where ρ g ∼ T X + (1 − T )Y (with X ∼ P g , Y ∼ Q, and T ∼ Unif([0, 1]) all being independent.) The one-sided GP weight is set to λ 2 = 10 according to Birrell et al. [2022]. Unequal learning rates were set to η G = 0.0001 and η D = 0.0004 respectively. The neural architectures for the generators and discriminators are displayed in Table 5 and  Table 6.

D.3 ANHIR and LYSTO
Similar to RotMNIST, the GP weights are set to λ 1 = 0.1 for the RA-GAN in (75) and λ 2 = 10 for the D L α -GAN in (76), and we consider only the case α = 2. The learning rates were set to η G = 0.0001 and η D = 0.0004 respectively. ResNets instead of CNNs are used as baseline generators and discriminators, and the detailed architectural designs are specified in Table 7 and Table 8. Table 5: Generator architectures used in the RotMNIST experiment. ConvSN and C 4 -ConvSN stand for spectrallynormalized 2D convolution and its C 4 -equivariant counterpart. The incomplete attempt at building equivariant generators ((I)Eqv G) does not have the "C 4 -symmetrization" layer. The C 8 -equivariant generator (Eqv G, Σ = C 8 ) is built by replacing "3 × 3 C 4 -ConvSN" with "5 × 5 C 8 -ConvSN" while adjusting the number of filters to maintain a similar number of trainable parameters.