{'fraction_non_alphanumeric': 0.05444394723944682, 'fraction_numerical': 0.017366660274030984, 'mean_word_length': 4.414167051222889, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Quasi-Conservative Score-based Generative Models

Existing Score-based Generative Models (SGMs) can be categorized into constrained SGMs (CSGMs) or unconstrained SGMs (USGMs) according to their parameterization approaches. CSGMs model the probability density functions as Boltzmann distributions, and assign their predictions as the negative gradients of some scalar-valued energy functions. On the other hand, USGMs employ flexible architectures capable of directly estimating scores without the need to explicitly model energy functions. In this paper, we demonstrate that the architectural constraints of CSGMs may limit their score-matching ability. In addition, we show that USGMs' inability to preserve the property of conservativeness may lead to serious sampling inefficiency and degraded sampling performance in practice. To address the above issues, we propose Quasi-Conservative Score-based Generative Models (QCSGMs) for keeping the advantages of both CSGMs and USGMs. Our theoretical derivations demonstrate that the training objective of QCSGMs can be efficiently integrated into the training processes by leveraging the Hutchinson trace estimator. In addition, our experimental results on the Cifar-10, Cifar-100, ImageNet, and SVHN datasets validate the effectiveness of QCSGMs. Finally, we justify the advantage of QCSGMs using an example of a one-layered autoencoder.


INTRODUCTION
Score-based Generative Models (SGMs) are parameterized functions for estimating scores, which are vector fields corresponding to the gradients of log probability density functions. According to their parameterization approaches, SGMs can be categorized into constrained or unconstrained SGMs (Salimans & Ho, 2021).
Constrained SGMs (CSGMs), also known as Energy-Based Models (EBMs), model probability density functions as Boltzmann distributions, and assign their predictions as the negative gradients of some scalar-valued energy functions (Salimans & Ho, 2021). CSGMs are able to ensure the conservativeness of their output vector fields. This property is essential in guaranteeing that the Metropolis-Hasting decisions used in the sampling process are determined based on the probability ratio between two consecutive sampling steps (Salimans & Ho, 2021). This, in turn, is necessary to ensure that the sample distribution converges to the true data distribution. Such a concept has been explored by the researchers of (Salimans & Ho, 2021;Alain & Bengio, 2014;Nguyen et al., 2017;Chen et al., 2014). However, this parameterization approach requires specific model designs, limiting the choices of model architectures for CSGMs. For example, the authors of (Vincent, 2011;Kamyshanska & Memisevic, 2013) proposed to restrict a CSGM to be a one-layered autoencoder with symmetric weights in its linear layer, which hinders its ability to be extended to more sophisticated architectures such as convolution neural networks. On the other hand, the authors of (Salimans & Ho, 2021;Saremi et al., 2018; divided a CSGM into two halves: the first half explicitly parameterizes the negative energy function, while the second half is generated by automatic differentiation tools (Martens et al., 2012) to output the estimated scores. Nevertheless, these methods limit that the output of the first half can only be a scalar, and the second half has to be generated using automatic differentiation tools.
In contrast, unconstrained SGMs (USGMs) employ flexible architectures capable of directly estimating the scores without the need of modeling the energy functions. Due to their architectural Preprint Version flexibility, USGMs have been extensively utilized in contemporary machine learning tasks such as image generation Ho et al., 2020;Song & Ermon, 2020;Song et al., 2021b;Nichol & Dhariwal, 2021) and audio generation (Lam et al., 2022;Kong et al., 2021;Chen et al., 2021). Among these works, the authors in (Song et al., 2021b) proposed a unified framework based on a USGM, which achieved remarkable performance on several benchmarks. Their success demonstrated that architectural flexibility can be beneficial for SGMs. However, in spite of their empirical benefit, our analyses in Section 3 indicate that USGMs' inability to ensure conservativeness may lead to degraded sampling performance.
To preserve both the conservativeness of CSGMs and the architectural flexibility of USGMs, we propose Quasi-Conservative Score-based Generative Models (QCSGMs). Instead of constraining the model architecture, QCSGMs resort to enhancing the conservativeness of USGMs through minimizing a regularization loss. Our theoretical derivations demonstrate that such a regularization term can be integrated into the training processes of SGMs efficiently through the Hutchinson trace estimator (Hutchinson, 1989). Moreover, our experimental results showcase that the performance of Noise Conditional Score Network++ (NCSN++) (Song et al., 2021b), which is considered the state-of-the-art USGM, can be further improved by incorporating our regularization method on the Cifar-10, Cifar-100, ImageNet-32x32, and SVHN datasets.

BACKGROUND AND RELATED WORKS
In this section, we walk through the background material and the related works for understanding the contents of this paper. We first introduce a number of score matching methods for training an SGM. Next, we describe the sampling algorithms for generating samples through an SGM. Lastly, we elaborate on the conservative property of SGMs, and the differences between CSGMs and USGMs.

SCORE MATCHING METHODS
Given a true data distribution p data , its empirical distribution p 0 (x) is established through sampling M independent and identically distributed D-dimensional vectors {x (i) : x (i) ∈ R D } M i=1 , represented as a Dirac delta distribution, i.e., p 0 (x) 1 M M i=1 δ( x − x (i) ). To ensure the probability density function (pdf) is everywhere non-zero and differentiable, previous literature (Vincent, 2011) proposed to replace p 0 (x) with Parzen density estimator p σ (x) x p σ (x|x)p 0 (x)dx, where p σ (x|x) is an isotropic Gaussian smoothing kernel with a standard deviation σ. When σ > 0, the score function of p σ (x) has a closed form (Chao et al., 2022), which can be formulated as: . (1) Score matching (Hyvärinen, 2005) describes the learning process to approximate the score function ∂ ∂x log p σ (x) in Eq. (1) using a neural network s(· ; θ) : R D → R D , which is parameterized by θ and is trained through minimizing the Explicit Score Matching (ESM) objective expressed as follows: Eq.
(2) involves the explicit calculation of Eq. (1), which suffers from serious training inefficiency when the dataset size M is large. To address this issue, an alternative method called Implicit Score Matching (ISM) (Hyvärinen, 2005), which excludes ∂ ∂x log p σ (x) in the training objective, was introduced to efficiently train s(x; θ). ISM employs an equivalent loss L ISM expressed as follows: where ∂ ∂x s(x; θ) corresponds to the Jacobian matrix of s(x; θ), and tr (·) denotes the trace of a matrix. Although L ISM avoids the calculation of Eq. (1), the calculation of tr ∂ ∂x s(x; θ) in Eq. (3) still requires D times of backpropagations , which hinders L ISM 's ability of being utilized in high-dimensional context. To alleviate it, a scalable objective, called Sliced Score Matching (SSM)  loss, was proposed to approximate tr ∂ ∂x s(x; θ) in L ISM with the Hutchinson trace estimator (Hutchinson, 1989). Given a random vector v drawn from a distribution p(v) satisfying E p(v) vv T = I, the Hutchinson trace estimator replaces the trace of a square matrix A with E p(v) v T Av , which can be derived as: The above derivation suggests that tr ∂ ∂x s(x; θ) in Eq.
(3) can be substituted with , resulting in an equivalent objective L SSM expressed as follows: The vector-Jacobian product v T ∂ ∂x s(x; θ) can be calculated with a single backward propagation using automatic differentiation (Martens et al., 2012), and the expectation can be approximated using The Denoising Score Matching (DSM) (Vincent, 2011) loss is another scalable objective formulated based on the Parzen density estimator, which further prevents the computational overhead incurred by the backward propagation in L SSM : where ∂ ∂x log p σ (x|x) = 1 σ 2 (x −x). Since the computational cost of L DSM is relatively low in comparison to the other score matching losses, it has been widely adopted in contemporary modeling methods Song et al., 2021b;a) that pursue training efficiency.

SAMPLING PROCESS
Given an optimal SGM s(x; θ) = ∂ ∂x log p σ (x), ∀x ∈ R D that minimizes the score-matching objectives (i.e., Eqs. (2), (3), (5), and (6)), Langevin dynamics (Roberts & Tweedie, 1996;Roberts & Rosenthal, 1998) enables p σ (x) to be iteratively approximated through the following equation: where α is the step size, t is the timestep, z t ∈ R D is a noise vector sampled from a normal distribution N (0, I). Under the condition where α → 0 and T → ∞,x T can be generated as if it is directly sampled from p σ (x) (Roberts & Rosenthal, 1998;Welling & Teh, 2011). In practice, however, α > 0 and T < ∞, which violate the convergence guarantee. Fortunately, the convergence guarantee can be recovered by the Metropolis-Hasting decisions (Roberts & Tweedie, 1996;Roberts & Rosenthal, 1998), which adjust the updates in Eq. (7) by the acceptance ratio p σ (x t+1 )/p σ (x t ). This can be estimated using the discretized line integral through s(· ; θ) (Alain & Bengio, 2014) as: . The Metropolis-Hastings acceptance decisions are determined when the value of min{1, p σ (x t+1 )/p σ (x t )} is greater than a random noise sampled from a uniform distribution. Otherwise, the update is rejected.
The sampling process that involves the iterative updates in Eq. (7) and the adjustment in Eq. (8) is called Metropolis-Adjusted Langevin Algorithm (MALA). This method was adopted in traditional score-based generative modeling methods (Alain & Bengio, 2014;Nguyen et al., 2017), but becomes less popular in contemporary deep neural network (DNN) based methods due to the computational overhead of Eq. (8). In fact, recent studies Song et al., 2021b) followed a different route and proposed to extend the Langevin dynamics to time-inhomogeneous variants by making s(· ; θ) and α dependent on t. They demonstrated that such time-inhomogeneous variants can achieve superior results on several benchmarks within limited timesteps without the assistance of the Metropolis-Hasting adjustment. In Appendix A.4, we elaborate on the implementation details of such a time-inhomogeneous sampling process.

CONSERVATIVENESS AND ROTATION DENSITY OF A SCORE-BASED GENERATIVE MODEL
A vector field is said to be conservative if it can be written as the gradient of a scalar function (Im et al., 2016). As proved in (Im et al., 2016), the output vector field of an SGM s(x; θ) is said to be conservative over a smooth and simply-connected domain S ⊆ R D if and only if its Jacobian is symmetry for allx ∈ S, which can be equivalently expressed as the zero-rotation-density (ROT ij ) (Glotzl & Richters, 2020) condition expressed as follows: where ∂ ∂xj s(x; θ) i corresponds to the gradient of the i-th element of s(x; θ) with respect to the j-th element ofx. ROT ij s(x; θ) in Eq. (9) describes the infinitesimal circulation of s(x; θ) aroundx.
For CSGMs, p σ (x) is modeled as a Boltzman distribution p(x; θ) = exp (−E(x; θ)) /Z(θ), where exp (·) indicates the exponential function, E(· ; θ) : R D → R represents a scalar-valued energy function, and Z(θ) refers to the partition function. Therefore, the output vector field of a CSGM can be represented as s(x; θ) = ∂ ∂x log p(x; θ) = − ∂ ∂x E(x; θ). This implies that s(x; θ) is conservative. In other words, s(x; θ) satisfies the zero-rotation-density condition in Eq. (9), since the mixed second derivatives of E(x; θ) are equivalent (Alain & Bengio, 2014), which can be shown as the following: On the other hand, since USGMs do not follow the aforementioned modeling procedure, the conservativeness of USGMs is not guaranteed. Although it is possible to ensure the conservativeness of an USGM under an ideal scenario that s(x; θ) perfectly models ∂ ∂x log p σ (x) for allx ∈ R D , a trained USGM typically contains approximation errors in practice. This suggests that USGMs are non-conservative in most cases, and do not satisfy the zero-rotation-density condition.

MOTIVATIONAL EXAMPLES
In this section, we demonstrate the importance of preserving the conservativeness as well as the architectural flexibility of SGMs. In addition, we provide the motivation behind the adoption of QCSGMs through two motivational experiments.

THE INFLUENCES OF NON-CONSERVATIVENESS ON SAMPLING EFFICIENCY
The sampling processes described in Section 2.2 are formulated under an ideal scenario that s(x; θ) = ∂ ∂x log p σ (x), ∀x ∈ R D . In practice, however, a trained USGM contains approximation errors, which could lead to its failure in preserving its conservativeness, as stated in Section 2.3. In the following example, we show that the non-conservativeness of a USGM may cause serious sampling inefficiency even in the presence of slight approximation errors. Given a target distribution p σ (x) = N (0; σ 2 I) and a small constant > 0, a USGM s (x) with its approximation errors less than , i.e., L ESM < , can be expressed as the following non-conservative vector field: where µ(x) = 1 10 10 i=1 N ([2 cos( 2iπ 10 ), 2 sin( 2iπ 10 )] T ; I) is a mixture of isotropic Gaussians, and x = [x 1 , x 2 ] T is a two-dimensional vector. The derivation of Eq. (11) and the rationale behind the formulation of s (x) are provided in Appendix A.2.1. To quantitatively evaluate the non-conservativeness of s (x), we measure the magnitude of ROT ij s (x) using two metrics: the asymmetry metric Asym ∈ [0, ∞) and its normalized variant NAsym ∈ [0, 1], defined as follows: where J = ∂ ∂x s (x), · F is the Frobenius norm. The details of NAsym are elaborated in Appendix A.3. We leverage the precision (P) and recall (R) metrics (Kynkäänniemi et al., 2019) to examine the sampling performance. Fig. 1 (a) depicts the trends of precision and recall during the sampling processes under different choices of . It is observed that the number of steps required for achieving the same sampling performance increases rapidly as the values of Asym and NAsym grow. This suggests that the nonconservativeness of s (x) may decelerate the sampling processes even when L ESM is bounded by . In addition, the visualized trajectories shown in Fig. 1 (b) further reveal that the existence of approximation errors in s (x) may incur rotational vector fields tangent to the true score, leading to inefficient updates during the sampling processes. The above experimental evidences thus demonstrate that the non-conservativeness of a USGM may potentially cause negative impacts on the sampling efficiency. In other words, the non-conservativeness of a USGM may lead to degraded sampling performance within limited timesteps.

THE IMPACTS OF ARCHITECTURAL FLEXIBILITY ON SCORE-MATCHING ACCURACY AND SAMPLING PERFORMANCE
In this experiment, we inspect the influence of architectural flexibility on the training and sampling processes. For a fair evaluation, a USGM s U and a CSGM s C are implemented as neural networks consisting of the same number of parameters. Following the approach described in (Salimans & Ho, 2021), these two models are represented as follows: where f : R D → R D is a neural network, and θ U and θ C are the parameters. We then compare the conservativeness, the score-matching ability, and the sampling performance of both s U and s C , which are trained independently on three two-dimensional datasets. The conservativeness is measured using Asym and NAsym. The sampling performance is measured using the precision and recall metrics, where the number of the sampling timesteps is fixed at 1,000. Finally, the score-matching ability is evaluated using the score-matching error , where s can be either s C or s U , and ∂ ∂x log p σ (x) is calculated based on Eq. (1). In this experiment, f (·; θ U ) and f (·; θ C ) are modeled using three-layered multilayer perceptrons (MLPs), which are the same as those used Table 1: The evaluation results of CSGMs, USGMs, and QCSGMs in terms of their means and variances of five independent runs on the '8-Gaussian,' 'Spirals,' and 'Checkerboard' datasets, which are detailed in Appendix A.5.1. L denotes 'Langevin dynamics,' and M represents 'MALA.' The arrow symbols ↑ / ↓ indicate that higher / lower values correspond to better performance, respectively. Please note that the values of Asym and NAsym of CSGMs are non-zero due to floating-point errors.
The experimental insights in Sections 3.1 and 3.2 shed light on two essential issues to be further explored and addressed. First, although USGMs benefit from their architectural flexibility, their non-conservativeness may lead to serious sampling inefficiency and degraded sampling performance in practice. Second, despite that CSGMs are conservative, their architectural requirement may limit their score-matching ability. Based on the above observations, this paper intends to investigate a new type of SGMs, called Quasi-Conservative Score-based Generative Models (QCSGMs), which are developed to maintain both the conservativeness as well as the architectural flexibility. As revealed in Table 1, QCSGMs are able to achieve improved results in terms of score-matching accuracy and sampling performance in comparison to CSGMs and USGMs under most of the experimental setups.
In the next section, we elaborate on the formulation and our implementation of QCSGMs.

METHODOLOGY
In this section, we introduce QCSGMs and present an efficient implementation of them. In Section 4.1, we describe the learning objective of QCSGMs, and derive its scalable variant. In Section 4.2, we detail the training procedure for QCSGMs, and discuss the time complexity of this implementation.

QUASI-CONSERVATIVE SCORE-BASED GENERATIVE MODELS
Instead of following the concept of CSGMs to ensure the conservativeness through architecture constraints, QCSGMs resort to penalizing the non-conservativeness through a regularization loss. The training objective for QCSGMs is defined as L Total , which is expressed as the following equation: where L SM can be any one of the score-matching objectives (i.e., Eqs. (2), (3), (5), or (6)), L QC represents the regularization term reflecting the non-conservativeness, and λ is a balancing factor. As discussed in Section 3.1, the non-conservativeness of an SGM can be measured using the magnitude of its rotation densities in the Frobenius norm (i.e., Eq. (12)), suggesting a formulation of L QC as: where J = ∂ ∂x s(x; θ). This objective function, however, requires D times of backpropagations to explicitly calculate the Jacobian matrix of s(x; θ). In order to reduce the computational cost, we first formulate an equivalent objective L tr QC , and then utilize the Hutchinson trace estimator to approximate L tr QC . The loss of L tr QC is derived in Appendix A.2.2, and is formulated as follows: By applying the Hutchinson trace estimator to both tr JJ T and tr (JJ) according to Eq. (4), L tr QC can be equivalently replaced by an another objective L est QC , which is expressed as the following: Additionally, the computational graph of v T JJ T v − v T JJv can be efficiently constructed without increasing the asymptotic time complexity with respect to D, which is later explained in Section 4.2. As a result, under such an implementation, the computational cost of L est QC is significantly lower than L QC and L tr QC when K D.

THE TRAINING PROCEDURE OF QCSGMS
In this subsection, we walk through the proposed training procedure of QCSGMs. This procedure is detailed in Algorithm 1, and the corresponding computational graph is illustrated in Fig. 2. The entire training procedure is divided into five steps, denoted as Steps (1)∼(5), respectively. Steps (1)∼(3) describe the forward propagation process of L Total (θ), which is depicted by the black arrows in Fig. 2. Steps (4) and (5) correspond to the backpropagation processes of the two gradient components comprising ∂ ∂θ L Total (θ), which are named the primary and secondary components, and are depicted as the blue and red arrows in Fig. 2, respectively. The detailed formulations for these two components and the rationale behind such a two-step backpropagation process are further elaborated in Appendix A.2.3. The details of Steps (1)∼(5) are explained as follows.
(1) Computing v T JJ T v. First, v T J is computed by performing backpropagation of v T s(x; θ) with respect tox via automatic differentiation, which is depicted as the upper 'Auto. Diff.' block in Fig. 2.
(2) Computing v T JJv. First, sg v T J s(x; θ) is calculated by taking the inner product between sg v T J and s(x; θ), where the stop-gradient operator sg [·] is applied to v T J to detach it from the computational graph built in Step (1). Then, sg v T J J is calculated by differentiating sg v T J s(x; θ) via performing backpropagation. Stopping the gradient of v T J is necessary to ensure that the automatic differentiation (i.e., the lower 'Auto. Diff.' block in Fig. 2) excludes the computational graph used for differentiating v T J, allowing v T JJ to be correctly derived. Lastly, sg v T J Jv is obtained by taking the inner product of sg v T J J and v.
Algorithm 1 Training Procedure of QCSGM (3) Computing L Total (θ). Based on the results of Steps (1) and (2), L est QC (θ) is computed by taking the expectation of (v T JJ T v − sg v T J Jv). Meanwhile, the score matching loss L SM (θ) can be derived using any one of the Eqs.
(4) Deriving the primary component of  Fig. 2 using automatic differentiation. Note that these gradients are not equal to ∂ ∂θ L Total (θ) due to the adoption of the stop-gradient operator sg [·] in Step (2). As a result, an additional secondary gradient component, which is derived in Step (5), is included to compensate it.
(5) Deriving the secondary component of ∂ ∂θ L Total (θ). The secondary component of ∂ ∂θ L Total (θ) is derived by performing backward propagation through the paths in the computational graph highlighted by the red arrows in Fig. 2 using automatic differentiation. By accumulating the gradients of the primary and the secondary components, the gradients ∂ ∂θ L Total (θ) can be correctly calculated.  Based on the above implementation, the computation of L est QC does not require D times of backpropagation, justifying the computational efficiency of L est QC over L QC and L tr QC . We summarize this section with the time complexity of different training objectives discussed in this paper in Table 2. In this table, H denotes the dimension of the largest hidden layer in s(· ; θ), L denotes the number of layers in s(· ; θ), M denotes the dataset size, D denotes the data dimension, and K denotes the number of random vectors used in the Hutchinson trace estimator. Please note that a reasonable assumption for deep generative tasks is M D K Grathwohl et al., 2019).

EXPERIMENTAL RESULTS ON THE REAL-WORLD DATASETS
In this section, we examine the effectiveness of the proposed QCSGMs on four real-world datasets: Cifar-10, Cifar-100 (Krizhevsky & Hinton, 2009), ImageNet-32x32 (Van Oord et al., 2016), and SVHN (Netzer et al., 2011) datasets. We employ the unconstrained architecture as well as the training procedure adopted by NCSN++ (VE) (Song et al., 2021b) as our baseline. On the other hand, QC-NCSN++ (VE), a variant of NCSN++ (VE) regularized by L est QC , is compared against NCSN++ (VE) using the precision (P), recall (R), Asym, NAsym, Fréchet Inception Distance (FID) (Heusel et al., 2017), and Inception Score (IS) (Barratt & Sharma, 2018) metrics. The Asym and NAsym metrics are calculated based on Eqs. (A12) and (A13) for inspecting the non-conservativeness of the SGMs. Please note that the details of the experimental setups are provided in Appendix A.5.2. Table 3 presents the quantitative results of NCSNs++ (VE) and QC-NCSNs++ (VE) on the Cifar-10, Cifar-100, ImageNet-32x32, and SVHN datasets. It is observed that QC-NCSN++ (VE) is able to achieve better results on the Asym and NAsym metrics than the NCSN++ (VE) baseline on both datasets, indicating that minimizing the proposed L est QC indeed enhances the conservativeness of QC-NCSN++ (VE). Moreover, QC-NCSN++ (VE) can outperform the baseline in terms of the FID, IS, P, and R metrics under a majority of the experimental setups, suggesting that minimizing the proposed L est QC is able to improve the sampling performance. The above results thus demonstrate the effectiveness of the proposed regularization term L est QC on the real-world datasets. In Appen-  dices A.6.5 and A.6.6, we offer more quantitative and qualitative results, and provide a comparison of NCSN++ (VE) and QC-NCSN++ (VE) in terms of their sampling efficiency.

QCSGM IMPLEMENTED AS A ONE-LAYERED AUTOENCODER
A line of research (Vincent, 2011;Kamyshanska & Memisevic, 2013;Im et al., 2016;Kamyshanska & Memisevic, 2015) focuses on a type of SGM constructed as a one-layered autoencoder, since its property of conservativeness can be systematically analyzed. Such an SGM is represented as s(x; θ) = Rh(W Tx + b) + c, where h(·) is an activation function, b, c ∈ R D , R, W ∈ R D×H are the weights of s(· ; θ) (i.e., θ = {R, W, b, c}), and H is the width of the hidden-layer. As proved in (Im et al., 2016), the output vector field of s(· ; θ) is conservative if and only if W R T = RW T . To ensure the conservativeness of such an SGM, a number of works (Vincent, 2011;Kamyshanska & Memisevic, 2013;2015) follow the concept of CSGMs and restrict the weights of s(· ; θ) to be 'tied,' i.e., W = R. An SGM with tied weights, however, is only a sufficient condition for its conservativeness, rather than a necessary one. This implies that there must exist some conservative s(· ; θ) that cannot be modeled using tied weights (Im et al., 2016).
Instead of enforcing an SGM's weights to be tied (i.e., W = R), QCSGMs indirectly learn to satisfy the conservativeness condition (i.e., W R T = RW T ) through minimizing L QC . Fig. 3 depicts the trends of W R T − RW T F and W − R F during the minimization process of L QC . As the training progresses, the values of W R T − RW T F approach zero, indicating that s(· ; θ) learns to output a conservative vector field through minimizing L QC . In contrast, the values of W − R F do not decrease to zero, revealing that minimizing L QC does not necessarily lead to W = R. The experimental results thus suggest that QCSGMs can learn to output conservative vector fields that cannot be modeled by one-layered autoencoders with tied weights. This justifies the advantage of QCSGMs over CSGMs. In Appendix A.6.1, we offer more examples to support this observation.

CONCLUSION
In this paper, we unveiled the underlying issues of CSGMs and USGMs, and highlighted the importance of preserving both of the architectural flexibility and the property of conservativeness through two motivational experiments. We proposed a new category of SGMs, named QCSGMs, in which the magnitudes of their rotation densities are minimized through a regularization loss for enhancing their property of conservativeness. We showed that such a regularization loss can be reformulated as a scalable variant based on the Hutchinson trace estimator, and demonstrated that it can be efficiently incorporated into the training procedure of SGMs. Finally, we validated the effectiveness of QCSGMs through the experimental results on the real-world datasets, and showcased the advantage of QCSGMs over CSGMs using the example of a one-layered autoencoder.

A APPENDIX
In this Appendix, we first provide the definitions for the symbols used in the main manuscript and the Appendix in Section A.1. Next, we detail the backpropagation processes described in Section 4.2, and provide the derivations for Eqs. (11) and (17) in Section A.2. Then, we offer a discussion on the normalized asymmetry metric in Section A.3. Subsequently, in Section A.4, we describe the approach to extend a QCSGM to the time-inhomogeneous variant, i.e., QC-NCSN++ (VE) described in Section 5 of the main manuscript. Finally, we provide the detailed experimental configurations in Section A.5, and a number of qualitative and quantitative experimental results in Section A.6.

A.1 LIST OF NOTATIONS
In this section, we offer the list of notations used throughout the main manuscript and the Appendix. T the number of discretized timesteps for the sampling algorithm.
α the step size used in Langevin dynamics.
the score-matching error described in Section 3.1.
σ the standard deviation for Gaussian distribution.
θ the parameters of an SGM.
x ∈ R D a data sample.
x ∈ R D a perturbed data sample.
z ∈ R D a noise vector used in Langevin dynamics.
v ∈ R D a random vector used in the Hutchinson trace estimator.
b, c ∈ R D the bias for the one-layered autoencoder described in Section 6.
W, R ∈ R D×H the weights for the one-layered autoencoder described in Section 6.
{xt} T t=1 a set of discretized timesteps in Langevin dynamics.
a set of discretized points of the t-th timestep used for estimating line integral.
a set of random vectors drawn from p(v).
p data the unknown true probability density function (pdf).
2σ 2 x−x 2 the smoothing kernel with mean x and standard deviation σ used in Parzen density estimator.
p(v) a distribution satisfying E p(v) [vv T ] = I such as a Gaussian or a Radamacher distribution.
Euclidean norm of a vector u.
Frobenius norm of a matrix A.
To justify this, we first demonstrate the derivation of s (x), and show that s (x) satisfies L ESM < .
As suggested in (Schey, 1997), a non-conservative vector field can be constructed by perturbing a conservative vector field with a unit tangent vector field, which can be established through rotating each vector in the conservative vector field by 90 degrees clockwise. For example, given the true score function ∂ ∂x log p σ (x) = 1 σ 2 [−x 1 , −x 2 ] T of a Gaussian distribution, the associated unit tangent vector field can be written as u(x) = 1 x [−x 2 , x 1 ] T . Based on such an approach, the non-conservative vector field s (x) can be constructed by mixing ∂ ∂x log p σ (x) and u(x), expressed as the following: In Eq. (A1), the unit tangent vector field u(x) is scaled by a scaling factor √ 2 such that s (x) satisfies the constraint L ESM = . However, s (x) in Eq. (A1) is undefined atx = [0, 0] T . To resolve this issue, we substitute the denominator of u(x) with x 2 + , and reformulate s (x) as the following: This modification causes L ESM < . To redistribute the errors of s (x) in Eq. (A2), a factor µ(x)/p σ (x) is included to adjust the magnitude of u(x), where µ(x) is an arbitrary probability density function (pdf). This suggests that s (x) can be further rewritten into the following form.
To show that s (x) in Eq. (A3) satisfies L ESM < , we provide the following proposition. Proposition 1. Given > 0, p σ (x) = N (0; σ 2 I), and an arbitrary pdf µ(x), s ( In Section 4.1, we derived the computationally efficient objective L est QC based on the assumption that L QC equals L tr QC . To show that the equivalence holds, we provide a formal derivation as follows. Proposition 2. L QC (θ) = L tr QC (θ). Proof.
Remark 1. Proposition 2 can also be proved by utilizing the properties of trace, i.e., A 2 F = tr A T A and tr (AB) = tr (BA), leading to a simplified proof as follows:

A.2.3 THE DERIVATION OF THE PRIMARY AND SECONDARY COMPONENTS OF ∂ ∂θ L Total (θ)
In Section 4.2, we decompose ∂ ∂θ L Total (θ) as the primary and secondary components, and separately derive them in Steps (4) and (5). To further elaborate on such a backward propagation process, we offer a detailed description in this subsection. For the sake of notational simplicity, we assume that both the batch size and the number of random vectors K are 1.
According to the rule of sum and the rule of product from vector calculus, the gradient of the total loss ∂ ∂θ L Total (θ) can be decomposed as the sum of ∂ ∂θ L SM (θ), λ ∂ ∂θ v T JJ T v, −λ( ∂ ∂θ v T J)Jv, and −λv T J( ∂ ∂θ Jv), indexed as (i)∼(iv) respectively. The derivation is shown as the following: .
We name the sum of (i)∼(iii) the primary component of ∂ ∂θ L Total (θ), and the term (iv) the secondary component of ∂ ∂θ L Total (θ). Such a decomposition suggests that ∂ ∂θ L Total (θ) can be separately computed based on the computational graph built in Steps (1)∼(3) as shown in the upper subplot of Fig. A1. For the primary component, the sum of (i)∼(iii) is computed by performing backward propagation through the paths in the computational graph highlighted by the blue arrows in the lower subplot of Fig. A1 using automatic differentiation. For the secondary component, the term (iv) is calculated by performing backward propagation through the red arrows in the lower subplot of Fig. A1. Through these two steps, ∂ ∂θ L Total (θ) can be correctly derived.

A.2.4 THE EQUIVALENCE BETWEEN ZERO-ROTATION-DENSITY CONDITION AND THE CONSERVATIVENESS OF A SCORE MODEL
Lemma 1. ROT ij s(x; θ) = 0 for all 1 ≤ i, j ≤ D if and only if the Jacobian of s(x; θ) (i.e., J) is symmetric.
Proof. As defined in Eq. (9), the following holds: Lemma 2. According to (Im et al., 2016), the Jacobian of s(x; θ) (i.e., J) is symmetric if and only if s(x) is conservative. Proof. Based on Lemma 1 and Lemma 2, the proof completes.

A.3 NORMALIZED ASYMMETRY METRIC
In this section, we elaborate on the formulation of the normalized asymmetry metric NAsym, which was introduced in Section 3.1 of the main manuscript. In addition, we derive a computationally efficient implementation of NAsym using the Hutchinson trace estimator. (1) (2) Auto. Diff. (4) and (5) (4)

Operation
The goal of backward propagation Figure A1: The computational graphs of L Total in QCSGMs. The upper and lower subplots depict the forward and backward propagation processes, respectively. The 'Auto. Diff.' blocks represent the operation of differentiating u T s(x; θ), where u is a constant vector with respect tox.
Derivation of the NAsym Metric. As described in (Andrilli & Hecker, 2016), any matrix A can be uniquely decomposed into a symmetric matrix A sym and a skew-symmetric matrix A skew as follows: Based on Eq. (A4), the Jacobian J of an SGM s(x; θ) can be written as the sum of a symmetric matrix J sym = (J + J T )/2 and a skew-symmetric matrix J skew = (J − J T )/2. Under such a definition, the NAsym metric introduced in Section 3.1 can be formulated as follows: This metric measures the ratio of the squared Frobenius norm of the skew-symmetric matrix J skew 2 F to the squared Frobenius norm of the Jacobian matrix J 2 F , and falls within the range [0, 1]. NAsym = 1 corresponds to the condition where J skew dominates J, implying that J is skewsymmetric. On the contrary, NAsym = 0 indicates that J only contains the symmetric component J sym , suggesting that J is symmetric. Since the squared Frobenius norm of the skew-symmetric matrix can be written as the sum of the squared rotation densities of s(x; θ), i.e., J skew can be adopted to measure the non-conservativeness of s(x; θ), as mentioned in Section 3.1.
An Efficient Implementation of NAsym. Since Eq. (A5) involves the explicit calculation of the Jacobian matrix J, evaluating the NAsym metric for a single instancex requires D times of backward propagations. This indicates that the evaluation cost may grow significantly when D and a test set size become large. To reduce the evaluation cost, we utilize the Hutchinson trace estimator to approximate Algorithm A1 Predictor-Corrector (PC) Sampler 1:x = N (0, σ 2 max I) 2: for t = T − 1 to 0 do 3: the NAsym metric based on the following derivation: where can be approximated using K random vectors. In addition, the terms v T JJ T v and v T JJv in Eq. (A6) can be efficiently calculated based on Steps (1) and (2) described in Section 4.2. This suggests that the computational cost of evaluating NAsym can be significantly reduced when K D. In Section A.5.2, we describe an approach to measure NAsym on the Cifar-10 and ImageNet-32x32 datasets in detail.

A.4 TIME-INHOMOGENEOUS QCSGMS
In this section, we demonstrate how a QCSGM is converted to its time-inhomogeneous variant QC-NCSN++ (VE), which was described in Section 5 of the main manuscript. We first explain the modifications made in the sampling process. Then, we elaborate on the corresponding adjustments in the score-matching objective and the regularization loss.
Sampling Process. QC-NCSN++ (VE) adopts the Predictor-Corrector (PC) sampler identical to that employed in NCSN++ (VE) (Song et al., 2021b), which is a time-inhomogeneous sampling algorithm as shown in Algorithm A1. In this sampler, SGM and step size are respectively represented as s(· ; θ, σ t ) and α t = ∂ ∂t σ 2 t , where σ t is a time-dependent standard deviation. In QC-NCSN++ (VE) and NCSN++ (VE), σ t is set to σ min (σ min /σ max ) t T (Song et al., 2021b), where T is the total number of timesteps in the sampling process, σ min is a constant representing a minimal noise scale, and σ max is a constant denoting a maximal noise scale.
Training Objectives. Since the above time-inhomogeneous sampling process requires the SGM s(· ; θ, σ t ) to be conditioned on a time-dependent standard deviation σ t , the training objectives of s(· ; θ, σ t ) have to be modified accordingly. For example, the score-matching objective L DSM used in QC-NCSN++ (VE) and NCSN++ (VE) is modified as follows: where U(t) is a uniform distribution defined on the interval [0, T ], and λ(t) is a time-dependent coefficient for balancing the loss functions of different t. Meanwhile, the regularization term L est QC used in QC-NCSN++ (VE) is adjusted according to λ(t), which is formulated as follows: where J = ∂ ∂x s(x; θ, σ t ).

A.5 EXPERIMENTAL SETUPS
In this section, we elaborate on the experimental configurations and provide the detailed hyperparameter setups for the experiments presented in Sections 3 and 5 of the main manuscript.

A.5.1 EXPERIMENTAL SETUPS FOR THE MOTIVATIONAL EXAMPLES
Datasets. The motivational experiments in Section 3 are performed on the 8-Gaussian, Spirals, and Checkerboard datasets as shown in Fig. A2 (a). The data points of the 8-Gaussian dataset are sampled from eight separate Gaussian distributions centered at (cos( πw 4 ), sin( πw 4 )), where w ∈ {1, ..., 8}. The data points of the Spirals dataset are sampled from two separate curves Lastly, the data points of the Checkerboard dataset are sampled from (4w − 2, t − 2s + w mod 2), where w ∈ [0, 1], s ∈ {0, 1}, · is a floor function, and mod represents the modulo operation. For all of these three datasets, p 0 (x) is established by sampling 50,000 data points (i.e., M = 50, 000).
Training and Implementation Details. The network architecture of f is a three-layered multilayer perceptron (MLP) with (128, 64, 32) neurons and Swish (Ramachandran et al., 2017) as its activation function. This model architecture is the same as that used in the two-dimensional experiments of (Chao et al., 2022). Please note that we do not adopt the architecture in (Salimans & Ho, 2021) since sophisticated architectures such as ResNet and U-Net employed in (Salimans & Ho, 2021) are not suitable for low-dimensional datasets. The SGMs s U and s C are trained utilizing the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 1 × 10 −2 and a batch size of 5,000 for 100,000 iterations. The balancing factor λ and the standard deviation σ are fixed to 5 and 0.5, respectively.
Evaluation Method. The precision and recall metrics are calculated using 10,000 sample points. On the other hand, the Asym, NAsym metrics, and the score errors are approximated based on the following formulas: where D denotes a set of 1,600 grid points. A visualization of D is depicted in Fig. A2 (b).

A.5.2 EXPERIMENTAL SETUPS FOR THE EVALUATIONS ON THE REAL-WORLD DATASETS
Datasets. The experiments presented in Section 5 are performed on the Cifar-10, Cifar-100 (Krizhevsky & Hinton, 2009), ImageNet-32x32 (Van Oord et al., 2016), and SVHN (Netzer et al., 2011) datasets. The training and test sets of Cifar-10 and Cifar-100 contain 50,000 and 10,000 images, respectively. The training and test sets of SVHN contain 73,257 and 26,032 images, respectively. On the other hand, the training and the test sets of ImageNet-32x32 consist of 1,281,149 and 49,999 images, respectively.
Training and Implementation Details. NCSN++ (VE) and QC-NCSN++ (VE) are both implemented using the Pytorch framework. NCSN++ (VE) and QC-NCSN++ (VE) are trained using the Adam optimizer with a learning rate of 2 × 10 −4 . The batch size is fixed to 128, while the value of K for QC-NCSN++ (VE) is fixed to 1. The training procedure of QC-NCSN++ (VE) consists of two stages. In the first stage, QC-NCSN++ (VE) is optimized according to the score matching objective, which requires 600,000 training iterations for convergence. In the second stage, the regularization term L est QC is incorporated during the training process, which requires 150,000 training iterations for convergence. The maximal and minimal noise scales σ max and σ min are set to 50 and 0.01, respectively. The total number of timesteps T in the sampling process is set to 1,000. The balancing factor λ is set to 0.0001.
Evaluation Method. The asymmetry Asym and normalized asymmetry NAsym metrics are evaluated on 100 discretized timesteps on the test sets of both datasets. Specifically, the Asym metric is calculated based on the following equation: where D represents the test set, and T = { i 100 T } 100 i=1 . On the other hand, the NAsym metric is evaluated based on the following equation: The expectations E p(v) [·] in Eqs. (A12) and (A13) are estimated with K = 1. The metrics for sampling performance (i.e., FID, IS, precision and recall) are evaluated using the tensorflow gan library as well as the official evaluation package implemented by (Kynkäänniemi et al., 2019;Naeem et al., 2020).

A.6 ADDITIONAL EXPERIMENTAL RESULTS
In this section, we provide a number of additional experimental results. In Section A.6.1, we present additional experimental results of QCSGMs implemented as one-layered autoencoders to support our observation presented in Section 6 of the main manuscript. In Section A.6.2, we provide a comparison between USGMs, CSGMs, and QCSGMs in terms of their performance during the training time. In Section A.6.3, we demonstrate the impact of the model architecture on the performance of USGMs and CSGMs. In Section A.6.4, we demonstrate the impact of the choices of λ on the performance of QCSGMs. Finally, in Sections A.6.5 and A.6.6, we provide additional quantitative and qualitative results on the real-world datasets.
A.6.1 QCSGMS IMPLEMENTED AS ONE-LAYERED AUTOENCODERS In Section 6, we leveraged the example of an one-layered autoencoder s(x; θ) = Rh(W Tx + b) + c to demonstrate the advantage of QCSGMs over CSGMs. Our experimental results in Fig. 3 reveals that QCSGMs can learn to output conservative vector fields, which cannot be captured by CSGMs with tied weights (i.e., R = W ). To further solidify our assumption, we provide additional examples in Fig. A3. Fig. A3 depicts the trends of W R T − RW T F and W − R F during the minimization process of L QC with four different seeds. As the training progresses, L QC and W R T − RW T F both approach zero in all of these four examples. In contrast, the values of W − R F do not approach zero, and the trends of W − R F for these four examples differ. The above experimental evidences demonstrate that QCSGMs can learn to output conservative vector fields with R = W , and thus justify the advantage of QCSGMs over CSGMs.
To further showcase the benefit of adopting QCSGMs over CSGMs, we include a comparison of QCSGMs and CSGMs in terms of their score matching ability. Fig. A4 shows that QCSGMs demonstrate lower score errors in comparison to CSGMs when both of them are implemented as one-layered autoencoders and trained on a Gaussian distribution.

Steps
Steps Steps Figure A3: The trends of W R T − RW T F and W − R F during the minimization process of L QC . The 'steps' on the x-axes refer to the training steps.
Score Error QCSGM CSGM Figure A4: The trends of W R T − RW T F , W − R F , L QC , and the score error of QCSGM and CSGM during the minimization process of L Total . The 'step' on the x-axes stands for the training steps. The curves depict the mean and 95% confidence interval of three times of training. In this experiment, CSGM and QCSGM are implemented using one-layered autoencoders.  In Section 3.2, we show that USGMs and QCSGMs have better empirical performance in terms of the score-matching accuracy in comparison to CSGMs. To demonstrate that this statement holds under different implementations of f , we provide a performance comparison of these SGMs with different architectural settings in Table A3. Table A3 compares the performance of CSGMs, USGMs, and QCSGMs. All of them are constructed as three-layered MLPs with different number of neurons in each layer. It is observed that USGMs and QCSGMs are able to outperform CSGMs in terms of score errors with a noticeable margin. This result demonstrates the advantage of USGMs and QCSGMs over CSGMs, and highlights the importance of preserving the architectural flexibility.

A.6.4 THE IMPACT OF THE CHOICES OF λ ON THE PERFORMANCE OF QCSGMS
Conservativeness. The first and second rows of Table A4 compare the conservativeness of USGM, CSGM, and QCSGMs trained under different values of λ in terms of the Asym and NAsym metrics on the Checkerboard dataset. The experimental results demonstrate that the conservativeness of QCSGMs is improved (i.e., Asym and NAsym decrease) as the value of λ increases, and therefore validate the effectiveness of L QC .

A.6.5 A COMPARISON OF SAMPLING EFFICIENCY ON THE CIFAR-10 DATASET
The experimental results presented in Section 3.1 indicate that the non-conservativeness of an SGM may degrade its sampling performance under a fixed number of sampling steps. To further examine the sampling inefficiency issue, we perform experiments on the Cifar-10 dataset and compare the sampling performance of NCSNs++ (VE) and QC-NCSNs++ (VE) under different numbers of sampling steps. The evaluation results are shown in Table A5. It is observed that QC-NCSNs++ (VE) are able to outperform NCSNs++ (VE) in terms of the precision and recall metrics. The experimental results therefore justify the effectiveness of L QC on the real-world dataset.
A.6.6 VISUALIZED EXAMPLES Figs. A6-A9 depict a few uncurated visualized examples that qualitatively demonstrate the sampling quality of NCSN++ (VE) and QC-NCSN++ (VE) on the real-world datasets.