{'fraction_non_alphanumeric': 0.043074993398468445, 'fraction_numerical': 0.009044098230789542, 'mean_word_length': 4.456952449567724, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 8, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unbalanced Sobolev Descent

We introduce Unbalanced Sobolev Descent (USD), a particle descent algorithm for transporting a high dimensional source distribution to a target distribution that does not necessarily have the same mass. We define the Sobolev-Fisher discrepancy between distributions and show that it relates to advection-reaction transport equations and the Wasserstein-Fisher-Rao metric between distributions. USD transports particles along gradient flows of the witness function of the Sobolev-Fisher discrepancy (advection step) and reweighs the mass of particles with respect to this witness function (reaction step). The reaction step can be thought of as a birth-death process of the particles with rate of growth proportional to the witness function. When the Sobolev-Fisher witness function is estimated in a Reproducing Kernel Hilbert Space (RKHS), under mild assumptions we show that USD converges asymptotically (in the limit of infinite particles) to the target distribution in the Maximum Mean Discrepancy (MMD) sense. We then give two methods to estimate the Sobolev-Fisher witness with neural networks, resulting in two Neural USD algorithms. The first one implements the reaction step with mirror descent on the weights, while the second implements it through a birth-death process of particles. We show on synthetic examples that USD transports distributions with or without conservation of mass faster than previous particle descent algorithms, and finally demonstrate its use for molecular biology analyses where our method is naturally suited to match developmental stages of populations of differentiating cells based on their single-cell RNA sequencing profile. Code is available at https://github.com/ibm/usd .


Introduction
Particle flows such as Stein Variational Gradient descent [1], Sobolev descent [2] and MMD flows [3], allow the transport of a source distribution to a target distribution, following paths that progressively decrease a discrepancy between distributions (Kernel Stein discrepancy and MMD, respectively). Particle flows can be seen through the lens of Optimal Transport as gradient flows in the Wasserstein geometry [4], and they've been recently used to analyze the dynamics of gradient descent in overparametrized neural networks in [5] and of Generative Adversarial Networks (GANs) training [2].
Unbalanced Optimal Tansport [6][7][8][9] is a new twist on the classical Optimal Transport theory [10], where the total mass between source and target distributions may not be conserved. The Wasserstein Fisher-Rao (WFR) distance introduced in [7] gives a dynamic formulation similar to the so-called Benamou-Brenier dynamic form of the Wasserstein-2 distance [11], where the dynamics of the transport is governed by an advection term with a velocity field V t and a reaction term with a rate of growth r t , corresponding to the construction and destruction of mass with the same rate: s.t ∂q t (x) ∂t = −div(q t (x)V t (x)) + α r t (x)q t (x), q 0 = q, q 1 = p.
From a particle flow point of view, this advection-reaction in Unbalanced Optimal Transport corresponds to processes of birth and death, where particles are created or killed in the transport from source to target. Particle gradient descent using the WFR geometry have been used in the analysis of over-parameterized neural networks and implemented as Birth-Death processes in [12] and as conic descent in [13]. In the context of particles transportations, [14] showed that birth and death processes can accelerate the Langevin diffusion. On the application side, Unbalanced Optimal Transport is a powerful tool in biological modeling. For instance, the trajectories of a tumor growth have been modeled in the WFR framework by [15]. [16] and [17] used Unbalanced Optimal Transport to find differentiation trajectories of cells during development.
The dynamic formulation of WFR is challenging as it requires solving PDEs. One can use the unbalanced Sinkhorn divergence and apply an Euler scheme to find the trajectories between source and target as done in [18] but this does not give any convergence guarantees.
In this paper we take another approach similar to the one of Sobolev Descent [2]. We introduce the Kernel Sobolev-Fisher discrepancy that is related to WFR and has the advantage of having a closed form solution. We present a particle descent algorithm in the unbalanced case named Unbalanced Sobolev Descent (USD) that consists of two steps: an advection step that uses the gradient flows of a witness function of the Sobolev-Fisher discrepancy, and a reaction step that reweighs the particles according to the witness function. We show theoretically that USD is convergent in the Maximum Mean Discrepancy sense (MMD), that the reaction step accelerates the convergence, in the sense that it results in strictly steeper descent directions, and give a variant where the witness function is efficiently estimated as a neural network. We then empirically demonstrate the effectiveness and acceleration of USD in synthetic experiments, image color transfer tasks, and finally use it to model the developmental trajectories of populations of cells from single-cell RNA sequencing data [16].

Sobolev-Fisher Discrepancy
In this Section we define the Sobolev-Fisher Discrepancy (SF) and show how it relates to advectionreaction PDEs. While this formulation remains computationally challenging, we'll show in Section 3 how to approximate it in RKHS.

Advection-Reaction with no Conservation of Mass
Definition 1 (Sobolev-Fisher Discrepancy). Let p, q be two measures defined on X ⊂ R d . For α > 0, the Sobolev-Fisher Discrepancy is defined as follows: Note that the objective of SF is an Integral Probability Metric (IPM) objective, and the function space imposes a constraint on the weighted Sobolev norm of the witness function f on the support of the distribution q. We refer to q as the source distribution and p as the target distribution. The following theorem relates the solution of the Sobolev-Fisher Discrepancy to an advection-reaction PDE: Theorem 1 (Sobolev-Fisher Critic as Solution of an Advection-Reaction PDE). Let u be the solution of the advection-reaction PDE: , with witness function f * p,q = u/SF(p, q).
From Theorem 1 we see that the witness function of SF 2 solves an advection-reaction where the mass is transported from q to p, via an advection term following the gradient flow of ∇ x u, and a reaction term amounting to construction/destruction of mass that we also refer to as a birth-death process with a rate given by u. Intuitively, if the witness function u(x) > 0 we need to create mass, and destruct mass if u(x) < 0. This is similar to the notion of particle birth and death defined in [12] and [14].
In Proposition 1 we give a convenient unconstrained equivalent form for SF 2 : Proposition 1 (Unconstrained Form of SF 2 ). SF satisfies the expression: SF 2 (p, q) = sup u L(u), Theorem 2 gives a physical interpretation for SF 2 as finding the witness function u that has minimum sum of kinetic energy and rate of birth-death while transporting q to p via advection-reaction: Theorem 2 (Kinetic Energy & Birth-Death rates minimization). Consider the following minimization: We then have that P = 1 2 SF 2 (p, q), and moreover: Remarks. a) When α = 0 we obtain the Sobolev Discrepancy, or p − q Ḣ−1 (q) , that linearizes the Wasserstein-2 distance. b) Note that this corresponds to a Beckman type of optimal transport [19], where we transport q to p (q and p do not have the same total mass) via an advection-reaction with mass not conserved. It is easy to see that X (p(x) − q(x))dx = α X u(x)q(x)dx.

Advection-Reaction with Conservation of Mass
Define the Sobolev-Fisher Discrepancy with conservation of mass: The only difference between the previous expression and SF 2 in Proposition 1 is that the variance of the witness function is kept under control, instead of the second order moment. Defining one can similarly show that SF has the primal representation: Hence, we see that SF is the minimum sum of kinetic energy and variance of birth-death rate for transporting q to p following an advection-reaction PDE with conserved total mass. The conservation of mass comes from the fact that χ(x) = −div(q(x)∇ x u(x)) + α(u(x) − E x∼q u(x))q(x) satisfies:

Kernel Sobolev-Fisher Discrepancy
In this section we turn to the estimation of SF discrepancy by restricting the witness function to a Reproducing Kernel Hilbert Space (RKHS), resulting in a closed-form solution.

Estimation in Finite Dimensional RKHS
Consider the finite dimensional RKHS, corresponding to an m dimensional feature map Φ:  When γ = 0 this corresponds to the unbalanced case, i.e. birth-death with no conservation of total mass, while for γ = 1 we have birth-death with conservation of total mass.

Kernel SF for Direct Measures
Consider direct measures p = N i=1 a i δ xi and q = n j=1 b j δ yj (with no conservation of mass we can have i a i = j b j = 1 ). An estimate of the Sobolev-Fisher critic is given byû λ,γ p,q = (D(q) + αĈ γ (q) + λI m ) −1 (μ(p) −μ(q)), where the empirical Kernel Mean Embeddings areμ

Unbalanced Continuous Kernel Sobolev Descent
Given the Kernel Sobolev-Fisher Discrepancy defined in the previous sections and its relation to advection-reaction transport, in this section we construct a Markov process that transports particles drawn from a source distribution to a target distribution. Note that we don't assume that the densities are normalized nor have same total mass.

Constructing the Continuous Markov Process
Given α, λ > 0, γ ∈ {0, 1} and n weighted particles drawn from the source distribution : Recall that the target distribution is given by p = N i=1 a i δ xi . We define the following Markov Process that we name Unbalanced Kernel Sobolev Descent: where u λ,γ p,q n t is the critic of the Kernel Sobolev-Fisher discrepancy, whose expression and gradients are given in Proposition 2. We see that USD consists of two steps: the advection step that updates the particles positions following the gradient flow of the Sobolev-Fisher critic, and a reaction step that updates the weights of the particles with a growth rate proportional to that critic. This reaction step consists in mass construction or destruction, that depends on the confidence of the witness function. This can be seen as birth-death process on the particles, where the survival log probability of a particle is proportional to the critic evaluation on this particle.

Generator Expression and PDE in the limit of n → ∞
Proposition 3 gives the evolution equation of a functional of the intermediate distributions q n t produced in the descent, at the limit of infinite particles n → ∞: Proposition 3. Let Ψ : P(X ) → R, be a functional on the probability space. Let q n t be the distribution produced by USD at time t. Let q t be its limit as n → ∞, we have: Where the functional derivative D µ is defined through first variation for a signed measure χ ( χ(x)dx = 0): In particular, the paths of USD in the limit of n → ∞ satisfy the advection-reaction equation:

Unbalanced Sobolev Descent decreases the MMD.
The following Theorem shows that USD when the number of the particles goes to infinity decreases the MMD distance at each step, where: Theorem 3 (Unbalanced Sobolev Descent decreases the MMD). Consider the paths q t produced by USD. In the limit of particles n → ∞ we have In particular, in the regularized case Similarly to [2], strict descent is ensured if the kernel and the target distribution p satisfy the condition: USD Accelerates the Convergence. We now prove a Lemma the can be used to show that Unbalanced Sobolev Descent has an acceleration advantage over Sobolev Descent [2]. Lemma 1. In the regularized case λ > 0 with α > 0, the Kernel Sobolev-Fisher Discrepancy SF H,γ,λ is strictly upper bounded by the Kernel Sobolev discrepancy S H,λ (i.e for α = 0) [2]: From Lemma 1 and Eq. (3), we see that USD (α > 0), results in a larger decrease in MMD than SD [2] (α = 0), resulting in a steeper descent. Hence, USD advantages over SD are twofold: 1) it allows unbalanced transport, 2) it accelerates convergence for the balanced and unbalanced transport.
USD with Universal Infinite Dimensional Kernel. While we presented USD with a finite dimensional kernel for ease of presentation, we show in Appendix D that all our results hold for an infinite dimensional kernel. For a universal or a characteristic kernel, convergence in MMD implies convergence in distribution (see [21,Theorem 12]). Hence, using a universal kernel, USD guarantees the weak convergence as MMD(p, q t ) → 0.

Understanding the effect of the Reaction Step: Whitened Principal Transport Directions
In [2] it was shown that the gradient of the Sobolev Discrepancy can be written as a linear combination of principal transport directions of the Gramian of derivatives D(q). Here we show that unbalanced descent leads to a similar interpretation in a whitened feature space thanks to the 2 regularizer. Let It is easy to see that the critic of the SF can be written as: . Note thatΦ q is a whitened feature map andD(q) is the Gramian of its derivatives. Letd j , λ j be the eigenvectors and eigenvalues ofD(q). We have: v λ,γ p,q = m j=1 1 λj +αd j d j ,δ p,q . Hence, we write the gradient of the Sobolev-Fisher critic as ∇ x u λ,γ p,q (x) = m j=1 . This says that the mass is transported along a weighted combination of whitened principal transport directions ∇ xdj (x). α introduces a damping of the transport as it acts as a spectral filter on the transport directions in the whitened space.

Discrete time Unbalanced Kernel and Neural Sobolev Descent
In order to get a practical algorithm in this Section we discretize the continuous USD given in Eq. (2). We also give an implementation parameterizing the critic as a Neural Network.
Discrete Time Kernel USD. Recall that the source distribution q 0 = q = n j=1 b j δ yj , note w 0 j = b j and x 0 j = y j , j = 1 . . . n. The target distribution p = N j=1 a j δ xj , and assume for simplicity N j=1 a j = 1. Let ε > 0, for = 1 . . . L, for j = 1 . . . n, we discretize the advection step: ). For τ > 0, similarly we discretize the reaction step as: If γ = 0 (total mass not conserved) we define the reweighing as follows: w j = exp(a j ) and if γ = 1 (mass conserved): w j = exp(a j )/ n i=1 exp(a i ), and finally : q = n j=1 w j δ x j . Neural Unbalanced Sobolev Descent. Motivated by the use of neural network critics in Sobolev Descent [2], we propose a Neural variant of USD by parameterizing the critic of the Sobolev-Fisher Discrepancy as a Neural network f ξ trained via gradient descent with the Augmented Lagrangian Method (ALM) on the loss function of SF given in Definition 1. The re-weighting is defined as in the kernel case above. Neural USD with re-weighting is summarized in Algorithm 1 in Appendix B. Note that the re-weighting can also be implemented via a birth-death process as in [12]. In this variant, particles are duplicated or killed with a probability driven by the growth rate given by the critic. We give the details of the implementation as birth-death process in Algorithm 2 (Appendix B).
Computational and Sample Complexities. The computational complexity Neural USD is given by that of updating the witness function and particles by SGD with backprop, i.e. O(N (T + B)), where N is the mini-batch size, T is the training time, B is the gradient computation time for particles update. T corresponds to a forward and a backward pass through the critic and its gradient. The sample complexity for estimating the Sobolev Fisher critic scales like 1 / √ N similar to MMD [22]. Table 1 in Appendix A summarizes the main differences between Sobolev descent [2], which only implements advection, and USD that also implements advection-reaction. Our work is related to the conic particle descent that appeared in [13] and [12]. The main difference of our approach is that it is not based on the flow of a fixed functional, but we rather learn dynamically the flow that corresponds to the witness function of the Sobolev-Fisher discrepancy. The accelerated Langevin Sampling of [14] also uses similar principles in the transport of distributions via Langevin diffusion and a reaction term implemented as a birth-death process. The main difference with our work is that in Langevin sampling the log likelihood of the target distribution is required explicitly, while in USD we only need access to samples from the target distribution. USD relates to unbalanced optimal transport [6][7][8][9] and offers a computational flexibility when compared to Sinkhorn approaches [8,9], since it scales linearly in the number of points while Sinkhorn is quadratic. Compared to WFR (Eq. (1)), USD finds greedily the connecting path, while WFR solves an optimal planning problem.

Applications
We experiment with USD on synthetic data, image coloring and prediction of developmental stages of scRNA-seq data. In all our experiments we report the MMD distance with a gaussian kernel, computed using the random Fourier features (RF) approximation [23] with 300 RF and kernel bandwith equal to √ d (the input dimension). We consider the conservation of mass case, i.e. γ = 1.
Synthetic Examples. We test Neural USD descent (Algorithms 1 and 2) on two synthetic examples.
In the first example (Figure 1), the source samples are drawn from a 2D standard Gaussian, while target samples are drawn from a Mixture of Gaussians (MOG). Samples from this MOG have uniform weights. In the second example (Figure 2), source samples are drawn from a 'cat'-shaped density whereas the target samples are drawn uniformly from a 'heart'. Samples from the targets have non-uniform weights following a horizontal gradient. In order to target such complex densities USD exploits advection and reaction by following the critic gradients and by creation and destruction of mass. We see in Figs 1 and 2 a faster mixing of USD in both, implementation with weights (w) and as birth-death (bd) processes compared to the Sobolev descent algorithm of [2].
(a) Neural USD paths in transporting a Gaussian to a MOG. We compare Sobolev descent (SD, [2]) to both USD implementations: with birthdeath process (bd: Algorithm 2) and weights (w: Algorithm 1). USD outperforms SD in capturing the modes of the MOG.  (a) Neural USD transporting a 'cat' distributed cloud to a 'heart'. The main difference with the example above is that the points of the target distribution have non uniform weights describing a linear gradient as seen from the color code in the figure. Similarly to the MOG case, USD outperforms SD and better captures the non uniform density of the target.  Image Color Transfer. We test Neural USD on the image color transfer task. We choose target images that have sparse color distributions. This is a good test for unbalanced transport since intuitively having birth and death of particles accelerates the transport convergence in this case. We compare USD to standard optimal transport algorithms. We follow the recipe of [24] as implemented in the POT library [25], where images are subsampled for computational feasibility and then interpolated for out-of-sample points. We compare USD to Earth-Moving Distance (EMD), Sinkhorn [26] and Unbalanced Sinkhorn [8] baselines. We see in Figure 3 that USD achieves smaller MMD to the target color distribution. We give in Appendix H.2 in Fig 7 trajectories of the USD.
Developmental Trajectories of Single Cells. When the goal is not only to transport particles but also to find intermediate points along trajectories, USD becomes particularly interesting. This type of use case has recently received increased attention in developmental biology, thanks to single-cell RNA sequencing (scRNA-seq), a technique that records the expression profile of a whole population of cells at a given stage, but does so destructively. In order to trace the development of cells inbetween such destructive measurements, [16] proposed to use unbalanced optimal transport [8]. Denoting those populations q t0 (source) and q t1 (target), then, in order to predict the population at an intermediate time t0+t1 2 , [16] used a linear interpolation between matches between the source and target populations based on the coupling of unbalanced OT. This type of interpolation is a form of McCann interpolate [27]. As an alternative, we propose to use the mid-point of the USD descent as an interpolate, i.e. the timestamp in the descent t1 /2 such that MMD(q t1 /2 , q t0 ) = MMD(q t1 /2 , q t1 ). We test this procedure on the dataset released by [16]. For all time intervals [t 0 , t 1 ] in the dataset, we compute the intermediate stage q t1 /2 . We compare the quality of this interpolate with that obtained by the WOT algorithm of [16] in terms of MMD to the ground truth intermediate population q * t1 /2 , normalized by MMD between initial and final population, i.e. MMD(q t1 /2 , q * t1 /2 )/MMD(q t0 , q t1 ). Fig. 4 gives mean and standard deviation of the normalized MMD between intermediate stages predicted by USD and the ground truth. Note that mean and standard deviations are computed across 35 time intervals, individual MMDs can be found in Figure 8 in Appendix H. From Figure 4 we see that USD outperforms WOT in MMD, since USD is designed to decrease the MMD distance. On the other hand, for fairness of the evaluation we also report Normalized EMD (Earth-Mover Distance, normalized similarly) for which WOT outperforms USD. This is not surprising since WOT relies on unbalance OT, while USD instead provides guarantees in terms of MMD.

Conclusion
In this paper we introduced the KSFD discrepancy and showed how it relates to an advection-reaction transport. Using the critic of KSFD, we introduced Unbalanced Sobolev Descent (USD) that consists in an advection step that moves particles and a reaction step that re-weights their mass. The reaction step can be seen as birth-death process which, as we show theoretically, speeds up the descent compared to previous particle descent algorithms. We showed that the MMD convergence of Kernel USD and presented two neural implementations of USD, using weight updates, and birth and death of particle, respectively. We empirically demonstrated on synthetic examples and in image color transfer, that USD can be reliably used in transporting distributions, and indeed does so with accelerated convergence, supporting our theoretical analysis. As a further demonstration of our algorithm, we showed that USD can be used to predict developmental trajectories of single cells based on their RNA expression profile. This task is representative of a situation where distributions of different mass need to be compared and interpolated between, since the different scRNA-seq measurements are taken on cell populations of dissimilar size at different developmental stages. USD can naturally deal with this unbalanced setting. Finally we compared USD to unbalanced OT algorithms, showing its viability as a data-driven, more scalable dynamic transport method.

Broader Impact Statement
Our work provides a practical particle descent algorithm that comes with a formal convergence proof and theoretically guaranteed acceleration over previous competing algorithms. Moreover, our algorithm can naturally handle situations where the objects of the descent are particles sampled from a source distribution descending towards a target distribution with different mass.
The type of applications that this enables range from theoretically principled modeling of biological growths processes (like tumor growth) and developmental processes (like the differentiation of cells in their gene expression space), to faster numerical simulation of advection-reaction systems.
Since our advance is mainly theoretical and algorithmic (besides the empirical demonstrations), its implications are necessarily tied to the utilization for which it is being deployed. Beside the applications that we mentioned, particle descent algorithms like ours have been proposed as a paradigm to characterize and study the dynamics of Generative Adversarial Network (GANs) training. As such, they could indirectly contribute to the risks associated with the nefarious uses of GANs such as deepfakes. On the other hand, by providing a tools to possibly analyze and better understand GANs, our theoretical results might serve as the basis for mitigating their abuse.

Supplementary Material: Unbalanced Sobolev Descent .1 Relation to Unbalanced Optimal Transport
We now relate our definition of the Sobolev-Fisher discrepancy to the following norm. For a signed measure χ define χ It can be shown that SF 2 (p, q) = p − q 2Ḣ −1,2 (q) . The dynamic formulation of the Wasserstein Fisher-Rao metric given in Equation (1) can therefore be compactly written as: From this connection to WFR through .

2Ḣ
−1,2 (q) , we see the link of the Sobolev-Fisher discrepancy to unbalanced optimal transport, since it linearizes the WFR for small perturbations. Table   α γ Markov Process Particles j = 1 . . . n PDE (As n → ∞) Guarantee

A Summary
Whitened Principal Transport Directions : Whitened Principal Transport Directions :

B Algorithms
Critic Parameters Update (between particles updates, gradient descent on the critic is initialized from previous episodes) Particles and Weights Update for j = 1 to n do ) (current f ξ is the critic between q −1 and p, advection step) ) (current f ξ is the critic between q −1 and p )

C Proofs
Proof of Theorem 1. Define the following dot product between u, v in the the Sobolev Space: and the norm : Let f be any function such that f | ∂X =0 , and f W 2 0 ≤ 1: , we have f * p,q W 2 0 = 1 and hence feasible, and it is easy to see that : , and hence we have that for all f feasible we have: , and hence f * p,q achieves the sup.
Proof of Proposition 1. This can be easily proved using that u * solution of the PDE with source term is solution of that sup problem. L(u * ) = SF 2 (p, q) is clear from definition of u * we are left showing L(u) ≤ L(u * ) for all u, this can be shown by proving that : and hence L(u) ≤ L(u * ) , hence u * achieves the sup.
Proof of Theorem 2. Writing the Lagrangian u we have: where By convexity of the cost we exchange sup and inf for L(V, r, u) Similarly we have: inf r Hence the dual problem is : By Proposition 1 ,we have : Hence SF 2 (p, q) has the equivalent form : Subject to: p(x) − q(x) = −div(q(x)V (x)) + αr(x)q(x) Since V * = ∇ x u and r * = u we have finally: Proof of Proposition 2.
Setting first order optimality for the sup we obtain: Proof of Proposition 3. For simplicity we give here the proof for γ = 1. γ = 0 has a similar proof. The proof follows ideas from [12]. Let Ψ be a measure valued functional Ψ : P(R d ) → R. For a measure µ, Ψ(µ) ∈ R. The functional derivative D µ is defined through first variation for a signed measure χ ( χ(x)dx = 0): A generator function is defined as follows for a measure valued markov process µ (n) t (defined with n particles) is defined as follows: is the expectation of the functional Ψ evaluated on the trajectory of the markov process µ 2. Since the process is markovian letting t → 0 and considering the generator it will give us the evolution also between t and t + dt of Ψ[µ (n) t ]: 3. Consider n → ∞, identify the PDE corresponding to the generator As s → 0, and ε → 0, we have: The advection part: For the weight update part note that we have: Hence the variation of Φ: As s, ε → 0 we obtain the effect of weights updates as follows: Hence the Generator has the following form: , ∇ x D q n Ψ(x) )q n (dx)+α D q n Ψ(x)(u p,q n (x)−E q (n) u p,q n )q n (x)dx and we have: t ], withq (n) 0 = q As n → ∞ we have the evolution of the PDE: Proof of Theorem 3 (Decrease of the MMD loss of the (Continous) Gradient Flow). For u γ,λ p,qt we omit the up-scripts γ and λ in the following. Note that we have the following two expressions using the fact our functions are in the RKHS: ∇ x u p,qt (x), ∇ x δ p,qt q t (dx) = u p,qt , (JΦ(x)) JΦ(x)δ p,qt q t (dx) = u p,qt , E qt (JΦ(x)) (JΦ(x))δ p,qt = u p,qt , D(q t )δ p,qt .
Notice that A = D + λI m and B = αC γ are both symmetric positive definite (SPD). Because the inverse of a SPD matrix is itself a SPD matrix, B −1 is SPD. Because the product of SPD matrices is itself SPD, AB −1 A is SPD. Because the inverse of the sum of SPD matrices is itself SPD, E is SPD.
Equation (5) then implies: which, together with the definitions of SF 2 H,γ,λ and S 2 H,λ , concludes the proof.

D.3 USD with Infinite dimensional Kernel decreases the MMD distance
Theorem 3 holds for the infinite dimensional case. To see that it is enough to replace in the proof of Theroem 3 finite dimensional operators and embeddings D(q), C γ (q), µ(q) with their infinite dimensional counterparts given in Equation in Equations (6), (7) and (8) [2], out_features=1, bias=False) ) (a) Neural Unbalanced Sobolev Descent paths in transporting a Gaussian to circles). We compare Sobolev descent (SD, [2]) to both USD implementations with birth and death processes (bd: Algorithm 2) as well as the weighted version implementation (w: Algoritm 1, note that in this case we overlay the points with their respective weights where coloring density encodes the weights). We see that birth and death processes helps USD to outperform SD in capturing the two modes.  (a) Neural Unbalanced Sobolev Descent paths in transporting a disk to a heart/spiral. We compare Sobolev descent (SD, [2]) to both USD implementations with birth and death processes (bd: Algorithm 2) as well as the weighted version implementation (w: Algoritm 1, note that in this case we overlay the points with their respective weights where coloring density encodes the weights). We see that birth and death processes helps USD to outperform SD in capturing the two modes. (b) MMD function of the time in the descent from a disk to a heart/spiral: We see that birth and death processes in both implementations of USD accelerate the convergence to the target distribution and reaches lower MMD than Sobolev Descent that relies on advection only. Figure 6: Neural Unbalanced Sobolev Descent transporting a 'disk' to a 'heart' weighted by a spiral-shaped gradient.