{'fraction_non_alphanumeric': 0.03785835342419975, 'fraction_numerical': 0.010646170004535363, 'mean_word_length': 4.67131447136862, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DeFlow: Learning Complex Image Degradations from Unpaired Data with Conditional Flows

The difficulty of obtaining paired data remains a major bottleneck for learning image restoration and enhancement models for real-world applications. Current strategies aim to synthesize realistic training data by modeling noise and degradations that appear in real-world settings. We propose DeFlow, a method for learning stochastic image degradations from unpaired data. Our approach is based on a novel unpaired learning formulation for conditional normalizing flows. We model the degradation process in the latent space of a shared flow encoder-decoder network. This allows us to learn the conditional distribution of a noisy image given the clean input by solely minimizing the negative log-likelihood of the marginal distributions. We validate our DeFlow formulation on the task of joint image restoration and super-resolution. The models trained with the synthetic data generated by DeFlow outperform previous learnable approaches on three recent datasets. Code and trained models are available at: https://github.com/volflow/DeFlow


Introduction
Deep learning based methods have demonstrated astonishing performance for image restoration and enhancement when large quantities of paired training data are available. However, for many real-world applications, obtaining paired data remains a major bottleneck. For instance, in real-world super-resolution [23,8,9] and denoising [2,3], collecting paired data is cumbersome and expensive, requiring careful setups and procedures that are difficult to scale. Moreover, such data is often limited to certain scenes and contains substantial misalignment issues. In many settings, including enhancement of existing image collections or restoration of historic photographs, the collection of paired data is even impossible.
To tackle this fundamental problem, one promising direction is to generate paired training data by applying syn- thesized degradations and noise to high-quality images. The degraded image then has a high-quality ground-truth, allowing effective supervised learning techniques to be applied directly to the synthesized pairs. However, in most practical applications the degradation process is unknown. It generally constitutes a complex combination of sensor noise, compression, and post-processing artifacts. Modeling the degradation process by hand is therefore a highly challenging problem, calling for learnable alternatives.
Since paired data is unavailable, learning the degradation process requires unpaired or unsupervised techniques. Several approaches resort to hand-crafted strategies tailored to specific types of degradations [17]. Existing learnable solutions mostly adopt generative adversarial networks (GANs) with cycle-consistency constraints [39,23,7] or domainaware adversarial objectives [12,34,6] for unpaired training. However, these approaches require careful tuning of several losses. Moreover, cycle-consistency is a weak constraint that easily leads to changes in color and content [10]. Importantly, the aforementioned works rely on fully deterministic mappings, completely ignoring the fundamental stochasticity of natural degradations and noise. In this work, we therefore take a radically different approach.
We propose DeFlow: a novel conditional normalizing flow based method for learning degradations from unpaired data. DeFlow models the conditional distribution p(y|x) of a degraded image y given its clean counterpart x. As shown in Fig. 1, this allows us to sample multiple degraded versions y of any clean image x, which closely resemble the characteristics of the unknown degradations. However, conventional conditional flow models [35,26,5,1] require sample pairs (x, y) for supervised training. We therefore propose a novel formulation for conditional flows, capable of unpaired learning. Specifically, we treat the unpaired setting as the problem of learning the conditional distribution p(y|x) from observations of the marginals p(x) and p(y). By modeling both domains x and y in the latent space of a joint flow network, we ensure sufficient constraints for effective unpaired learning while preserving flexibility for accurate modeling of p(y|x). We additionally introduce a method for conditioning the flow on domain invariant information derived from either x or y to further facilitate the learning problem.
We apply our DeFlow formulation to the problem of joint image restoration and super-resolution in the realworld setting. DeFlow is tasked with learning complex image degradations, which are then used to synthesize training data for a baseline super-resolution model. We perform comprehensive experiments and analysis on the AIM2019 [25] and NTIRE2020 [24] real-world super-resolution challenge datasets. Our approach sets a new state-of-the-art among learning-based approaches by outperforming GANbased alternatives for generating image degradations from unpaired data on three datasets.

Related Work
Learning degradations from unpaired data Realistic noise modeling and generation is a long-standing problem in Computer Vision research. The direction of finding learning-based solutions capable of utilizing unpaired data has received growing interest. One line of research employs generative adversarial networks (GANs) [13]. To learn from unpaired data, either cycle-consistency losses [23,7] or domain-based adversarial losses [12,34,6] are employed. Yet, these approaches suffer from convergence and mode collapse issues, requiring elaborate fine-tuning of their losses. Importantly, such methods learn a deterministic mapping, ignoring the stochasticity of degradations.
Other works [21,30,22,36] learn unsupervised denoising models based on the assumption of spatially uncorrelated (i.e. white) noise. However, this assumption does not apply to more complex degradations, which have substantial spatial correlation due to e.g. compression or post-processing artifacts. Our approach exploits fundamentally different constraints to allow for unpaired learning in this more challenging setting. Recently Abdelhamed et al. [1] proposed a conditional flow based architecture to learn noise models. Yet, their method relies on the availability of paired data for training. Moreover, the authors employ an architecture that is specifically designed to model low-level sensor noise. In contrast, we aim to model more general degradations with no available paired training data. Unpaired Learning with Flows Whilst not for the application of learning image degradations, a few methods have investigated unpaired learning with flows. Grover et al. [14] trained two flow models with a shared latent space to obtain a model that adheres to exact cycle consistency. Their approach then requires an additional adversarial learning strategy based on CyCADA [15], to successfully perform domain translations. Further, Yamaguchi et al. [37] proposed domain-specific normalization layers for anomaly detection. As a byproduct, their approach can perform crossdomain translations on low-resolution images, by decoding an image of one domain with the normalization layer statistics of a different domain. Our proposed unpaired learning approach for flows is, however, fundamentally different from these methods. We do not rely on adversarial training nor normalization layers. Instead, we introduce a shared latent space formulation that allows unpaired learning soley by minimizing the marginal negative log-likelihood.

DeFlow
In this paper, we strive to develop a method for learning a mapping from samples of a source domain x ∼ p x to a target domain y ∼ p y . While there are standard supervised learning techniques for addressing this problem, paired training datasets {(x i , y i )} n i=1 are not available in a variety of important real-world applications. Therefore, we tackle the unpaired learning scenario, where only unrelated sets of source , y i ∼ p y samples are available. While we formulate a more general approach for addressing this problem, we focus on the case where x ∼ p x represent noncorrupted observations, while y ∼ p y are observations affected by an unknown degradation process x → y. In particular, we are interested in image data.
Our aim is to capture stochastic degradation operations, which include noise and other random corruptions. The mapping x → y therefore constitutes an unknown conditional distribution p(y|x). The goal of this work is to learn a generative model p(y|x; θ) of this conditional distribution, without any paired samples (x i , y i ).

Learning the Joint Distribution from Marginals
The unpaired learning problem defined above corresponds to the task of retrieving the conditional p(y|x), or equivalently, the joint distribution p(x, y) = p(y|x)p(x) given only observations from the marginals p(x) and p(y).
In general this is a highly ill-posed problem. However, under certain assumptions solutions can be inferred. As the most trivial case, assuming independence yields the solution p(x, y) = p(x)p(y), which is not relevant since we are interested in finding correlations between x and y. Instead, we first present a simple univariate Gaussian model, which serves as an illustrative starting point for our approach. As we will see, this example forms the simplest special case of our general DeFlow formulation.
Let us assume a 1D Gaussian random variable x ∼ p x = N (µ x , σ 2 x ) with unknown mean µ x and variance σ 2 x . We additionally postulate that y = x + u, where u ∼ p u = N (µ u , σ 2 u ) is a Gaussian random variable that is independent of x. As a sum of independent Gaussian random variables is again Gaussian, it follows that y ∼ p y = N (µ x + µ u , σ 2 x + σ 2 u ). Moreover, it is easy to see that p(y|x) = N (y; x + µ u , σ 2 u ). Under these assumptions, we can estimate all unknown parameters θ = {µ x , σ 2 x , µ u , σ 2 u } in p(x, y) by minimizing the combined negative log-likelihood of the marginal observations, The derivation and resulting analytic solution is given in Appendix A. This shows that inferring the full joint distribution p(x, y) given only unpaired examples is possible in this simple case. Next, we generalize this example using normalizing flows to achieve a highly powerful class of models capable of likelihood-based unpaired learning.

Unpaired Learning of Conditional Flows
In this section, we introduce DeFlow, a normalizing flow based formulation capable of learning flexible conditional distributions from unpaired data. Its core idea is to model the relation between x and y in a Gaussian latent space. We then use a deep invertible encoder-decoder network to map latent variables to the output space. Our model is trained end-to-end by optimizing only the marginal log-likelihoods.
We first detail the proposed latent space formulation. Our model postulates that the random variables x ∼ p x and y ∼ p y are related through a shared latent space. Let z x and z y denote the latent variables corresponding to x and y respectively. In particular, we let z x ∼ N (0, I) follow a standard Normal distribution. The latent variable of y is modeled to depend on z x , but perturbed by another Gaussian random variable u ∼ p u = N (µ u , Σ u ) such that z y = z x + u. The perturbation u is independent of x, and therefore also of z x . The mean µ u and covariance Σ u of u are unknown. Note that, our latent space model is the multivariate generalization of the example presented in Sec. 3.1.
As the next step we use a powerful deep network, capable of disentangling complex patterns and correlations of e.g. images to the Gaussian latent space. In particular, we model this relation between the observations and the latent space with an invertible neural network f θ . Our complete model is then summarized as, Here, ⊥ denotes stochastic independence. Note, that we can sample from the joint distribution by directly applying (2). More importantly, we can also easily sample from the conditional distribution y |x ∼ p(y|x). The invertibility of f θ implies p(y|x) = p(y|z x ). From (2), we thus achieve, In words, y |x is obtained by first encoding z x = f θ (x) then sampling and adding u before decoding again.
To train DeFlow with the likelihood-based objective from (1), we employ the differentiable expressions of the marginal probability densities p x (x) and p y (y). The invertible normalizing flow f θ allows us to apply the change of variables formula in order to achieve the expressions, In both cases, the first factor is given by the determinant of the Jacobian Df θ of the flow network. The second factors stem from the Gaussian latent space distribution of z x and z y , respectively. For an in depth explanation of this fundamental step of normalizing flows we refer the reader to Eq. (1) in [20]. It follows from (3), that f θ (y |x ) = f θ (x)+u. Therefore, we can derive the conditional density, again using change of variables, as Using (4), our model can be trained by minimizing the negative log-likelihood of the marginals (1) in the unpaired setting. Furthermore, the conditional likelihood (5) also enables the use of paired samples, if available. Our approach can thus operate in both the paired and unpaired setting. It is worth noting that the 1D Gaussian example presented in Sec. 3.1 is retrieved as a special case of our model by setting the flow f θ to the affine map The deep flow f θ thus generalizes our initial example beyond the Gaussian case such that complex correlations and dependencies in the data can be captured. In the case of modeling image degradations our formulation has a particularly intuitive interpretation. The degradation process x → y can follow a complex and signal-dependent distribution in the image space. Our approach thus learns the bijection f θ that maps the image to a space where this degradation can be modeled by additive Gaussian noise u. This is most easily seen by studying (3), which implements the stochastic degradation x → y for our model. The clean data x is first mapped to the latent space and then corrupted by the random Gaussian 'noise' u. Finally, the degraded image is reconstructed with the inverted mapping f −1 θ . Lastly, we note that our proposed model achieves conditioning through a very different mechanism compared to conventional conditional flows [35,26,5,1]. These works learn a flow network that is directly conditioned on x as z = f θ (y; x). Thus, a generative model of x is not learned. However, these methods rely on paired data since both x and y are simultaneously required to compute z and its likelihood. In contrast, our approach learns the full joint distribution p(x, y) and uses an unconditional flow network. The conditioning is instead performed by our latent space model (2). However, we show next that our approach can further benefit from the conventional technique of conditional flows, without sacrificing the ability of unpaired learning.

Domain Invariant Conditioning
The formulation presented in Sec. 3.2 requires learning the marginal distributions p x and p y . For image data, this is a difficult task, requiring a large model capacity and big datasets. In this section, we therefore propose a further generalization of our formulation, which effectively circumvents the need for learning the full marginals and instead allows the network to focus on accurately learning the conditional distribution p(y|x).
Our approach is based on conditioning the flow model on auxiliary information h(x) or h(y). Here, h represents a known mapping from the observation space to a conditional variable. We use the conventional technique for creating conditional flows [35,26,5] by explicitly inputting h(x) into the individual layers of the flow network f θ (as detailed in Sec. 4.1). The flow is thus a function z x = f θ (x; h(x)) that is invertible only in the first argument. Instead of the marginal distributions in (4), our approach thus models the conditional densities p(x|h(x)). Since h is a known function, we can still learn p(x|h(x)) and p(y|h(y)) without paired data. Importantly, learning p(x|h(x)) is an easier problem since information in h(x) does not need modeling.
In order to ensure unpaired learning of the conditional distribution p(y|x), the map h must satisfy an important criterion. Namely, that h only extracts domain invariant information about the sample. Formally, this is written as, It is easy to verify the existence of such a function h by taking h(x) = 0 for all x. This choice, where h carries no information about the input sample, retrieves the formulation presented in Sec. 3.2. Intuitively, we wish to find a function h that preserves the most information about the input, without violating the domain invariance condition (6). Since the det Df✓(y) (a) Training (b) Sampling Figure 2. (a) DeFlow is trained to minimize the loss L(θ) in (7). Unpaired inputs x and y are separately encoded by the flow f θ to evaluate the NLL conditioned on h. (b) We sample y ∼ p(y|x) using (8) by first encoding x, then adding the sampled noise u in the latent space and finally decoding it with the inverse flow f −1 θ . joint distribution p(x, y) is unknown, strictly ensuring (6) and z y = f θ (y; h(y)). The training and inference stages of our resulting DeFlow formulation are visualized in Figure  2. The model is trained by minimizing the negative loglikelihood conditioned on h, ln p(y j |h(y j )). (7) During inference, we sample from the conditional distribution p(y|x) using, To avoid repetition, we include a detailed derivation of the generalized formulation in Appendix C.

Learning Image Degradations with DeFlow
In this section we discuss the application of our flowbased unpaired learning formulation to the problem of generating complex image degradations. We detail the model architecture used by DeFlow and explain our approach for obtaining domain invariant conditioning in this setting.

Model Architecture
Flow models are generally implemented as a composition of N invertible layers. Let f n θ denote the n-th layer. Then the model can be expressed recursively as a n = f n θ (a n−1 ; h(x)) where x = a 0 , z = a N and the remaining a n represent intermediate feature maps. By the chain rule, (4) gives det Df n θ (a n ; h(x)) (10) allowing for efficient log-likelihood optimization. We parametrize the distribution p u in (2) with mean µ u the weight matrix M , such that u = Mũ + µ u wherẽ u ∼ N (0, I) is a standard Gaussian. Consequently, the covariance is given by Σ u = M M T . To ensure spatial invariance, we use the same parameters µ u and M at each spatial location in the latent space. We initialize both µ u and M to zero, ensuring that p(x) and p(y) initially follow the same distribution.
Our DeFlow formulation for unsupervised conditional modeling can in principle be integrated into any (conditional) flow architecture f θ . We start from the recent SR-Flow [26] network architecture, which itself is based on the unconditional Glow [19] and RealNVP [11] models. We use an L = 3 level network. Each level starts with a squeeze operation that halves the resolution. It is followed by K flow steps, each consisting of four different layers. The level ends with a split, which removes a fraction of the activations as a latent variable. In our experiments we use K = 16 flow steps, unless specified otherwise. Next, we give a brief description of each layer in the architecture and discuss our modifications. Please, see [26,19] for details. Conditional Affine Coupling [26]: extends the affine coupling layer from [11] to the conditional setting. The input feature map a is split into two parts (a 1 , a 2 ) along the channel dimension. From the subset a 1 and the conditional h(x), a scaling and bias is computed using an arbitrary neural network. These are then applied to the other subset a 2 providing an invertible yet flexible transformation. Affine injector [26]: computes an individual scaling and bias for each entry of the input feature map a from the conditional h(x). The function computing the scaling and bias is not required to be invertible, enabling h(x) to have direct influence on all channels. Invertible 1x1 Convolution [19]: multiplies each spatial location with an invertible matrix. We found the LUdecomposed parametrization [19] to improve the stability and conditioning of the model. Actnorm [19]: learns a channel-wise scaling and shift to normalize intermediate feature maps. Flow Step: is the block of flow layers that is repeated throughout the network. Each flow step contains the above mentioned four layers. First, an Actnorm is applied, followed by the 1 × 1 convolution, Conditional Affine Coupling, and the Affine Injector. Note, that the last two layers are applied not only in reverse order but also in their inverted form compared to the Flow Step in SRFlow [26].
Feature extraction network: we encode the domaininvariant conditional information h using the low-resolution encoder employed by SRFlow. It consists of a modified Residual-in-Residual Dense Blocks (RRDB) model [32]. For our experiments, we initialize it with pretrained weights provided by the authors of [32]. Although this network was originally intended for super-resolution, it is here employed for an entirely different task, namely to encode domaininvariant information h for image degradation learning.

Domain-Invariant Mapping h
The goal of our domain-invariant conditioning h is to provide image information to the flow network, while hiding the domain of the input image. In our application, the domain invariance (6) implies that the mapping h needs to remove information that could reveal whether input is a clean x or a degraded y image. On the other hand, we want to preserve information about the underlying image content to simplify learning. We accomplish this by utilizing some prior assumptions that are valid for most stochastic degradations. Namely, that they mostly affect the high frequencies in the image, while preserving the low frequencies.
We construct h by down-sampling the image to a sufficient extent to remove the visible impact of the degradations. We found it beneficial to also add a small amount of noise to the resulting image to hide remaining traces of the original degradation. The domain invariant mapping is thus constructed as h(x) = d ↓ (x) + n, n ∼ N (0, σ 2 ), where d ↓ (x) denotes bicubic downsampling. Note that this operation is only performed to extract a domain-invariant representation, and is not related to the degradation x → y learned by DeFlow. The purpose of h is to remove the original degradation, while preserving image content.

Experiments and Results
We validate the degradations learned by DeFlow by applying them to the problem of real-world super-resolution (RWSR). Here, the task is to train a joint image restoration and super-resolution model without paired data that is able to translate degraded low-resolution images to high-quality and high-resolution images. In particular, we employ De-Flow to learn the underlying degradation model and use it to generate paired training data for a supervised superresolution model. Experiments are performed on three recent benchmarks designed for this setting. Detailed results with more visual examples are shown in Appendix D.  high-quality non-degraded images of the DIV2k dataset [4].

Datasets
The task is to 4× super-resolve images from the source domain to high-quality images as featured in the target domain. Since the degradations were generated synthetically, there exists a validation set of 100 paired degraded lowresolution and high-quality ground-truth images, allowing the use of reference-based evaluation metrics. NTIRE-RWSR: Track 1 of the NTIRE 2020 RWSR challenge [24] follows the same setting as AIM-RWSR. However, it features a completely different type of degradation, namely highly correlated high-frequency noise. As before, a validation set exists enabling a reference-based evaluation. DPED-RWSR: Differing from the other two datasets, the source domain of Track 2 of the NTIRE 2020 RWSR challenge consists of real low-quality smartphone photos that are to be jointly restored and super-resolved. A high-quality target domain dataset is also provided. The source domain stems from the iPhone3 images of the DPED dataset [16], while the target domain corresponds to the DIV2k [4] training set. Because reference images do not exist evaluation is restricted to no-reference metrics and visual inspection.

Evaluation Metrics
For the synthetic datasets, we report the peak signalto-noise ratio (PSNR) and the structural similarity index (SSIM) [33]. In addition we compute the Learned Perceptual Image Patch Similarity (LPIPS) [38] metric, a reference-based image quality metric based on feature distances in CNNs. As LPIPS has been shown to correlate well with human perceived image quality, we consider it the most important metric for the RWSR task. For the DPED-RWSR we report the NIQE [28], BRISQUE [27] and PIQE [29] no-reference metrics. We also conduct a user study comparing the best models with our DeFlow method. For each compared method, we show participants randomly selected crops super-resolved by both DeFlow and the compared method. Users are then asked to select the more realistic image. We report user preference as the percentage of images where the compared method was preferred over DeFlow. A User Preference <50% indicates that DeFlow obtains 'better' images than the comparison method. More details on the user study are provided in Appendix F.

Baselines and other Methods
We compare DeFlow against Impressionism [17] the winner of the NTIRE 2020 RWSR challenge [24] and Frequency Separation [12] the winner of the AIM 2019 RWSR challenge [25]. Further, we compare with the very recent DASR [34] and the CycleGan based method introduced in [23]. All aforementioned methods apply the same twostage approach, where first a degradation model is learned to generate synthetic training data that is then used to train a supervised ESRGAN [32] based super-resolution model. We also validate against simple baselines. Our No Degradation baseline is trained without any degradation model. The White Noise model adds zero mean Gaussian noise to the low-resolution patches during training. Here, we tested two variants, either fixing the standard deviation σ or sampling the standard deviation of the noise added to each image patch uniformly from U(0, σ max ). For each dataset we tuned both variants with different choices of σ and σ max , respectively, and only report the model that obtained the best LPIPS score.

Training Details
We train all DeFlow models for 100k iterations using the Adam [18] optimizer. The initial learning rate is set to 5 · 10 −5 on the synthetic datasets and to 5 · 10 −6 on the DPED-RWSR dataset and is halved at 50k, 75k, 90k and 95k iterations. We use a batch size of 8 with random crops of size 160 × 160 on the AIM-RWSR and NTIRE-RWSR dataset. On DPED-RWSR we obtained better performance with a patch size of 80 × 80 and a batch size of 48. Batches are sampled randomly such that images of both domains are drawn equally often. Random flips are used as a data augmentation. We use 64 hidden channels in the affine injector layer for NTIRE-RWSR and DPED-RWSR and 128 on AIM-RWSR. Similar to [19,26], we apply a 5bit de-quantization by adding uniform noise to the input of the flow model. We train the DeFlow models using the 4× bicubic downsampled clean domain as X and the noisy domain as Y. Given the large domain gap between the source and target images in DPED-RWSR we do not use the target images and instead use 4× and 8× bicubic downsampled noisy images as the clean domain X . For DPED-RWSR we further follow the approach of [17] and estimate blur kernels of the degraded domain using KernelGAN [6]. These are then applied to any data from the clean domain, i.e. on the clean training data and before degrading images. On AIM-RWSR we normalize X and Y to the same channelwise means and standard deviations. Degraded images are then de-normalized before employing them as training data for the super-resolution model. For the conditional h(x) we used σ = 0.03 in conjunction with 4× bicubic downsampling on NTIRE-RWSR and DPED-RWSR and 8× bicubic downsampling on AIM-RWSR.

Super-Resolution Model
To fairly compare with existing approaches, we an ES-RGAN [32] as the super-resolution model. Specifically, we employ the training code provided by the authors of Impressionism [17]  VGG discriminator is used while on DPED-RWSR a patch discriminator is applied. As in [17], we use the 2× downsampled smartphone images of the DPED-RWSR dataset as clean images and do not use the provided high-quality data. Unlike [17] however, we do not use any downsampled noisy images as additional clean training data. We evaluate the trained models after 10k, 20k, 40k and 60k iterations and report the model with the best LPIPS on the validation set. For DPED-RWSR we simply choose the final model. To better isolate the impact of the learned degradations, we further report the performance of other methods when using their degradation pipeline with our super-resolution model. We mark these models with the † symbol.

Comparison with State-of-the-Art
First, we discuss the results on the AIM-RWSR dataset shown in Tab. 1. The GAN-based Frequency Separation approach [12], the winner of this dataset's challenge, obtains an LPIPS similar to the White Noise baseline. DASR [34] obtains a highly competitive LPIPS, yet it is strongly outperformed by DeFlow in our user study. In fact, as shown in Fig. 3, DASR generates strong artifacts. This can be explained by overfitting, as DASR directly optimizes for LPIPS during training. When using the degradation model of DASR in conjunction with our super-resolution pipeline the resulting model DASR † performs slightly better in the user study while obtaining an LPIPS score of 0.370 compared to DeFlow's 0.349. Notably, DeFlow outperforms all previous methods by a large margin in the user study. It also obtains a higher PSNR and SSIM than all methods with learned, but GAN based degradation models.
On the NTIRE-RWSR dataset (see Tab. 2) DeFlow obtains the best scores among all reference metrics, making it the only model that consistently outperforms the White Noise baseline. In the user study DeFlow is also preferred to all learned degradation models. Yet, the user study indicates better quality from the hand-crafted degradation models, namely Impressionism and the White Noise baseline, compared to the learned approach of DeFlow. However, as shown in the second row of Fig. 3, the White Noise baseline generates highly visible artifacts in smooth regions, e.g. sky, whereas DeFlow removes all noise from these areas.
Lastly, we compare the results on the DPED-RWSR dataset in Tab. 3. Similar to [24], we find that the noreference metrics do not correlate well with the perceived quality of the images. As shown in Fig. 3, DeFlow obtains sharp images with pleasing details clearly outperforming all other learned approaches. Compared to Impressionism [17], we find that our method produces fewer artifacts and does not over-smooth textures. However, we notice that our images retain more noise and are sometimes less sharp. This is supported by the user study where DeFlow significantly outperforms the Frequency Separation method [12], while being head-to-head with Impressionism † [17].
Overall, DeFlow is the only method with consistently good performance across all three datasets, whereas the handcrafted approaches obtain the worst performance on the AIM-RWSR dataset and the other learned approaches are struggling to create artifact-free yet detailed images on the NTIRE-RWSR dataset. It is also noteworthy that Cy-cleGAN [25], despite its immense popularity for unpaired learning, does not perform well on any of these datasets. This can be partly explained by the weak cycle consistency constraint and the use of a deterministic generator.

Ablation Study
In this section, we analyze DeFlow through an ablation study. We train a variety of models on the AIM-RWSR dataset and evaluate their downstream super-resolution performance. These models deviate only in the choice of a single hyper-parameter with all other training settings remaining as described in 5.4. In particular, we scrutinize on three core segments: the depth of the model, the choice of conditioning h(x), and the method of learning the domain shift. For each segment we show the results of this study in a separate section of Tab formation at all i.e. h(x) = 0 leads to a significantly worse performance where the translated images exhibits strong color shifts and blur. This highlights the importance of the conditional and shows that even little auxiliary information yields drastic performance improvements.
Learned shift (Tab. 4, bottom): Last, we investigate our latent space formulation. We first restrict the added noise u ∼ p u to be uncorrelated across the channels by constraining Σ u to a diagonal covariance matrix. We notice a negative impact on performance. This demonstrates the effectiveness of our more general Gaussian latent space model. Further, we validate our choice of using domain dependent base distributions. We train a DeFlow model with a standard normal Gaussian as the base distribution for both domains (i.e. setting u = 0 in (2)). We then infer the domain shift after training by computing the channel-wise mean and covariance matrix in the latent space for each domain. The resulting empirical distributions of both domains become very similar and the inferred shift does no longer model the domain shift faithfully. This results in a substantially worse performance in the down-stream task and further shows the potential of our unpaired learning formulation.

Conclusion
We propose DeFlow, a method for learning conditional flow networks with unpaired data. Through a constrained latent space formulation, DeFlow learns the conditional distribution by minimizing the marginal negative log-likelihoods. We further generalize our approach by conditioning on domain invariant information. We apply De-Flow to the unsupervised learning of complex image degradations, where the resulting model is used for generating training data for the downstream task of real-world superresolution. Our approach achieves state-of-the-art results on three challenging datasets.
To ensure the estimated variances are non-negative, i.e. σ x ≥ 0 and σ u ≥ 0, we introduce the Lagrange multipliers λ x and λ u and have, By the Karush-Kuhn-Tucker theorem, θ * is a optimal solution to l(θ) if ∂l(θ * ) ∂θ = 0 while λ x ≥ 0, λ u ≥ 0, λ x σ 2 x = 0 and λ u σ 2 u = 0 hold. Next, we take partial derivatives ofl(θ) w.r.t. the individual parameters and set them to 0 to obtain the optimal estimates. First, we differentiate w.r.t. the means µ x and µ u , and obtain It directly follows, that the optimal estimates of µ x and µ u can be written as the empirical meansμ x andμ y , Now we turn to the estimation of the variances. We first obtain the following partial derivatives, Setting ∂l(θ) ∂σ 2 u to 0 and using the complementary slackness condition that λ u σ u = 0 must hold at the minimum we obtain, whereσ y = 1 M y∈Y (y −μ y ) 2 is used as short-hand notation for the empirical variance of Y.
Similarly, we set ∂l(θ) ∂σ 2 x to 0. We first define the empirical variance of X asσ 2 By using the complementary slackness condition and the fact that ∂l(θ) ∂σ 2 u = 0, we achieve Finally, the complementary slackness condition leaves us with two cases to consider: (1) λ u = 0 and (2) σ 2 u = 0. In the former case, it directly follows from (22) and then (26) that In the case of σ 2 u = 0, we first obtain from (22) that Inserting this into (26) gives the desired solution for σ 2 x as Case 2: valid iff.σ 2 y ≤σ 2 The second case thus corresponds to the solution where u is an unknown constant variable.
B. Closed-Form Solution for the 1-Dimensional Gaussian Case using DeFlow with a Single Affine Layer In our proposed DeFlow method, we restrict the base distribution p x to be N (0, 1), while keeping p u = N (µ u , σ 2 u ). We show that a single-affine-layer flow f θ (x) = ax + b is able to obtain the an optimal solution for the 1-dimensional Gaussian setting from the previous section under this restriction. To do so, we simply set where µ x and σ x are the optimal estimates obtained in the previous section. Intuitively, we can interpret the singlelayer flow as a learned normalization layer, that ensures a standard normal distribution in the latent space. To recover the optimal parametersμ 2 u andσ 2 u of p u , we need to adjust the optimal values retrieved in the previous section accordingly to this normalization and obtaiñ This shows that the restriction of p x to be standard normal simply leads to an absorption of the required normalization in an affine layer of the flow model.

C. Derivation of the Domain Invariant Conditional DeFlow Method
To generalize the formulation of DeFlow from Sec. 3.2 to include the domain invariant conditioning h(x), we extend the flow network to z x|h(x) = f θ (x; h(x)) and z y|h(y) = f θ (y; h(y)). By invertibility in the first arguments of f θ , samples can then be retrieved by Then, by domain invariance h(x) = h(y), it follows that we can sample from the conditional distribution p(y|x, h(x), h(y)) = p(y|x) using where u ∼ N (µ u , Σ u ). By the change of variables formula, we obtain the differentiable expressions for the conditional marginal distributions, p(y|h(y)) = det Df θ (y; h(y)) · N (f θ (y; h(y)); µ u , I + Σ u ) .
As in the unconditional case, the first factor is given by the determinant of the Jacobian Df θ of the flow network, while the second factor stems from the Gaussian base distributions from out latent space formulation. We can then use (38) to allow the optimization of the new negative log-conditional-likelihood objective ln p y (y j |h(y j ) .

D. DeFlow Degradation Results
Stochasticity of Degradtations Current GAN based approaches [12,17,34,25] model the degradation process as a deterministic mapping, ignoring its inherent stochastic nature. In contrast, DeFlow learns the conditional distribution p(y|x) of a degraded image y given a clean image x and thereby allows sampling multiple degraded versions of a single clean image. As shown in Fig. 4, different degraded samples from DeFlow feature different yet realistic noise characteristics without noticeable bias or recurring patterns.

Clean Input
Different Samples with τ = 1.0 Varying Degradation Strength We further show that De-Flow can be extended to enable sampling degradations at different strengths. To do so, we include a temperature parameter τ that scales the sampled shift-vector u in the latent space. This extends (8) to As shown in Figure 5, setting τ < 1 yields more nuanced degradations, while τ > 1 amplifies the noise.

E. Visual Comparison
While we compared DeFlow to current methods using reference and no-reference based evaluation metrics and a user study, we here provide detailed visual results. Degradation Results: We thus show examples of the synthetic degradations generated from different methods in Figures 7, 9, and 11 for the AIM-, NTIRE-, and DPED-RWSR datasets. As a reference, we further provide examples of real noisy image patches from the respective datasets in Figures 6, 8, and 10. We notice that DeFlow consistently adds more noise compared to the other methods. Yet, on all datasets, the degradations from DeFlow resemble the real noisy data, whereas other learned methods struggle to pickup on the noise characteristics. Real-World Super-Resolution Performance: Further, we provide results of the downstream real-world superresolution task of the different methods on the AIM-, NTIRE-, and DPED-RWSR datasets in Figures 12, 13, and 14, respectively. It is noticeable, that our proposed approach introduces fewer artifacts than the other methods across all datasets. Further, DeFlow is able to reconstruct fine details and provides sharper images than the White Noise model, which performs surprisingly well on the synthetic datasets. On DPED, the performance of the DeFlow degradations is comparable to the handcrafted approach of Impressionism [17]. While DeFlow retains more noise in smooth patches, Impressionism tends to over-smooth textures.

F. Details of the User Study
In this section, we give insight into how we conducted the user study. On AIM and DPED we chose the top 7 models by their LPIPS score to compare in the user study. On DPED we decided to only compare against Frequency Separation [12] and Impressionism [17] both with their superresolution pipeline and ours, as we found that other methods performed considerably worse. For all datasets we used the following set-up for the user study: Participants were shown the same random crop from two different super-resolution models. In addition, we showed them the whole image where the cropped patch was marked in red. Participants were then asked to pick the super-resolved patch that looks more realistic. For that we used three random crops of size 80 × 80 pixels per image of each validation dataset and asked five different study participants per pair. Clean Input DASR [34] Frequency Separation [12] Impressionism [17] DeFlow (ours)  Clean Input CycleGAN [25] Frequency Separation [12] Impressionism [17] DeFlow (ours)  Clean Input [25] Frequency Separation [12] Impressionism [17] DeFlow (ours) Figure 11. DPED RWSR: examples of clean inputs and corresponding synthetically degraded versions from different domain adaption methods. Note, that we did not include CycleGAN [25] as differing to the other approaches it is trained to degrade images from DIV2k with DPED noise instead of down-sampled DPED images.