{'fraction_non_alphanumeric': 0.04015877707867067, 'fraction_numerical': 0.014653097419872472, 'mean_word_length': 4.917541229385307, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Neural Architecture Search for Deep Image Prior

We present a neural architecture search (NAS) technique to enhance the performance of unsupervised image de-noising, in-painting and super-resolution under the recently proposed Deep Image Prior (DIP). We show that evolutionary search can automatically optimize the encoder-decoder (E-D) structure and meta-parameters of the DIP network, which serves as a content-specific prior to regularize these single image restoration tasks. Our binary representation encodes the design space for an asymmetric E-D network that typically converges to yield a content-specific DIP within 10-20 generations using a population size of 500. The optimized architectures consistently improve upon the visual quality of classical DIP for a diverse range of photographic and artistic content.


Introduction
Many common image restoration tasks require the estimation of missing pixel data: de-noising and artefact removal; in-painting; and super-resolution. Usually, this missing data is estimated from surrounding pixel data, under a smoothness prior. Recently it was shown that the architecture of a randomly initialized convolutional neural network (CNN) could serve as an effective prior, regularizing estimates for missing pixel data to fall within the manifold of natural images. This regularization technique, referred to as the Deep Image Prior (DIP) [34] exploits both texture self-similarity within an image and the translation equivariance of CNNs to produce competitive results for the image restoration tasks mentioned above. However, the efficacy of DIP is dependent on the architecture of the CNN used; different content requires different CNN architectures for excellent performance and care over meta-parameter choices e.g. filter sizes, channel depths, epoch count, etc.
This paper contributes an evolutionary strategy to automatically search for the optimal CNN architecture and associated meta-parameters given a single input image. The core technical contribution is a genetic algorithm (GA) [8] for representing and optimizing a content-specific network architecture for use as DIP regularizer in unsupervised im- Figure 1. Neural Architecture Search yields a content-specific deep image prior (NAS-DIP, right) that enhances performance over classical DIP (left) for unsupervised image restoration tasks including de-noising (top) and in-painting (bottom). Source inset (red) age restoration tasks. We show that superior results can be achieved through architecture search versus standard DIP backbones or random architectures. We demonstrate these performance improvements for image de-noising, Content-Aware Fill (otherwise known as inpainting), and content upscaling for static images; Fig. 1 contrasts the output of classic DIP [34] with our neural architecture search (NAS-DIP).
Unlike image classification, to which NAS has been extensively applied [29,28,23,11], there is no ground truth available in the DIP formulation. An encoder-decoder network (whose architecture we seek) is overfitted to reconstruct the input image from a random noise field, so acquiring a generative model of that image's structure. Parallel work in GANs [40] has shown that architectural design, particularly of the decoder, is critical to learning a sufficient gen-erative model, and moreover, that this architecture is content specific. Our work is the first to both automate the design of encoder-decoder architectures and to do so without supervision, leveraging a state-of-the-art perceptual metric to guide the optimization [44]. Further, we demonstrate that the optimized architectures are content specific, and enable clustering on visual style without supervision.

Related Work
Neural architecture search (NAS) seeks to automate the design of deep neural network architectures through datadriven optimization [12], most commonly for image classification [47,28], but recently also object detection [47] and semantic segmentation [6]. NAS addresses a facet of the automated machine learning (AutoML) [20] problem, which more generally addresses hyper-parameter optimization and tuning of training meta-parameters.
Early NAS leveraged Bayesian optimization for MLP networks [4], and was extended to CNNs [9] for CIFAR-10. In their seminal work, Zoph and Le [46] applied reinforcement learning (RL) to construct image classification networks via an action space, tokenizing actions into an RNN-synthesised string, with reward driven by validation data accuracy. The initially high computational overhead (800GPUs/4 weeks) was reduced while further enhancing accuracy. For example, by exploring alternative policies such as proximal policy optimization [31] or Q-learning [2], RL approaches over RNN now scale to contemporary datasets e.g. NASNet for Ima-geNet classification [26,47] and was recently explored for GAN over CIFAR-10 [16]. Cai et al. similarly tokenizes the architecture, but explore the solution space via sequential transformation of the string via function-preserving mutation operations [5] on an LSTM-learned embedding of the string. Our work also encodes architecture as a (in our case, binary) string, but optimizes via an evolutionary strategy rather than training a sequential model under RL.
Evolutionary strategies for network architecture optimization were first explored for MLPs in the early nineties [25]. Genetic algorithms (GAs) were used to optimize both the architecture and weights [1,33], rather than rely upon backprop, in order to reduce the GA evaluation bottleneck; however, this is not practical for contemporary CNNs. While selection and population culling strategies [11,29,28] have been explored to develop high performing image classification networks over ImageNet [23,28]. Our work is unique in that we explore image restoration encoder-decoder network architectures via GA optimization, and as such, our architecture representation differs from prior work.
Single image restoration has been extensively studied in classical vision and deep learning, where priors are prescribed or learned from representative data. A common prior to texture synthesis is the Markov Random Field (MRF) in which the pair-wise term encodes spatial coherence. Several in-painting works exploit MRF formulations of this kind [22,18,24], including methods that source patches from the input [10] or multiple external [17,14] images, or use random propagation of a few good matched patches [3].
Patch self-similarity within single images has also been exploited for single image super-resolution [15] and denoising. The Deep Image Prior (DIP) [34] (and its video extension [42]) exploit translation equivariance of CNNs to learn and transfer patches within the receptive field. Very recently, single image GAN [32] has been proposed to learn a multi-resolution model of appearance and DIP has been applied to image layer decomposition [13]. Our work targets the use cases for DIP proposed within the original paper [34]; namely super-resolution, de-noising and region inpainting. All three of these use cases have been investigated substantially in the computer vision and graphics literature, and all have demonstrated significant performance benefits when solved using a deep neural network trained to perform that task on representative data. Generative Adversarial Networks (GANs) are more widely used for in-painting and super-resolution [43,36] by learning structure and appearance from a representative corpus image data [41], in some cases explicitly maintaining both local and global consistency through independent models [21]. Our approach and that of DIP differs in that we do not train a network to perform a specific task. Instead, we use an untrained (randomly initialized) network to perform the tasks by overfitting such a network to a single image under a task-specific loss using neural architecture search.

Architecture Search for DIP
The core principle of DIP is to learn a generative CNN G θ (where θ are the learned network parameters e.g. weights) to reconstruct x from a noise field N of identical height and width to x, with pixels drawn from a uniform random distribution. Ulyanov et al. [34] propose a symmetric encoder-decoder network architecture with skip connections for G θ , comprising five pairs of (up-)convolutional layers with varying architectures depending on the image restoration application (de-noising, in-painting or super-resolution) and the image content to be processed. A reconstruction loss is applied to learn G θ for a single given image x: Our core contribution is to optimize not only for θ but also for architecture G using a genetic algorithm (GA), guided via a perceptual quality metric [44], as now described.

Network Representation
We encode the space of encoder-decoder (E-D) architectures from which to sample G as a constant length binary sequence, representing N paired encoder decoder units U = {U 1 , ..., U N }. Following [34], G is a fully convolutional E-D network and optimize for the connectivity and meta-parameters of the convolutional layers. A given unit U n comprises encoder E n and decoder D n convolutional stages denoted E n and D n respectively each of which requires 7 bits to encode its parameter tuple. Unit U n requires a total 14 + 4N bits to encode, as an additional 4N -bit block for the unit, encodes a tuple specifying the configuration of skip connections from its encoder stage to each of the decoder stages i.e. both within itself and other units. Thus, the total binary representation for an architecture in neural architecture search for DIP (NAS-DIP) is N (14 + 4N ). For our experiments, we use N = 6 but note that the effective number of encoder or decoder stages varies according to skip connections which may bypass the stage. Fig. 2 illustrates the organisation of the E-D units (right) and the detailed architecture of a single E-D unit (left). Each unit U n comprises the following binary representation, where super-scripts indicate elements of the parameter tuple: • E s n ∈ [0, 1] (1 bit) a binary indicator of whether the encoder stage of unit U n is skipped (bypassed).
• E h n ∈ [0, 7] (3 bits) encoding number of filters h = 2 E h n −1 and so channels output by the encoder stage.
• D s n ∈ [0, 1] (1 bit) a binary indicator of whether the decoder stage of unit U n is skipped (bypassed).
• D h n ∈ [0, 7] (3 bits) encoding number of filters h = 2 D h n −1 and so channels output by the decoder stage.

NAS-DIP-T.
We explore a variant of the representation encoding maximum epoch count T = 500 * (2 t − 1) via two additional bits coding for t and thus a representation length for NAS-DIP-T of N (14 + 4N ) + 2.
Symmetric NAS-DIP. We also explore a compact variant of the representation that forces E n = D n for all parameters, forcing a symmetric E-D architecture to be learned and requiring only 10 bits to encode U n . However, we later show that such symmetric architectures are almost always underperformed by asymmetric architectures discovered via our search (subsec. 4.1).

Evolutionary Optimization
DIP provides an architecture specific prior that regularises the reconstruction of a given source image x from a uniform random noise field N . Fixing N constant, we use a genetic algorithm (GA) to search architecture space for the optimal architecture G * to recover a 'restored' e.g. denoised, inpainted or upscaled versionx of that source image: GAs simulate the process of natural selection by breeding successive generations of individuals through the processes of cross-over, fitness-proportionate reproduction and mutation. In our implementation, such individuals are network configurations encoded via our binary representation. Individuals are evaluated by running a pre-specified DIP image restoration task using the encoded architecture. We consider unsupervised or 'blind' restoration tasks (denoising, in-painting, super-resolution) in which an idealx is unknown (is sought) and so cannot guide the GA search. In lieu of this ground truth we employ a perceptual measure (subsec. 3.2.1) to assess the visual quality generated by any candidate architecture by training G * θ (N ) via backpropogation to minimize a task specific reconstruction loss: where D is a downsampling operator reducing its target to the size of x via bi-linear interpolation, and M (.) is a masking operator that returns zero within the region to be in-painted.
We now describe a single iteration of the GA search, which is repeated until the improvements gained over the previous few generations are marginal (the change in both average and maximum population fitness over a sliding time window fall below a threshold).

Population Sampling and Fitness
We initialize a population of K = 500 solutions uniformly sampling B N (14+4N ) to seed initial architectures G = {G 1 , G 2 , ..., G K }. The visual quality ofx under each architecture is assessed 'blind using a learned perceptual measure (LPIPS score) [44] as a proxy for individual fitness: where L is the task specific loss (Eq.3-5) and P (.) is the perceptual score [44] for a given individual G i in the population. We explored several blind perceptual scores from the GAN literature including Inception Score [30] and Frechet Inception Distance [19] but found LPIPS to better correlate with PSNR and SSIM during our evaluation, improving convergence for NAS-DIP. Individuals are selected stochastically (and with replacement) for breeding, to populate the next generation of solutions. Architectures that produce higher quality outputs are more likely to be selected. However, population diversity is encouraged via the stochasticity of the process and the introduction of a random mutation into the offspring genome. We apply elitism; the bottom 5% of the population is culled, and the top 5% passes unperturbed to the next generation -the fittest individual in successive generations is thus at least as fit as those in the past. The middle 90% is used to produce the remainder of the next generation. Two individuals are selected stochastically with a bias to fitness , and bred via cross-over and mutation (subsec. 3.2.2) to produce a novel offspring for the successive generation. This process repeats with replacement until the population count matches the prior generation.

Cross-over and Mutation
Individuals are bred via genetic crossover; given two constant length binary genomes of form G{(E 1 , R 1 , D 1 ), ..., (E N , R N , D N )} ∈ B N (14+4N ) , a splice point S = [1, N ] is randomly selected such that units from a pair of parents A and B are combined via copying of units U N <S from A and U N ≥S from B. Such cross-over could generate syntactically invalid genomes e.g. due to  [34]. Examples further quantified in Table 2. tensor size incompatibilities between units in the genome. During the evaluation, an invalid architecture evaluates to zero prohibiting its selection for subsequent generations.
Population diversity is encouraged via the stochasticity of the selection process and the introduction of a random mutation into the offspring genome. Each bit within the offspring genome is subject to random flip with low probability p m ; we explore the trade-off of this rate against convergence in subsec. 4.2.

Experiments and Discussion
We evaluate the proposed neural architecture search technique for DIP (NAS-DIP) for each of the three blind image restoration tasks proposed originally for DIP [34]: image in-painting, super-resolution and de-noising.
Datasets. We evaluate over three public datasets: 1) Places2 [45]; a dataset of photos commonly used for inpainting (test partition sampled as [21]); 2) Behance Artistic Media (BAM!) [38]; a dataset of artwork, using the test par-tition of BAM! selected to evaluate in-painting in [14] comprising of 8 media styles; 3) DIP [34]; the dataset of images used to evaluate the original DIP algorithm of Ulyanov et al. Where baseline comparison is made to images from the latter, the specific network architecture published in [34] is replicated using according to [35] and the authors' public implementation. Results are quantified via three objective metrics; PSNR (as in DIP) and structural similarity (SSIM) [37] are used to evaluate against ground truth, and we also report the perceptual metric (LPIPS) [44] used as fitness score in the GA optimization.
Training details. We implement NAS-DIP in Tensorflow, using ADAM and learning rate of 10 −3 for all experiments.     N (14+4N ) ) for NAS-DIP, N = 5 symmetric network: a) Population distribution at convergence; multi-modal distribution of similar best DIP architectures obtained for the super-resolution task (image inset); b) Visualizing best discovered architecture from the population (lower), and the classical DIP default architecture [35] (upper) for this task.
10-20 generations to convergence). Our DIP implementation extends the authors' public code for DIP [34]. Reproducibility is critical for NAS-DIP optimization; the same genome must yield the same perceptual score for a given source image. In addition to fixing all random seeds (e.g. for batch sampling and fixing the initial noise field), we take additional technical steps to avoid non-determinism in cuDNN e.g. avoidance of atomic add operations in image padding present in original DIP code.

Network Representation
We evaluate the proposed NAS method over three variants of the architecture representation proposed in subsec.3.1: NAS-DIP, NAS-DIP-T (in which epoch count is also optimized), and a constrained version of NAS-DIP forcing a symmetric E-D network. For all experiments, N = 5 enabling E-D networks of up to 10 (up-)convolutional layers with gated skip connections. Performance is evaluated using PSNR for comparison with original DIP [34] as well as SSIM, and the LPIPS score used to guide NAS is also reported. Table 2 provides direct comparison on images from Ulyanov et al. [34]; visual output and convergence graphs are shown in Fig. 4. Following random initialization relative fitness gains of up to 30% are observed after 10-20 generations after which performance converges to values above the DIP baseline [34] in all cases.
For both in-painting and de-noising tasks, asymmetric networks are found that outperform any symmetric network, including the networks published in the original DIP for those images and tasks. For super-resolution, a symmetric network is found to outperform asymmetric networks and constraining the genome in this way aids in discovering a performant network. In all cases, it was unhelpful to optimize for epoch count T , despite prior observations on the   Table 3. Quantitative results for varying mutation rates; chance of bit flip in offspring p(r). Comparing each of three variants of the proposed method: NAS-DIP, NAS-DIP-T and NAS-DIP constrained to symmetric E-D (subsec. 3.1) against original DIP. Quality metrics: LPIPS [44] (used in NAS objective); and both PSNR and SSIM as external metrics. importance of T in obtained good reconstructions under DIP [34]. Table 1 broadens the experiment for NAS-DIP, averaging performance for three datasets: BAM! with 80 randomly sampled images, 10 from each 8 styles; Places2 with a 50 image subset included in [21]; DIP with 7 images used in [34]. For all three tasks and all three datasets, NAS-DIP discovers networks outperforming those of DIP [34].

Evaluating Mutation Rate
Population elitism requires a moderate mutation rate p(r) to ensure population diversity and so convergence, yet raised too high convergence occurs at a lower quality level. We evaluate p m = {0.01, 0.02, 0.05, 0.10} in Table 3, observing that for all tasks a bit flip probability of 5% (i.e. an average of 4 bit flips per offspring for N = 5) encourages convergence the highest fitness value. This in turn correlates to highest visual quality according to both PSNR and SSIM external metrics. Fig. 6 provides representation visual output alongside convergence graphs. In general, convergence is achieved between 10-20 generations taking a few hours on single GPU for a constant population size K = 500.

Content aware in-painting
To provide a qualitative performance of the Content-Aware in-painting we compare our proposed approach against several contemporary baselines: PatchMatch [3] and Image Melding [7] which sample patches from within a single source image; and two methods that use millions or training images, the generative convnet approach 'Context encoder' [27] a recent GAN in-painting work [21], and the Style-aware in-painting [14]. We compare against the same baselines using scenic photographs in Places2, Fig 7 presents a visual comparison, and quantitative results are given in Tbl.1 over the image set included in [21] with the same mask regions published in that work.

Discovered Architectures
Architectures discovered by NAS-DIP are visualized in Fig. 5a, using t-SNE projection in the architecture space (B N (14+4N ) ) for a solution population at convergence (generation 20) for a representative super-resolution task for which symmetric E-D network was sought (source image Source / Mask (red) PatchMatch [3] ImgMelding [7] Context Encoder [27] ImgComp [21] Style-aware [ inset). Distinct clusters of stronger candidate architectures have emerged each utilising all 10 convolutional stages and similar filter sizes but with distinct skip activations and chan-nel widths; colour coding indicates fitness. Fig. 5b shows a schematic for the default architecture for this task published in [35] alongside the discovered best architecture for this task and image. Fig. 8 visualizes 80 best performing networks for inpainting BAM! artworks of identical content (flowers) but exhibiting 8 different artistic styles. Each image has been run through NAS-DIP under loss eq. 4; representative output is included in Fig. 8. Visualizing the best performing architectures via t-SNE in B N (14+4N ) reveals style-specific clusters; pen-and-ink, graphite sketches, 3D graphics renderings, comics, and vector artwork all form distinct groups while others e.g. watercolor form several clusters. We conclude that images that share a common visual style exhibit commonality in network architecture necessary perform image reconstruction well. This discovery suggests a potential application for NAS-DIP beyond architecture search to unsupervised clustering e.g. for visual aesthetics.

Conclusion
We reported the first neural architecture search (NAS) for image reconstruction under the recently proposed deep image prior (DIP) [34], learning a content-specific prior for a given source image in the form of an encoder-decoder (E-D) network architecture. Following the success of evolutionary search techniques for image classification networks [29,28] we leveraged a genetic algorithm to search a binary space representing asymmetric E-D architectures and demonstrated its efficacy for image de-noising, in-painting and super-resolution. For the latter case, we observed a constrained version of our genome yielding symmetric networks exceeded that of asymmetric networks which benefited the other two tasks. All image restoration tasks were 'blind' and optimization was guided via a proxy measure for perceptual quality [44]. In all cases, we observed the performance of the discovered networks to significantly exceed classical DIP and the potential for content-specific architectures beyond image restoration to unsupervised style clustering. Future work could pursue the latter parallel application, as well as explore further generalizations of the architecture space beyond fully convolutional E-Ds to incorporate pooling and normalization strategies.