{'fraction_non_alphanumeric': 0.03881523946016959, 'fraction_numerical': 0.021935586607747123, 'mean_word_length': 4.837787590053451, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ArtFID: Quantitative Evaluation of Neural Style Transfer

The field of neural style transfer has experienced a surge of research exploring different avenues ranging from optimization-based approaches and feed-forward models to meta-learning methods. The developed techniques have not just progressed the field of style transfer, but also led to breakthroughs in other areas of computer vision, such as all of visual synthesis. However, whereas quantitative evaluation and benchmarking have become pillars of computer vision research, the reproducible, quantitative assessment of style transfer models is still lacking. Even in comparison to other fields of visual synthesis, where widely used metrics exist, the quantitative evaluation of style transfer is still lagging behind. To support the automatic comparison of different style transfer approaches and to study their respective strengths and weaknesses, the field would greatly benefit from a quantitative measurement of stylization performance. Therefore, we propose a method to complement the currently mostly qualitative evaluation schemes. We provide extensive evaluations and a large-scale user study to show that the proposed metric strongly coincides with human judgment.


Introduction
Style transfer and texture transfer have been researched at least since the early 2000s [25,20]. In 2016, Gatys et al. proposed a novel method for style transfer based on the features of a pretrained convolutional neural network. The recent years have seen a surge of research exploring different avenues including, but not limited to, optimization-based approaches [23,40,36,41], feed-forward models [33,72], universal feed-forward models [30,46,7,50,57,32,65,44], ultraresolution techniques [75,11], meta-learning approaches [64,85], and video style transfer [60,10,29,6]. The developed techniques have not just progressed the field of style transfer, but also led to breakthroughs in other areas of computer vision, such as visual synthesis. For example, the perceptual loss [23,33] has been applied to tasks ranging from photographic image synthesis [9] to motion transfer [4], the generator architecture employed by Johnson et al. [33] has been used to synthesize photo-realistic images from semantic label maps [76] or for unpaired image-toimage translation [88], and the AdaIN layer [30] was a key ingredient for the StyleGAN generator [35] that produced state-of-the-art results in unconditional generative image modeling. In contrast to other areas of computer vision research such as classification or even natural image synthesis, where quantitative evaluation and benchmarking are already common practice, quantitative assessment of style transfer models is still lacking. Currently, the most common method for evaluating these models is a qualitative comparison for a few, commonly used, style and content images. This is generally useful for an initial impression, however, the performance of style transfer models greatly varies across different style and content images. For example, Fig. 1 shows stylized images from AdaIN [30] and WCT [46] for two different style/content pairs. One can argue that AdaIN [30] outperforms WCT [46] for the first pair and vice versa for the second pair. Some papers provide a quantitative comparison using measurements for speed, memory, or control [11,30,64,54,46,72,85,28,12,49,79,82,51,13]. These measurements are very helpful to highlight a model's performance with respect to a specific property such as speed, but are generally not intended to measure stylization performance. A few works [28,37,52,82] also employ classical perceptual metrics such as PSNR or SSIM [77], however, these metrics are generally not consistent with human perception [86]. Sanakoyeu et al. [62] proposed the Deception rate to measure the quality of stylized images. It is the fraction of stylized images that an artist classification network has assigned to the artist, whose artwork has been used for stylization. The deception rate can be useful in some cases but also suffers from certain drawbacks and is not widely used as a result. First, it only works for style images belonging to an artist for which the artist classifier network has been trained and thus cannot be used for arbitrary styles. And second, it completely disregards content preservation. In the absence of a suitable quantitative measurement of stylization performance, many authors have turned to human evaluation studies [40,54,28,5,57,62,50,41,75,37,8,81,80,7,82,16,83,15,43], where users are asked to compare stylized images of different models and specify their preference. These studies give better insights into a model's stylization performance in comparison to other methods, especially when conducted with many users and for many different style and content images. Nonetheless, there are cases where a user study is not feasible. Large-scale crowdsourcing can be expensive and may not be an option for researchers with limited funding. Furthermore, there are cases where a quantitative measurement is simply more suitable. As an example, consider the closely-related field of image synthesis, where quantitative measurements for image quality are used, not just for evaluation, but also as a core part of the approach. Recent examples include work by Mokady et al. [56], where Fréchet Inception Distance (FID) [26] and Learned Perceptual Image Patch Similarity (LPIPS) [86] measurements are employed for self-filtering a collection of images, work by Karras et al. [34] that use the FID to detect GAN [24] overfitting, or a large-scale GAN study by Lucic et al. [53] that utilize the FID to compare the sensitivity of different GAN models to hyper-parameters. A quantitative measurement of stylization performance would facilitate similar studies to further analyze and improve style transfer methods. In this work, we propose a method that fills this role and call it ArtFID. Our goal is not to replace the current evaluation techniques, but rather to complement them. Stylization performance is mostly determined by two factors: content preservation and style matching. Content preservation refers to the extent to which the semantic content from the content images is preserved in the stylized images. Style matching refers to the extent to which the style of the generated images resembles the style of the target style images. Measuring content preservation is straightforward using the well-established perceptual loss [23,33], CLIP loss [59], or LPIPS metric [86]. However, measuring style matching is more challenging because it confronts us with the question "what is style?", a question that has been debated relentlessly by art historians [42,1,73,55,63]. However, a commonly used definition is that style is the "distinctive manner which permits the grouping of works into related categories" [21]. This definition gives rise to a representation learning approach, where artworks are classified into categories by a neural network to obtain representations that reflect their stylistic similarity. To this end, we collect a large-scale dataset of artworks, labeled by artist and stylistic period. We make use of both labels by employing an architecture with two classification heads. Having obtained the image representations in the form of neural network features, we return to the issue of measuring style matching. As shown by Li et al. [45], the task of style transfer corresponds to aligning feature distributions of neural networks. To measure style matching, we can thus measure how well the feature distribution of style images matches the feature distribution of stylized images. Following the work by Heusel et al. [26], we measure the distance between these two feature distributions with respect to the first two moments using the Fréchet distance [22,18]. Our contributions are as follows: -We propose a method for quantitatively evaluating style transfer models with respect to stylization performance that strongly coincides with human judgment. -We introduce a large-scale dataset, containing 250k images of labeled artworks. -We provide extensive evaluations and a large-scale human evaluation study to support our claims.

Related Work
Style Transfer Research in the fields of style transfer and texture transfer goes back to the early 2000s. Efros and Freeman [20] proposed texture synthesis and transfer using image quilting and Hertzmann et al. [25] proposed image analogies based on a simple multi-scale autoregression. Other work in this area includes a method for synthesizing directional textures [74].
In 2016, Gatys et al. [23] proposed an iterative optimization algorithm for style transfer that leverages the representations learned by convolutional neural networks. In the following, Johnson et al. [33] and Ulyanov et al. [72] used feedforward networks to approximate the optimization problem formulated by Gatys et al. [23] for a specific style image. In order to use mutliple styles with a single network, Dumoulin et al. [19] proposed conditional instance normalization. Huang and Belongie [30] proposed the first method for arbitrary style transfer, using adaptive instance normalization. Li et al. [46] introduced another arbitrary style transfer model, based on whitening and colouring transformations. Li et al. [45] provided insights into the mechanics of style transfer by proving that matching the Gram matrices of feature maps is equivalent to minimizing the Maximum Mean Discrepancy with the second order polynomial kernel. Li et al. [47] proposed a closed-form solution for photorealistic stylization that is based on the aforementioned whitening and colouring transformations. Sheng et al. [65] introduced a method that semantically aligns content features to trade off generalization and efficiency and Li et al. [44] proposed a linear transformation for image and video style transfer. Shen et al. [64] and Zhang et al. [85] employed meta-learning approaches to handle the trade-off between speed, flexibility, and quality, and Svoboda et al. [68] used graph neural networks to recombine style and content in latent space. Another line of research used attention mechanisms to integrate local style patterns and align content and style manifolds [57,16,50,54,32]. Recent work includes exemplar-based portrait style transfer [83], online motion style transfer [71], transformer-based style transfer [15], lightweight photorealis-tic style transfer [13], style transfer for scene reconstruction [27], industrial style transfer [82], thumbnail instance normalization for ultra-resolution style transfer [11], style transfer based on text descriptions [43], internal-external style transfer with two contrastive losses [7], reversible neural flows for unbiased stylization [2], and a unified architecture for domain-aware style transfer [28].
Inception Score Salimans et al. [61] proposed the Inception score, an evaluation metric for generated images that correlates with human judgment. To compute the Inception score, the Inception network [69] is used to compare the conditional label distribution to the marginal distribution of the generated images. A downside of the Inception score is that the statistics of the generated images are not compared to the statistics of real images [26].
Fréchet Inception Distance Heusel et al. [26] proposed the Fréchet Inception Distance (FID) that improves the Inception score by computing the Fréchet distance [22,18] between the Gaussian distribution of Inception features of generated images and the Gaussian distribution of Inception features of real images.

ArtFID
As stated before, the stylization quality is determined by content preservation and style matching. In contrast to previous work, we measure both factors and combine them to form a quantitative metric. Let X c denote the set of content images, X s the set of style images, and X g the set of stylized images, generated from a particular style transfer model. To measure content preservation for a particular stylized image, we compute a distance d(X c and the corresponding stylized image X (i) g . Possible choices for d(·) are the VGG perceptual loss [23,33], the CLIP loss [59], or the LPIPS metric [86]. We select LPIPS and show in Sec. 4.5 that it slightly outperforms the alternatives. Measuring the content preservation for a style transfer model then simply corresponds to evaluating d(X (i) c , X (i) g ) for a large batch of stylized images and taking the mean. As stated above, to measure style matching for a model, we first learn suitable image representations by training a classifier on a large-scale artwork dataset (Sec. 3.3). For the classifier, we use the Inception network [69]. The architecture is modified by using two classification heads instead of one. In doing so, we are able to harness the information from both the artist labels as well as the labels for the stylistic period that are present in the dataset. For more information about the classifier training, see Sec. 3.4. The image representations are the Inception features from the last layer before the classifier heads. These features are computed for both the style images X s and the stylized images X g . To measure style matching between the style images X s and the stylized images X g , we measure the distance between these two feature distributions. This is motivated by work from Li et al. [45] that shows that the task of style transfer corresponds to aligning feature distributions of deep neural networks. Similar to Heusel et al. [26], we measure the distance between these two feature distributions with respect to the first two moments using the Fréchet distance [22,18] FID(X s , where (µ s , Σ s ) and (µ g , Σ g ) are the mean and covariance of the Inception features of the style images X s and the stylized images X g , respectively.
Having obtained a measure for content preservation and a measure for style matching, we combine these measures to form the ArtFID, formulated as follows Ones are added to both factors to avoid a degenerate minimum when X c = X g or X s = X g . The measures are combined to ensure that only models that perform well on both content preservation and style matching achieve a low ArtFID score. Multiplication lends itself well to this purpose. We also tested the addition operation but the effect on the overall ranking was negligible.

Unbiased ArtFID Computation
Chong and Forsyth [14] showed that the FID computed with a finite number of samples N , denoted FID N , is biased. This bias in FID N vanishes when N → ∞.
The authors [14] propose a method for computing FID ∞ , an unbiased estimate of FID ∞ , using N samples. The . To obtain FID ∞ , the linear regression model is evaluated at 0. For an in-depth analysis, we refer to the work of Chong and Forsyth [14]. We use FID ∞ for computing the ArtFID and denote it as ArtFID ∞ . Tab. 2 shows that it performs slightly better than the biased ArtFID.

Data
The dataset used for training the Inception network consists of 250K images, labeled by artist and stylistic period. The images were collected from public datasets (e.g. WikiArt [70]), as well as museum databases and art collections (e.g. Art UK 3 ). See Sec. A for more details about the dataset.

Training
We broadly follow the methodology described by Szegedy et al. [69] for training the Inception network. The classifier head is replaced by two separate heads, one for each label (artist and stylistic period). We use the Adam optimizer [38] with a learning rate of 0.0001 and β = (0.9, 0.999). The batch size is set to 64 and the weight of the auxiliary classifier is set to 0.3. See Sec. B for more information about the training and evaluation.

ArtFID Evaluation
We evaluate the ArtFID with 13 different style transfer methods from the literature [50,65,7,44,16,32,54,30,23,57,68,11,46], some of which are well-established, some are recently published, and some are in between. For each method, we generate stylized images using the same pairs of content and style images. Content images are sampled from the Places365 dataset [87] and the COCO dataset [48]. Style images are sampled from the WikiArt dataset [70] and the BAM dataset [78]. Images are sampled from multiple datasets to increase the variability of style and content images. Following the recommendations by Heusel et al. [26], we compute the ArtFID with samples of 50k images. Both style and content images are resized to 512x512, a commonly used image size for style transfer methods. As proposed by Parmar et al. [58], we resize style and content images with bicubic interpolation and antialiasing and save the stylized images using the PNG format. For each style transfer method, the ArtFID is computed with 5 different samples containing 50k images each. The results are reported in Tab. 1.

Large-Scale Human Evaluation
In order to test the validity of the ArtFID, we conduct a large-scale human evaluation study on Amazon Mechanical Turk (AMT). We use the same 13 style transfer methods as before. Similar to [36,40], we chose a pairwise comparison, where the outputs of two methods are compared head-to-head. This choice is in line with recent work from the field of psychology that provides evidence that human perceptual decision-making is significantly impaired when presented with multiple alternatives, rather than just two alternatives [84]. We compare each possible pairing of style transfer methods. For each pairing, we perform 80 comparisons. Each comparison is voted on by 5 workers and the winner is decided by majority vote. In total there were 31200 tasks and workers were rewarded $0.03 for each task. Fig. 2 shows the interface that was shown to the AMT workers for a particular task.

Correlation with Human Judgment
We evaluate the results of the AMT user study with the widely-used Bradley-Terry model [3]. Given m entities that are repeatedly compared with each another in pairs, the Bradley-Terry model yields an overall score γ i for each entity i that induces a global ranking of the entities. The model works under the assumption that if we partition the set of entities into two non-empty subsets, some entity in the first subset is compared to some entity in the second subset at least once [31]. This requirement is easily satisfied by comparing each entity with all the other entities, as described in Sec. 4.2. We can then estimate the scores γ 1 , ..., γ m using the log-likelihood of the Bradley-Terry model where w ij denotes the number of times that entity i was preferred over entity j, and we assume that w ii = 0 and i γ i = 1 [31]. There exists a simple iterative algorithm that yields a unique maximum of Eq. 3. See Sec. C for an outline of the algorithm and [31] for a study of its convergence properties.
To measure how well the ArtFID coincides with human judgment, we compute the correlation between the ranking of style transfer methods obtained from the human evaluation study and the ranking of the same methods induced by the ArtFID. A commonly-used measure for this is Spearman's rank correlation coefficient (Spearman's ρ) [66,67], a non-parametric measure of correlation between rankings. Spearman's ρ is calculated by replacing the actual observations with the corresponding ranks in the formula for the correlation coefficient [39]. Spearman's ρ is often used as a hypothesis test to determine if there is a relation between two random variables [17]. For the two-sided test, the null hypothesis is that both random variables are mutually independent and the alternative hypothesis is that there is either a positive or a negative correlation. For the one-sided test, the null hypothesis is that both random variables are mutually independent and the alternative hypothesis is that there is a positive correlation. We report both the Spearman's ρ as well as the p-values of the hypothesis tests in Tab. 2. To validate our results, we rely not only on our AMT user study but also consider user studies from other style transfer works. We observe that the ranking induced by the ArtFID coincides with the majority of user studies reported in the literature [7,54,57,50,15,44,16]. For example, Chen et al. [7] report that users preferred IEContraAST [7] over SANet [57], Gatys et al. [23], LST [44], AdaAttN [50], WCT [46], and Avatar-Net [65]. Luo et al. [54] report that users preferred PAMA [54] over AdaAttN [50], MANet [16], SANet [57], MAST [32], and AdaIN [30]. Park and Lee [57] report that users preferred SANet [57] over Gatys et al. [23], Avatar-Net [65], AdaIN [30], and WCT [46], Liu et al. [50] report that users preferred AdaAttN [50] over LST [44], SANet [57], MAST [32], AdaAttN [50], and Avatar-Net [65]. Deng et al. [16] report that users preferred MANet [16] over SANet [57] and AdaIN [30].

Consistency with Increasing Perturbations
Heusel et al. [26] showed that the FID is consistent with respect to increasing levels of perturbations that are applied to the images. We verify that the ArtFID ∞ inherits this property. Fig. 3 shows the ArtFID ∞ evaluated using the same image transformations as Heusel et al. [26]. The results for the ArtFID are comparable.

Ablations
We perform the following ablation experiments. All results are reported in Tab. 3.
Using ImageNet features for ArtFID computation Instead of using the modified Inception network trained on the art dataset (Sec. 3.3), we use the ImageNet features from the original Inception network to measure style matching. This shows the effectiveness of learning image representations that reflect stylistic similarity for the style matching task.
Training the Inception network only with artist labels We only use the artist labels when training the Inception network on the art dataset. This experiment highlights the importance of harnessing the information from both labels.
Training the Inception network only with style labels We only use the style labels when training the Inception network on the art dataset. Again, this shows the importance of using the information from both labels.
Using CLIP for measuring content preservation Instead of using the LPIPS metric [86] to measure content preservation, we use CLIP [59]. We observe that the LPIPS metric performs slightly better. Fig. 3. The ArtFID∞ is consistent with respect to increasing perturbations applied to the stylized images. The stylized images for this experiment were generated using the algorithm proposed by Gatys et al. [23]. The line represents the mean and the shaded area the standard deviation over 4 trials.
Using VGG for measuring content preservation Instead of using the LPIPS metric [86] to measure content preservation, we use the VGG perceptual loss [23,33]. Again, the LPIPS metric performs slightly better.

Limitations
Obtaining accurate ArtFID measurements is computationally expensive. It requires evaluating the Inception network and the LPIPS metric for several thousands of images.

Conclusion
In this work, we propose a method for quantitatively evaluating style transfer models. The goal of this work is to complement the currently used evaluation schemes by promoting the automatic comparison of different style transfer approaches and to study their respective strengths and weaknesses through indepth analyses. A given model is evaluated by measuring content preservation and style matching across many different style and content images. Content preservation is quantified using the well-established LPIPS metric. To measure style matching, we first learn suitable image representations using a large-scale art dataset. We then measure the distance between the feature distributions of Table 1. For each style transfer model, we report the user study score, the ArtFID∞ and the Deception rate.

Configuration
Spearman's ρ ↑ Two-sided p-value ↓ One-sided p-value ↓ ArtFID ArtFID∞ ArtFID ArtFID∞ ArtFID ArtFID∞ the stylized images and style images with respect to the first two moments. We evaluate our method by conducting a user study on Amazon Mechanical Turk and show that it strongly coincides with human judgment.

B Training
For training, the images are resized to 299x299. Commonly used augmentations such as random flipping are applied to the images. We use a weighted cross entropy loss for training, where the weight for a particular class is the reciprocal of the number of examples for that class. This is done to alleviate class imbalances. Tab. 4 shows the validation accuracy of the network.

C Bradley-Terry Model
In this section, we will explain the iterative algorithm that maximizes loglikelihood of the Bradley-Terry model. The explanation is mostly taken from [31]. We also refer to [31] for a study of the convergence properties. We start with an initial vector γ (1) . We use γ (1) = {1, ..., 1} in our experiments. We then iteratively update γ (k) as follows: where W i denotes the number of times that entity i was preferred in total and N ij = w ij +w ji is the number of pairings between entity i and entity j. If γ (k+1) does not satisfy the constraint i γ (k+1) i = 1, it is normalized accordingly.