{'fraction_non_alphanumeric': 0.03166067656177124, 'fraction_numerical': 0.011440643090345414, 'mean_word_length': 4.743775606681374, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence

This paper tackles the automatic colorization task of a sketch image given an already-colored reference image. Colorizing a sketch image is in high demand in comics, animation, and other content creation applications, but it suffers from information scarcity of a sketch image. To address this, a reference image can render the colorization process in a reliable and user-driven manner. However, it is difficult to prepare for a training data set that has a sufficient amount of semantically meaningful pairs of images as well as the ground truth for a colored image reflecting a given reference (e.g., coloring a sketch of an originally blue car given a reference green car). To tackle this challenge, we propose to utilize the identical image with geometric distortion as a virtual reference, which makes it possible to secure the ground truth for a colored output image. Furthermore, it naturally provides the ground truth for dense semantic correspondence, which we utilize in our internal attention mechanism for color transfer from reference to sketch input. We demonstrate the effectiveness of our approach in various types of sketch image colorization via quantitative as well as qualitative evaluation against existing methods.


Introduction
Early colorization tasks [48,21,22] have focused on colorizing a grayscale image, which have shown great progress so far. More recently, the task of colorizing a given sketch or outline image has attracted a great deal of attention in both computer vision and graphics communities, due to its significant needs in practice. Compared to a grayscale im-age, which still contains the pixel intensity, a sketch image is information-scarce, making its colorization challenging in nature. To remedy this issue, generally two types of approach of imposing additional conditions to the sketch image have been explored: user hints and reference image.
As explained in Section 2.2, there are previous works utilizing a reference or already-colored image, which shares the same semantic object of the target image. It requires an ability for the model to establish visual correspondences and inject colors through the mappings from the reference to the target. However, due to the huge information discrepancy between the sketch and reference, the sketch colorization guided by the reference is still under-explored compared to other sketch-based tasks (Section 2.1). Moreover, there are few datasets containing the labels of the correspondence between the two images, and the cost of generating a reliable matching of source and reference becomes a critical bottleneck for this task over a wide range of domains.
In this work, we utilize an augmented-self reference which is generated from the original image by both color perturbation and geometric distortion. This reference contains the most of the contents from original image itself, thereby providing a full information of correspondence for the sketch, which is also from the same original image. Afterward, our model explicitly transfers the contextual representations obtained from the reference into the spatially corresponding positions of the sketch by the attention-based pixel-wise feature transfer module, which we term the spatially corresponding feature transfer (SCFT) module. Integration of these two methods naturally reveals groundtruth spatial correspondence for directly supervising such an attention module via our similarity-based triplet loss. This direct supervision encourages the network to be fully optimized in an end-to-end manner from the scratch and does not require any manually-annotated labels of visual correspondence between source-reference pairs. Furthermore, we introduce an evaluation metric which measures how faithfully the model transfers the colors of the reference in the corresponding regions of sketch.
Both qualitative and quantitative experiments indicate that our approach exhibits the state-of-the-art performance to date in the task of information-scarce, sketch colorization based on a reference image. These promising results strongly demonstrate its significant potentials in practical applications in a wide range of domains.

Related work 2.1. Sketch-based Tasks
Sketch roughly visualizes the appearances of a scene or object by a series of lines. Thanks to its simple, easy-todraw, and easy-to-edit advantages, sketch has been utilized in several tasks including image retrieval [20], sketch recog-nition [25], sketch generation [3,29], and image inpainting [33]. However, due to the lack of texture and color information in sketch image, the research on sketch-based image colorization, especially reference-based colorization, is quite challenging and still under-explored.

Conditional Image Colorization
The automatic colorization has a limitation that users cannot manipulate the output with their desired color. To tackle this, recent methods come up with the idea of colorizing images with condition of the color given by users, such as scribbles [38], color palette [49,28,45], or text tags [18]. Even though these approaches have shown the impressive results in terms of the multi-modal colorization, they unavoidably require both precise color information and the geometric hints provided by users for every step.
To overcome the inconvenience, an alternative approach, which utilizes an already colored image as a reference, has been introduced. Due to the absence of geometric correspondence at the input level, early studies [17,1,26,4,7,2] utilized low-level hand-crafted features to establish visual correspondence. Recent studies [10,47,40] compose the semantically close source-reference pairs by using features extracted from the pre-trained networks [10,47] or color histogram [40] and exploit them in their training. These pair composition techniques however tend to be sensitive to domains, thereby limit their capability in a specific dataset.
Our work presents a novel training scheme to learn visual correspondence by generating augmented-self reference in the self-supervised manner at the training time, and then demonstrates it's scalability on various type of datasets.

Proposed method
In this section, we present our proposed model in detail, as illustrated in Fig. 2. We first describe overall workflow of the model and its two novel components called (1) Augmented-Self Reference Generation (Section 3.2) and (2) Spatially Corresponding Feature Transfer Module (Section 3.3). We then present our loss functions in detail.

Overall Workflow
As illustrated in Fig. 2, given a color image I in our dataset, we first convert it into its sketch image I s using an outline extractor. Additionally, we generate an augmentedself reference image I r by applying the thin plate splines (TPS) transformation. Taking these two images I s and I r as inputs, our model first encodes them into activation maps f s and f r using two independent encoders E s (I s ) and E r (I r ), respectively.
To transfer the information from I r to I s , we present a SCFT module inspired by a recently proposed self-attention mechanism [41], which computes dense correspondences between every pixel pair of I r to I s . Based on the visual mappings from SCFT, context features fusing the information between I r and I s passes through several residual blocks and our U-net-based decoder [35] sequentially to obtain the final colored output.

Augmented-Self Reference Generation
To generate a reference color image I r for a given sketch image I s , we apply to original color image I two nontrivial transformations, appearance and spatial transformation. Since I r is essentially generated from I, these processes guarantee that the useful information to colorize I s exists in I r , which encourages the model to reflect I r in the colorization process. The details on how these transformations operate are described as follows. First, the appearance transformation a(·) adds a particular random noise per each of the RGB channel of I. The resulting output a(I) is then used as the ground truth I gt for the colorization output of our model. The reason why we impose color perturbation for making reference is to prevent our model from memorizing color bias, which means that a particular object is highly correlated with the single ground truth color in train data (i,e., a red color for apples). Given different reference in every iteration, our model should reconstruct different colored output for the same sketch, by leveraging I r as the only path to restore I gt . In other words, it encourages the model to actively utilize the information from E r not just from E s and generates reference-aware outputs at test time. Afterwards, we further apply the TPS transformation s(·), a non-linear spatial transformation operator to a(I) (or I gt ), resulting in our final reference image I r . This prevents our model from lazily bringing the color in the same pixel position from I r , while enforcing our model to identify semantically meaningful spatial correspondences even for a reference image with a spatially different layout, e.g., different poses.

Spatially Corresponding Feature Transfer
The goal of this module is to learn (1) which part of a reference image to bring the information from as well as (2) which part of a sketch image to transfer such information to, i.e., transferring the information from where to where. Once   obtaining this information as an attention map, our model transfers the feature information from a particular region of a reference to its semantically corresponding pixel of a sketch.
To begin with, each of the two encoders E r and E s consists of L convolutional layers, producing L activation maps (f 1 , f 2 , · · · , f L ) including intermediate outputs. Now we downsample each of them to match the spatial size of f L and concatenate them along the channel dimensions, forming the final activation map V , i.e., where ϕ denotes a spatially downsampling function of an input activation map f l ∈ R h l ×w l ×c l to the size of f lp ∈ R hp×wp×cp . "; " denotes the channel-wise concatenation operator. In this manner, we capture all the available lowto high-level features simultaneously. Now we reshape V intoV = [v 1 , v 2 , · · · , v hw ] ∈ R dv×hw , where v i ∈ R dv indicates a feature representation of the i th region of the given image and d v = L l=1 c l . We then obtain v s i ofV s and v r j ofV r from the outputs of the sketch encoder E s and the reference encoder E r , respectively. Given v s i and v r j , our model computes an attention matrix A ∈ R hw×hw whose element α ij is computed by the scaled dot product [41], followed by a softmax function within each row, i.e., where W q , W k ∈ R dv×dv represent the linear transformation matrix into a query and a key vector, respectively, in the context of a self-attention module, and √ d v represents a scaling factor. α ij is a coefficient representing how much information v s i should bring from v r j . Now we can obtain the context vector v * i of the position i as where W v ∈ R dv×dv is the linear transformation matrix into a value vector containing the color feature in a semantically related region of a reference image.
Finally, v * i is added to the original feature v s i of a sketch image to form the feature vector enriched by the information of the corresponding region in the reference image, i.e., c i is then fed into the decoder to synthesize a colored image.

Objective Functions
Similarity-Based Triplet Loss. When applying the spatial transformation s(·), each pixel value in the output image is represented as a weighted average of pixels in the input image, revealing the spatial correspondences of pixel pairs between I s and I r . In other words, we can obtain the full information of the weight w ij , which represents how much the i th pixel position of the input image, or a query, is related to the j th pixel position of the output, or a key. Then, the value of w ij can be considered as the pixel-to-pixel correspondence, which can work as the groundtruth for supervising how semantically related the pixel of the reference to a particular pixel of sketch image. Utilizing this pixel-level correspondence information, we propose a similarity-based triplet loss, which is a variant of triplet loss [39], to directly supervise the affinity between the pixel-wise query and key vectors used to compute the attention map A in Eq. (2). The proposed loss term is computed as where S(·, ·) computes the scaled dot product. Given a query vector v q as an anchor, v p k indicates a feature vector sampled from the positive region, and v n k is a negative sample. γ denotes a margin, which is the minimum distance S(v q , v p k ) and S(v q , v n k ) should maintain. L tr encourages the query representation to be close to the correct (positive) key representation, while penalizing to be far from the wrong (negatively sampled) one. This loss plays a crucial role in directly enforcing our model to find the semantically matching pairs and reflect the reference color into the corresponding position.
The reason we adopt triplet loss instead of commonly used losses such as L 1 -loss is that the latter can overly penalize the affinities between semantically close but spatially distant query and key pixel pairs. This misleading result can be mitigated by only penalizing two cases: the semantically closest pair (positive sample) and randomly-sampled except it (negative sample), which is basically a triplet loss.
We further conduct a user study to compare the effects of our triplet loss to another possible loss, i.e., L 1 -loss and no supervision. Details about the experimental settings and results are explained in Section 6.2 in the supplementary material. L1 Loss. Since the groundtruth image I gt is generated as Section 3.2, we can directly impose a reconstruction loss to penalize the network for the color difference between the output and the ground truth image as below: Adversarial Loss. The discriminator D, as an adversary of the generator, has an objective to distinguish the generated images from the real ones. The output of real/fake classifier D(X) denotes the probability of an arbitrary image X to be a real one. We adopt conditional GANs which use both a generated sample and additional conditions [34,44,15].
In this work, we leverage the input image I s as a condition for the adversarial loss since it is important to preserve the content of I s as well as to generate a realistic fake image. The loss for optimizing D is formulated as a standard crossentropy loss as Perceptual Loss. As shown in previous work [33], perceptual loss [16] encourages a network to produce an output that is perceptually plausible. This loss penalizes the model to decrease the semantic gap, which means the difference of intermediate activation maps between the generated out-putÎ and the ground truth I gt from the ImageNet [36] pretrained network. We employ a perceptual loss using multilayer activation maps to reflect not only high-level semantics but also low-level styles as where φ l represents the activation map of the l'th layer extracted at the relul 1 from the VGG19 network. Style Loss. Sajjadi et al. [37] has shown that the style loss which narrow the difference between the covariances of activation maps is helpful for addressing checkerboard artifacts. Given φ l ∈ R C l ×H l ×W l , the style loss is computed as where G is a gram matrix. In summary, the overall loss function for the generator G and discriminator D is defined as

Implementation Details
We implement our model with the size of input image fixed in 256×256 on every datasets. For training, we set the coefficients for each loss functions as follows: λ adv = 1, λ rec = 30, λ tr = 1, λ perc = 0.01, and λ style = 50. We  Table 1: Quantitative comparisons over the datasets with existing baselines by measuring FID [11] score: a lower score is better.
set the margin of the triplet loss γ = 12 for overall data. We use Adam solver [19] for optimization with β 1 = 0.5, β 2 = 0.999. The learning rate of generator and discriminator are initially set to 0.0001 and 0.0002 for each. The detailed network architectures are described in Section 6.5 of supplementary material.

Experiments
This section demonstrates the superiority of our approach on wide range of domain datasets (Section 4.1) including real photos, human face and anime (comics). We newly present an evaluation metric, named SC-PSNR described in Section 4.2, to measure the faithfulness of reflecting the style of the reference. Afterwards, we compare our method against the several baselines of related tasks quantitatively as well as qualitatively (Section 4.3). An in-depth analysis of our approach is described across Section 4.4-4.5.

Datasets
Tag2pix Dataset. We use Tag2pix dataset [18], which contains large-scale anime illustrations filtered from Dan-booru2017 [8], to train our model for comic domain. Although there are various tag labels on this dataset, we only utilize images to train the model owing to our selfsupervised training scheme. It consists of one character object with white background images. We partition into 54,317 images for train, 6036 images for test and then combine source-reference pairs by randomly sampled from the test set for evaluation. Yumi Dataset. Like Yoo et al. [46], we collect images from the online cartoon named Yumi's Cells for the outline colorization of the anime domain. The dataset contains repeatedly emerging characters across 329 episodes. With this limited variety of characters, the network is required to find the correct character matching even if there is no explicit character supervisions. We randomly split into a train set of 7,014 images and test set of 380 images, and then manually construct source-reference pairs from the testset to evaluate the performance of the models.  Table 2: Quantitative comparisons over the SPair-71k with existing baselines by measuring SC-PSNR (dB) score: a higher score is better.
SPair-71k Dataset. SPair-71k dataset [31], which is manually annotated for a semantic correspondence task, consists of total 70,958 pairs of images from PASCAL 3D+ [43] and PASCAL VOC 2012 [6]. We select two non-rigid categories (cat, dog) and one rigid category (car), of which we can gather sufficient data points from ImageNet [36]. Note that this dataset is used to measure SC-PSNR (Section. 4.2) score only for the evaluation purpose. ImageNet Dataset. As above-mentioned, we collect subclasses that correspond to three categories (i.e., cat, dog, car) from ImageNet [36] dataset and use them for training data. Images in each class are randomly divided into two splits with an approximate ratio of 9:  Table 3: FID scores [11] according to the ablation of loss function terms described in Section 4.4. A lower score is better. approaches.

Evaluation Metrics
Semantically Corresponding PSNR. This work proposes a novel evaluation metric to measure how faithfully the model transfers the style of reference in the corresponding regions. In the traditional automatic colorization setting where a groundtruth image is available, pixel-level evaluation metric, such as peak signal-to-noise ratio (PSNR), has been widely used. In reference-based colorization setting, however, there is no ground truth that have both the shape of the content and the style of the reference.
The key idea behind the semantically corresponding PSNR (SC-PSNR) is leveraging the datasets created for keypoint alignment tasks [6,43,31], thereby providing patch-level groundtruth. We use SPair-71k dataset [31] which contains semantically corresponding annotation pairs between two different images. Only the pixel values in a certain size of patch surrounding the corresponding keypoints of two images are used instead of the whole pixels for computing mean square error (MSE), and then PSNR is computed with the MSE. We refer to this measurement as the SC-PSNR. Fig. 6 shows first and last two examples of images queried by the leftmost image. The list of images are retrieved in a decreasing order of the value of SC-PSNR being computed with query. This figures demonstrates that this metric captures perceptually plausible distance of the pixel values between the keypoint regions of two images. Fréchet Inception Distance (FID) [11]. FID is a wellknown metric for evaluating the performance of a gener-…    ative model by measuring the Wasserstein-2 distance between the feature space representations of the real images and its generated outputs. A low score of FID indicates that the model generates the images with quality and diversity close to real data distribution.

Comparisons to Baselines
We compare our method against recent deep learningbased approaches on the various types of datasets both qualitatively and quantitatively. The baselines are selected from not only the colorization task [28, 40] but also the related problems tackling multi-modal image generation, such as exemplar-guided image translation [13,24] and style transfer [12]. Fig. 4 shows the overall qualitative results of our model and other baselines on 5 different datasets. Datasets vary from real image domain like ImageNet or Human face dataset to sketch image domain like Edges→Shoes, Yumi's Cells, and Tag2pix. The leftmost and second column are sketch and reference, respectively. On every dataset our model brings the exact colors from the reference image and injects them into the corresponding position in the sketch.
For example, our model colorizes the character's face in third row with red color from the reference, while baselines tend not to fully transfer it. Likewise, in fifth row, inner side of the shoes and shoe sole are elaborately filled with the color exactly referencing the exemplar image.
We report on Table 1 the FID score calculated over the 7 different datasets. Our method outperforms the existing baselines by a large margin, demonstrating that our method has the robust capability of generating realistic and diverse images. Improved scores of our model with triplet loss indicates that L tr plays a beneficial role in generating realistic images by directly supervising semantic correspondence. Table 2 presents the other quantitative comparisons in regard to the SC-PSNR scores as described in Section 4.2. We measure SC-PSNR only over cat, dog and car dataset which are subclasses belonging to both ImageNet and SPair-71k [31]. Our method outperforms all the baseline models, demonstrating that our model is superior at establishing visual correspondences, and then generating suitable colors.
We conduct a user study for human evaluation on our model and other existing baselines, as shown in Fig. 8. The detailed experimental setting is described in Section 6.2 in the supplementary material. Our model occupies a large percentage of Top1 and Top2 votes, indicating that our method better reflects the color from the reference and generates more realistic outputs than other baselines.

Analysis of Loss Functions
We ablate the loss functions individually to analyze the effects of the functions qualitatively, as shown in Fig. 5 and quantitatively, as shown in Table 3. When we remove L adv , output image contains inaccurate colors emerging in the background and dramatically appears unrealistic. Without L tr , character's back hair, forehead and ribbon tail are colorized with wrong color or even not colorized. The FID score in Table 3 third row also represents that model generates unrealistic output. This degraded performance is due to the absence of supervision which encourages to match the semantically close regions between content and reference. When we remove L perc and L style , the colorization tends to produce color bleeding or visual artifacts since there is no constraint to penalize the model for the semantic difference between the model output and the ground truth. Image generated with full losses have exact colors in its corresponding regions with fewer artifacts. Fig. 7 shows an example of an attention map A learned by our SCFT module. In this module, each pixel from the sketch is used as a query to retrieve the relevant local information from the reference. In the case of left-eye region as a query (red square in (a)), we visualize the top three, highlyattentive regions in the reference image (a highlighted re-

Conclusions
This paper presents a novel training scheme, integrating the augmented-self reference and the attention-based feature transfer module to directly learn the semantic correspondence for the reference-based sketch colorization task. Evaluation results demonstrate that our SCFT module exhibits the state-of-the-art performance over the diverse datasets, which demonstrates the significant potentials in practice. Finally, SC-PSNR, a proposed evaluation metric, effectively measures how the model faithfully reflects the style of the exemplar.

A. Supplementary Material
This supplementary document presents additional details of the paper. Section A.1 discusses the effects of our spatially corresponding feature transfer mechanism with quantitative results. Section A.2 demonstrates the human evaluation results that compare ours against baseline methods. Afterwards, Section A.3 reports implementation details including network architectures, the processes of generating augmented-self reference images, and other training details. Comparisons to an existing study which shares similar network architectures are described in Section A.4. Lastly, Section A.5 addresses the case where a reference image does not exist. Qualitative results generated by our method are also shown throughout the document.

A.1. Effects of Aggregation Methods
The key assumption behind SCFT is that integrating spatially aligned reference features with content features would help reflect the exact color from the reference into corresponding positions. To prove this assumption, we compare our SCFT with two simple types of aggregation methods as shown in Fig. 9. Methods are as follows: (a) representations of the reference are simply added to the features of the content. (b) AdaIN [12] is utilized to transfer the style of reference by aligning the channel-wise mean and variance of content to match those of reference. (c) our SCFT module.
Qualitative comparison over three methods is shown in Fig. 10. The leftmost column contains sketch and reference, while next three columns contain colorized images from (a), (b) and (c), respectively. Method (a) tends not to perfectly locate the corresponding regions and results in colorizing car with overly yellowish color, which is mainly background color in the exemplar. Method Quantitative results comparing these methods are represented in Table 4. The network with SCFT module produces the most realistic results over most of the datasets. This is because the SCFT module properly aligns the corresponding local regions between the sketch and the reference image by using the attention matrix A. On the other hand, the method (a) and (b) are not capable of aligning the local features of the reference with those of the sketch, resulting in low FID scores.
In Yumi's Cells [32] dataset, however, the SCFT module produced worse FID score than the others. The potential reason we assume is that the sketch and the reference we randomly pair for the inference time often contain different types of objects, e.g., Yumi (a human) and cells (non-human), which may have negatively impacted the colorization output.

Content / Reference
(a) Addition (b) AdaIN (c) Ours (SCFT) Figure 10: A qualitative example obtained from three different aggregation methods as shown in Figure 9.

A.2. User Study
We conduct two different human evaluation on the colorization outputs over various datasets. First, we randomly select ten sets of images per dataset, which contain the generated images from our method and other baselines. Second, we also randomly select ten sets of images for every dataset, and those contain the images obtained from the model trained with triplet loss, L1-loss and no supervision for correspondence, respectively. For both cases, participants with no prior knowledge in this work are asked to rank them in terms of two types of questions sequentially as follows: • Overall Colorization Quality and Realism How natural does the colorized image look? This question requires users to evaluate the overall quality of the generated colorization given an input sketch. The generated image should be perceptually realistic without any artifacts or color bleeding across sketches.
• Detailed Reflection of Reference How well is the colors of the reference image is reflected to a given sketch part by part? This question asks users to determine whether the particular color from a reference is injected into the corresponding regions in the sketch. For example, given an comics character image with green hair wearing a blue shirt as a reference, the generated output is expected to contain these colors at its corresponding hair and clothing part, respectively.
As seen in Fig. 23 and 24, superior measures indicate that our approach generates both more realistic and more faithfully colorized image than other methods. For both question type 1 and 2, it can be observed that our approach achieves the rank 1 votings more than 50% over all the dataset we adopt for user study. When asked the first question on Comics domain dataset including Tag2pix [18] and Yumi's Cell [32], Style2Paints [28] perform realistic generation quality comparable to our method with a small gap in top 1 rate. This notable measure is obtained as Style2Paints [28] is a adept baseline especially on comic domain. However, the difference in top 1 rate increases as the users are asked to choose based on faithful colorization performance. The results demon-  Table 4: FID scores [11] according to different aggregation methods. strate that our model utilizes the right color from the reference, which results in both realistic and exquisitely colorized output.
The results in Fig. 25 demonstrates that the model trained with triplet loss obtains more realistic and faithfully colorized outputs than with L1-loss or no loss. Furthermore, along with the explanation of similarity-based triplet loss in Section 3.4 of the paper, these results support that the supervision for semantic correspondence with the L1-loss leads to the inferior colorization performance even compared to the model without any supervision.

A.3. Implementation Details
This section provides the implementation details of our model, complementary to Section 3.5 of the paper. Augmented-Self Reference Generation To automatically generate a sketch image from an original color image, We utilize a widely-used algorithm called XDoG [42]. The outputs, however, often involves superfluous edges, so in order to suppress them, we apply Gaussian blurring (σ = 0.7) to the original images before extracting sketches. The appearance transformation a(·) adds randomly sampled value from a uniform distribution on [-50, 50] to each of the RGB channels of the original image. Encoder Our generator G contains two types of encoder, Es and Er. Both of them share the same architecture shown in Table 5, except for the number of input channels of the first layer, where Es takes a single-channel, binarized sketch input while Er takes a three-channel, RGB reference image. We utilize the an average pooling function for downsampling ϕ in Section 3.3 of the paper.

Layer
Encoder L1 Conv (I:C  Resblocks We place four stacked residual blocks [9] with a kernel size of 3 and a stride of 1. Batch normalization [14] follows each convolutional block, and ReLU is used as the activation function.
Discriminator We adopt our discriminator architecture as Patch-GAN [15]. We utilize the LSGAN [30] objective for the stable training.
Training Details For all the experiments, our network is trained using Adam optimizer [19] with β1 = 0.5 and β2 = 0.999. We set an initial learning rate for the generator as 0.0001 and that for the discriminator as 0.0002. We train the model for the first 100 epochs using the same learning rate, and then we linearly decay it to zero until the 200 epochs. We set the margin value γ = 12 for our triplet loss (Eq. 5 in the paper). The batch size is set as 16. The parameters of all our models are initialized according to the normal distribution which has a mean as 0.0 and a standard deviation as 0.02.
Baselines We exploit Sun [40] and Style2Paints [28] as the sketch image colorization methods, Huang [2018] [13], and Lee [24] as the image translation methods and Huang [2017] [12] as the style transfer method as our baselines. For Style2Paints [28], we generate the images based on the publicly available Style2Paints V3 in a similar manner to Tag2pix [18]. For the other methods, we utilize the officially available codes to colorize images after training them on our datasets.

Source
Reference Output (a) (b) Figure 11: Qualitative results of our Zhang et al. [47] given gray-scale source image (row (a)) and sketch image (row (b)). In contrast to the output in the row (a), output in (b) fails to colorize the eyes with the color from the reference and spreads the yellow color over the face.
In this section, we discuss the detailed comparison between our method and Zhang et al.. These two works have similarity in that they both exploit geometric distortion for data augmentation and semantic correspondence module for color guidance. However the significant difference of our model against Zhang et al. lies in (1) direct supervision of semantic correspondence and (2) generalized attention module. Direct supervision Our model directly supervises the attention module via a triplet loss, which enables the optimization of the attention module in an end-to-end manner. This fully trainable encoder encourages to generate plausible results over a wide range of datasets from real-world photos to comic images, as show in Fig. 4 of the paper and Fig. 17. In contrast, Zhang et al. requires a pre-trained, already reliable attention module, which is only indirectly supervised via a so-called contextual loss. According to Geirhos et al. (2019) [5], the features extracted from the Ima-geNet pre-trained encoder may be severely degraded for a sketch image due to large domain shifts. In this sense, Zhang et al.'s work may not be easily applicable to sketch image colorization tasks, and the examples of failure case are shown in Fig. 11. We reimplemented the code of Zhang et al., trained and tested the model over cat dataset. As this baseline exploits the ImageNet pre-trained encoder, row (a) shows that it produces the plausible colorized output given gray-scale source image. However, when given information scarce sketch image (row (b)), it fails to obtain the dense correspondence with the reference image, resulting in degraded output. Generalized attention module Inspired by the self-attention module in the Transformer networks, our attention module involves different query, key, and value mappings for flexibility, while Zhang et al. use a relatively simple module. More importantly, in terms of value vectors, Zhang et al. uses only raw color values, but ours uses all the available low-to high-level semantic information extracted from multiple layers. In this respect, ours is capable of transferring significantly richer contextual information than just low-level color information.

A.5. Colorization without reference.
Our main scope is focused on the colorization task with a reference available, but we can easily extend our method for noreference cases by occasionally providing a zero-filled image as a reference to the networks during the training time. We feed the zero-filled image to our model as a reference with a ratio of 9:1 at the training time. As shown in Fig. 12, we confirm that our network still generates a reasonable quality of colorization output at test time. In this case, the zero-filled reference image does not have any information to guide. Therefore, the model is encouraged to synthesize an output image with colors that often appear in trainset conditioned on the sketch image. We recall that the main goal of this work is not restricted to generating the original image.