{'fraction_non_alphanumeric': 0.027553576398552743, 'fraction_numerical': 0.006679654884497634, 'mean_word_length': 4.814724919093851, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Image Colorization Using a Deep Convolutional Neural Network

In this paper, we present a novel approach that uses deep learning techniques for colorizing grayscale images. By utilizing a pre-trained convolutional neural network, which is originally designed for image classification, we are able to separate content and style of different images and recombine them into a single image. We then propose a method that can add colors to a grayscale image by combining its content with style of a color image having semantic similarity with the grayscale one. As an application, to our knowledge the first of its kind, we use the proposed method to colorize images of ukiyo-e a genre of Japanese painting?and obtain interesting results, showing the potential of this method in the growing field of computer assisted art.


Introduction
Image colorization is the process of assigning colors to a grayscale image to make it more aesthetically appealing and perceptually meaningful. This is known to be a sophisticated task that often requires prior knowledge about the image content and manual adjustments in order to achieve artefactfree quality. Furthermore, since objects can be in different colors, there are many possible ways to assign colors to pixels in an image, which means there is no unique solution to this problem.
There are two main approaches for image colorization: one that requires user to assign colors to some regions and extends such information to the whole image, and another one that tries to learn the color of each pixel from a color image with similar content. In this paper, we use the latter approach; we extract the information about color from an image and transfer it to another image.
Recently, deep learning has gained increasing attention among researchers in the field of computer vision and image processing. As a typical technique, convolutional neural networks (CNNs) have been well-studied and successfully applied to several tasks such as image recognition, image reconstruction, image generation, etc. A CNN consists of multiple layers of small computational units that only process portions of the input image in a feed-forward fashion. Each layer is the result of applying various image filters, each of which extracts a certain feature of the input image, to the previous layer. Thus, each layer may contain useful information about the input image at different levels of abstraction.
With the evolution of computational resources, especially the computing power of GPUs, it has become possible to train very deep CNNs, and they have achieved some remarkable results recently. For example, a deep CNN (He et al., 2015) has surpassed human-level performance on ImageNet classification, or an adversarial network (Radford et al., 2015), in which two CNNs are trained simultaneously, is capable of generating plausible-looking images of many kinds of objects.
These amazing successes of CNNs have motivated us to further investigate and explore their potential in the aforementioned image colorization problem. Gatys et al. (2015) presented an application of deep neural networks that could learn to transfer style from an image to another one. Given an image the content of which is to be preserved (content image) and another image the style of which is to be transferred (style image), they passed both images into a pre-trained CNN and extracted the content representation and the style representation, respectively. They then did that to a noisy image and made changes to it until they obtained similar representations as of the content image and the style image. Indeed, this is an optimization problem in which the objective is to minimize the loss of reconstructing content and style simultaneously. As mentioned in their paper, they used gradient descent to solve this problem.

Related work
We follow Gatys et al.'s approach in our work, but use a different optimization method since gradient descent normally requires much fine-tuning of parameters to work well on a specific problem.

Methodology
In this paper, we combine content of a grayscale image and style of a color image, which results in colorizing the grayscale image. More details of our methodology are described below.
Considering a certain layer in the network, we denote the number of feature maps as and the size ( ) of a feature map in that layer as . Furthermore, we denote the content image, the style image, and the noisy image as , , and respectively. The content loss is calculated by the following equation: where ∈ and ∈ are the content representations, i.e., the features of and , respectively. On the other hand, the style representation of an image is given by the Gram matrix ∈ : The style loss at layer is calculated by 1 4 , , where and are the style representations of and , respectively. However, as shown in Gatys et al.'s work, it is better to consider style losses at multiple layers, therefore the style loss function is defined as , , where is the corresponding weighting factor of . Finally, the total loss function is defined as the weighted average of the content loss and the style loss: , , , , In order to improve optimization efficiency, we use L-BFGS-a quasi-Newton method that approximates the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm using a limited amount of computer memory-as an optimization method to find the minimum of . Moreover, we decrease the ratio / by 0.25% after each iteration, which is different from the related work, so that the colorization process does not give an implausible result by assigning too many colors to the original image.
As done in the related work, we also utilize the VGG-19 network (Simonyan and Zisserman, 2015) which was originally trained on the ImageNet dataset for image recognition. We use only layer 'conv4_2' for content reconstruction and 5 layers 'conv1_1', 'conv2_1', 'conv3_1', 'conv4_1', 'conv5_1' for style reconstruction. The weighting factor is set to 1/5 for all layers in the style loss function.
We applied our proposed method to colorize images of ukiyo-e, which is a genre of Japanese painting and represents one of the highpoints of Japanese cultural achievement. Figure 1 shows some examples of generated images, each of which was obtained after running 1000 iterations on a Torch implementation (Johnson, 2015). We also compared different optimization methods and illustrated the results in Fig. 2. For stochastic gradient descent (SGD), we had to do fine-tuning of parameters manually and showed only results generated with the best parameters we could find. In contrast, L-BFGS not only required no fine-tuning, but produced images that look slightly better. For both optimization methods, it can be seen that decreasing the style weight over iterations helped improve the visual quality of generated images.

Conclusions
In this paper, we presented a reliable method for colorizing grayscale images that uses a CNN to extract color information from an image and transfer to another image. We showed examples of plausible-looking generated images. Our results indicate that the presented method can be used as a creativity tool to assist human artists in near future.