{'fraction_non_alphanumeric': 0.038108065282839484, 'fraction_numerical': 0.012002852162890192, 'mean_word_length': 4.84375, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-Supervised Intrinsic Image Decomposition Network Considering Reflectance Consistency

We propose a novel intrinsic image decomposition network considering reflectance consistency. Intrinsic image decomposition aims to decompose an image into illumination-invariant and illumination-variant components, referred to as ``reflectance'' and ``shading,'' respectively. Although there are three consistencies that the reflectance and shading should satisfy, most conventional work does not sufficiently account for consistency with respect to reflectance, owing to the use of a white-illuminant decomposition model and the lack of training images capturing the same objects under various illumination-brightness and -color conditions. For this reason, the three consistencies are considered in the proposed network by using a color-illuminant model and training the network with losses calculated from images taken under various illumination conditions. In addition, the proposed network can be trained in a self-supervised manner because various illumination conditions can easily be simulated. Experimental results show that our network can decompose images into reflectance and shading components.


I. INTRODUCTION
Decomposing a natural image into illumination-invariant and -variant components enables us to easily modify the image. For example, since a low-light image can be considered as an image having lower illumination-variant components than those of the corresponding well-exposed image, it can be enhanced by amplifying its illumination-variant components [1][2][3]. Similarly, white balancing can be done by changing the color of illumination-variant components. For this reason, intrinsic image decomposition [4,5] has so far been studied to decompose a natural image into two such components.
Intrinsic image decomposition is based on the Retinex theory [6]. In this theory, a natural image consists of the reflectance and shading of a scene, where the reflectance and the shading correspond to the illuminationinvariant component and the illumination-variant component, respectively. To enable the decomposition, various methods have so far been proposed [6][7][8][9][10][11][12][13][14][15][16]. An early attempt at intrinsic image decomposition used imagegradient information [6]. The gradient-based method estimates reflectance by thresholding image gradients under the assumption that large gradients are due to spatial variations in reflectance and small gradients are due to light shading. However, the effectiveness of the gradient-based method is limited because it is heuristic. For this reason, optimization-based decomposition methods that utilize sparse modeling or Bayesian inference have also been widely studied [8,17,18]. In addition, recent methods Tokyo Metropolitan University, Tokyo, Japan Corresponding author: Hitoshi Kiya Email: kiya@tmu.ac.jp for intrinsic image decomposition are based on deep neural networks (DNNs). These DNN-based methods significantly improve the performance of the decomposition compared with conventional gradient-or optimization-based approaches, by supervised learning [9,11,13,15].
In intrinsic image decomposition, there are three premises regarding consistency: reconstruction consistency, reflectance consistency in terms of illumination brightness, and reflectance consistency in terms of illumination colors. Most conventional methods for intrinsic image decomposition use a white-illuminant model that does not consider illumination color, and thus consider only a part of these premises, i.e., reconstruction consistency and reflectance consistency (brightness). In such a case, reflectance components decomposed by these methods are affected by illumination-color conditions. Some conventional methods [13] consider all of the premises by using a color-illuminant model that considers illumination color. In these methods, DNNs are trained by using a highlysynthetic dataset or a human-labeled dataset of a real scene [5,8,19]. However, such datasets are insufficient to generalize real scenes. Although unsupervised decomposition methods have recently been proposed [3,10,12,14,16], they still have limited performance.
To solve these problems, in this paper, we propose a novel intrinsic image decomposition network that considers both all three premises and the problem with data. To consider the premises, we use a color-illuminant model and train the network with losses calculated by using image sets, where each image set consists of images capturing the same objects under various illuminant-brightness and -color conditions. In addition, we will show that such an image set can easily be generated from a single raw image. Namely, our network can be trained in a self-supervised An RGB vector at pixel (x, y) of an image R(x, y) An RGB vector at pixel (x, y) of a reflectance S(x, y) A scalar value at pixel (x, y) of a shading in gray scale c An RGB vector indicating illumination color S(x, y) An RGB vector at pixel (x, y) of a shading manner by using a general raw image dataset. Therefore, the difficulty in preparing a large amount of data can be overcome. Since the proposed network can perform intrinsic image decomposition so that the resulting reflectance is unaffected by the illumination conditions, the proposed network will be useful for image enhancement while preserving object color and more realistic white balance adjustment. We evaluate the performance of the proposed intrinsic image decomposition network in terms of the robustness against illumination-brightness/-color changes. To measure the robustness, we utilize images taken under various illumination-brightness and -color conditions and decompose them into reflectance and shading components. By comparing the reflectance components for the images on the basis of the peak signal-to-noise ratio (PSNR), mean squared error (MSE), and structural dissimilarity (DSSIM), the robustness against illumination-brightness/color changes can be evaluated. Experimental results show that the proposed decomposition network is robust against illumination-brightness/-color changes. In addition, we will discuss the effectiveness of the proposed network through a simulation experiment using the MIT intrinsic images dataset [5].

II. PRELIMINARIES
In this section, we briefly summarize intrinsic image decomposition and its issues. We use the notations shown in Table 1 throughout this paper.

A) Intrinsic image decomposition
The goal of intrinsic image decomposition is to decompose a given image into a pixel-wise product of an illuminantdependent component called reflectance and an illuminantindependent component called shading as shown in Fig. 1.
Here, there are two main models for intrinsic image decomposition as follows.
White-illuminant model, which describes the relationship among an RGB image, reflectance, and shading as where (x, y) indicates a pixel coordinate. I(x, y) and R(x, y) are 3-dimensional RGB vectors for the image and reflectance, respectively, and S(x, y) is a scalar value for shading. This model assumes the illumination color is white because S(x, y) is a scalar value. Color-illuminant model, which describes the relationship among an RGB image, reflectance, and shading as where I(x, y), S(x, y), and R(x, y) are 3-dimensional RGB vectors for the image, reflectance, and shading, respectively. This model considers illumination color because S(x, y) is an RGB vector.

B) Scenario
In intrinsic image decomposition, there are three premises:

Reconstruction consistency
The product of the estimated reflectance and shading matches the corresponding original image, as shown in Eq. (1) or Eq. (2).

Reflectance consistency (brightness)
Reflectances are invariant against a change in illumination brightness.

Reflectance consistency (color)
Reflectances are invariant against a change in illumination colors.
Most conventional work on intrinsic image decomposition uses the white-illuminant model in Section II.A) and thus considers only a part of these premises, i.e., reconstruction consistency and reflectance consistency (brightness). However, the white-illuminant model cannot satisfy reflectance consistency (color) because illumination color is not considered in this model. In such a case, reflectance components decomposed by these methods are affected by illumination-color conditions.
For this reason, the color-illuminant model is used in recent research work. In [13], all three premises are considered by training a DNN by using videos taken by a camera at a fixed location. However, the DNN still has limited performance due to there being a limited amount of real data for training.
Because of such situations, in this paper, we propose a novel deep intrinsic image decomposition network that considers all three premises and the problem with data. To consider the premises, losses are calculated by using image sets, where each image set consists of raw images capturing the same objects under various illumination-brightness and -color conditions. In addition, we will show that such an image set can easily be generated from a single raw image. Namely, our network can be trained in a self-supervised manner by using a general raw image dataset. Therefore, the problem with the amount of data can be overcome.

III. PROPOSED INTRINSIC IMAGE DECOMPOSITION NETWORK
In this paper, we aim to decompose an image into reflectance and shading by using a deep neural network.
The key idea of our approach is to utilize image sets where each set consists of images capturing the same objects under various illuminant-brightness and -color conditions in consideration of the three premises in Section II.B). We will show that such an image set can easily be generated from a single raw image, in Section IV.

A) Overview
For the proposed method, we consider the colorillumination model in Eq. (2). Additionally, we assume that the illumination color of an image is the same for all pixels. As a result, shading is written as where S(x, y) is shading in gray scale, and c is an RGB vector indicating the illumination color. Hence, the goal of the proposed method is to obtain reflectance R, grayshading S, and RGB vector c from given input image I. Figure 2 illustrates the architecture of the proposed network. Our network has a single encoder and three decoders. With the encoder, input image I is transformed into feature maps that are fed into decoders. An estimationR of reflectance in RGB color space is directly obtained as an output of a reflectance decoder. In contrast, an estimationŜ of shading is given as the product of estimated illumination colorĉ and estimated shadingŜ in gray scale.ĉ andŜ are obtained by an illumination-color decoder and a shading decoder, respectively.

B) Loss functions
Let the n-th image set in the training dataset be {I} n To fulfill all three premises in Section II.B), our network is trained to minimize loss L calculated by using I 1 , I 2 ∈ {I} n as where L reconst and L reflect are loss functions for considering reconstruction and reflectance consistencies, respectively. For the reconstruction consistency, in accordance with Eq. (2), we use L reconst given as whereÎ i , i ∈ {1, 2} is an image reconstructed from estimated reflectanceR i , gray-scale shadingŜ i , and illumination colorĉ i for image I i . From Eqs. (2) and (3),Î i is calculated aŝ L 1 is the mean absolute error, and L cos is cosine similarity given by For the reflectance consistencies, we designed L reflect as where vector v = (0.2126, 0.7152, 0.0722) consists of coefficients for calculating the luminance of an RGB vector, and L lum is utilized for normalizing the luminance of reflectance to 0.5. By normalizing the luminance to 0.5, our network can output reflectanceR i so that it is consistent regardless of the illumination-brightness conditions. Moreover, by minimizing 1 -distance between c i andĉ i , reflectanceR can also be consistent regardless of illumination-color conditions. λ 1 , λ 2 , λ 3 , λ 4 , and λ 5 are weights of the loss terms, In practice, we empirically set λ 1 = 3, λ 2 = 1, λ 3 = 2, λ 4 = 1, and λ 5 = 1 as weights.  Fig. 3. Network architecture of encoder and reflectance/shading decoders. Each box denotes multi-channel feature map produced by each layer. Number of channels is denoted above each box. Feature map resolutions are denoted to left of boxes.

C) Network architecture
The proposed network consists of an encoder, a reflectance decoder, a shading decoder, and an illuminant-color decoder. Figure 3 shows the network architecture of the encoder and reflectance/shading decoders in detail. The input image is an RGB color image of H × W pixels. For the encoder and the reflectance/shading decoders, we use four types of layers as shown in Fig. 3: 3 × 3 Conv. + BN + ReLU, which calculates a 3 × 3 convolution with a stride of 1 and a padding of 1. After the convolution, batch normalization [20] and the rectified linear unit activation function [21] (ReLU) are applied. Two adjacent 3 × 3 Conv. + BN + ReLU layers will have the same number K of filters, where K (i.e., the number of channels) is denoted in Fig. 3. 1 × 1 Conv. + ReLU, which calculates a 1 × 1 convolution with a stride of 1 and without padding. After the convolution, ReLU is applied. The number of filters in the layer is 3. 3 × 3 Conv. w/ stride 2, which downsamples feature maps using a 3 × 3 convolution with a stride of 2 and a padding of 1. 4 × 4 Transposed conv. w/ stride 1/2, which upsamples feature maps using a 4 × 4 transposed convolution with a stride of 1/2 and a padding of 1. The proposed network has concatenated skip connections between the encoder and the decoders like U-Net [22], Feature maps from intermediate layers in the encoder are concatenated with those from intermediate layers in the reflectance and shading decoders.
In addition to these three sub-networks, we utilize an illumination-color decoder that estimates the illumination color of a given image. The decoder consists of double 3 × 3 Conv. + BN + ReLU layers, a global average pooling layer, and a 1 × 1 conv. + ReLU layer. The 3 × 3 Conv. + BN + ReLU layers all have 64 filters, and the 1 × 1 conv. + ReLU layer has 3 filters. As a result, we can obtain a 3dimensional RGB vector that means the illumination color for each input image.

IV. TRAINING PROPOSED NETWORK IN SELF-SUPERVISED MANNER
To calculate the loss function for the proposed network, images of a single scene taken under various illuminationbrightness and -color conditions are required. However, it is very costly to collect such images. For this reason, we generate pseudo images from raw images and use them for training the proposed network.

A) Simulating brightness change
A change in illumination brightness, i.e., radiance, affects image brightness. Here, we assume that camera parameters except for the shutter speed are fixed, although the image brightness can be controlled by adjusting various camera parameters. In this case, the image brightness is determined by the radiant power density at an imaging sensor, i.e., irradiance, and shutter speed, where the irradiance is proportional to the radiance.
Under the assumption that a scene is static during the time that the shutter is open and the sensor has a linear response with respect to the light intensity, the image brightness is proportional to the radiance (and also the shutter speed) [23]. Hence, the change in illumination brightness is simulated by multiplying an image I by a scalar value of 2 v as where it is equivalent to the exposure compensation of v[EV] [23].

B) Simulating color change
The effect of illumination-color change is already considered in the research area of white balancing. White balancing aims to remove the effects of illumination color on an image. Typically, white balancing is performed by where and M A is a full-rank matrix with a size of 3 × 3 for color space conversion. Parameters α, β, and γ are non-negative, and they are calculated from the ideal white illumination and illumination color c of an input image. By using the inverse of M WB , we can perform inverse white balancing as where M c = M −1 WB . This corresponds to simulating illumination-color change.
For these reasons, illumination-brightness and -color changes can be simulated in accordance with Eqs. (10) and (11), respectively, by using various v, α, β, and γ. Images generated by using the equations can be used for training the proposed network.

C) Procedure for data generation
In accordance with Eqs. (10) and (13), we randomly simulate images of a single scene taken under various illuminant-brightness and -color conditions, at every step of training. Images I i are generated from a raw image I raw as follows.
where v i is a uniform random number in the range of where M ci is a diagonal matrix having c i as the main diagonal, and c i is a uniform random vector in [0.9, 1.1] 3 .

V. SIMULATION
We performed two simulation experiments to evaluate the proposed network in terms of the three consistencies, i.e., reconstruction consistency, reflectance consistency (brightness), and reflectance consistency (color). A) Simulation using raw images

1) Experimental conditions
In this experiment, 45 image sets were generated from 45 raw images from the RAISE Dataset [25], where each image set consists of 9 images. Figure 4 illustrates an example of an input image set. The generation procedure was the same as that in Section IV.C), where v i ∈ {−1, 0, +1}, and c i ∈ {(0.9, 1.0, 1.1), (1.0, 1.0, 1.0), (1.1, 1.0, 0.9)}. Each generated image I i was independently fed into the trained network and decomposed into gray-shadingŜ i , reflectancê R i , and illumination colorĉ i . To evaluate the reconstruction consistency, we used the peak signal-to-noise ratio (PSNR), mean squared error (MSE), and structural dissimilarity (DSSIM) between input image I i and reconstructed imageÎ i . Here, a larger PSNR value means higher similarity between two images. For MSE and DSSIM, a smaller value means higher similarity. To evaluate the reflectance consistency in terms of both brightness and color, we also used the PSNR, MSE, and DSSIM between a reference reflectance and an estimated reflectanceR i of an image under another illumination condition. Here,R 5 was used as the reference reflectance.
The proposed network was compared with the following methods: •Bell's decomposition method [8], •Unsupervised Learning for Intrinsic Image Decomposition from a Single Image (USI3D) [16], where Bell's method is an optimization based method and USI3D is a state-of-the-art DNN-based method. We used the authors' original implementations of the two methods, which are available on GitHub.
Our network was trained with 100 epochs by using 3620 raw images from the HDR+ burst photography dataset [26]. For data augmentation, we resized each original input image with a random scaling factor in the range of [0.6, 1.0] for every epoch. After the resizing, we randomly cropped the resized image to an image patch with a size of 256 × 256 pixels and flipped the patch horizontally with a probability of 0.5. In addition to the augmentation, we generated two images by applying the procedure in Section IV.C) to each augmented image in order to simulate various illumination-brightness and -color conditions. Loss was calculated by using the two generated images in accordance with Eq. (4) to Eq. (9). Here, the Adam optimizer [27] was utilized for optimization, where the parameters in Adam were set as α = 0.001, β 1 = 0.9, and β 2 = 0.999.
He's method [28] was used for initializing the network. Figure 5 shows an example of decomposition with our network. From Fig. 5, we can see that the brightness of estimated reflectanceR was constant, and the estimated shadingŜ had a single color. Figures 6, 7, and 8 are references of the images in Fig.  4 estimated by Bell's method, USI3D, and the proposed network, respectively. From these figures, we can see that the estimated reflectance by the three methods is different. This is because the proposed network takes into account not only conventional reconstruction consistency but also two reflectance consistencies. The two reflectance consistencies require estimated reflectance to satisfy that it is independent from lighting conditions, i.e., brightness and illumination color. As shown in Figs 6 and 7, the conventional decomposition methods produced reflectance that depended on illumination brightness and color owing to the lack of consideration of reflectance consistencies. In contrast, the proposed network produced almost the same reflectance for all nine images that depended on illumination brightness and color. For this reason, the proposed network satisfied the reflectance consistencies in terms of brightness and color, but the conventional methods did not.

2) Experimental results
The reflectance consistencies of the proposed network were also confirmed in a numerical evaluation. Tables 2, 3,  and 4 show the PSNR, MSE, and DSSIM values between R i andR 5 for each i for the three methods, where these values were averaged over the 45 image sets that we used in this simulation. From Table 2, the proposed network provided higher PSNR values (about 30dB) than those of the conventional methods. Hence, the proposed network was  shown to be robust against illumination-brightness/-color changes. This trend was also confirmed from Tables 3 and  4.  Tables 5, 6, and 7 show the PSNR, MSE, and DSSIM values between original image I i and reconstructed imagê I i for each i for the three methods, where these values were averaged over the 45 image sets that we used in this simulation. From Table 5, we confirmed that the proposed network outperformed USI3D in terms of the

B) Simulation using MIT intrinsic images dataset 1) Experimental conditions
To further study features of the proposed network, we conducted simulation experiments using the MIT intrinsic images dataset [5], which is a typical dataset for intrinsic image decomposition. An example of an image "box" in the dataset is shown in Fig. 9. The MIT intrinsic images dataset provides not only original images but also the corresponding groundtruth reflectance and shading for each image. For constructing the dataset, computer graphics techniques were utilized.  9. Groundtruth image "box" in Mit intrinsic image dataset [5] Because groundtruth reflectance and shading components are available for each image, in this experiment, we used the local mean squared error (LMSE) score [5] to evaluate the performance of the proposed network. Here, a smaller LMSE score value means higher similarity between an estimation and the corresponding groundtruth.
Other conditions were the same as those in Section V.A Figures 10 and 11 show estimated reflectance and shading components, respectively, for an image "box" in MIT intrinsic images dataset estimated by Bell's method, USI3D, and the proposed network. As shown in Fig. 10, the reflectance component estimated by the proposed network was significantly different from the groundtruth one shown in Fig. 9. This is because the proposed network was trained with luminance normalization of the reflectance in Eq. (9). When there are no constraints on the luminance of reflectance, intrinsic image decomposition suffers from scale indeterminacy. In this case, when an image I is decomposed as I(x, y) = S(x, y) R(x, y) there is another solution I(x, y) = (aS(x, y)) (a −1 R(x, y)), where a > 0. This scale indeterminacy may be solved by supervised learning using a large high-quality dataset.

2) Expeirmental results
However, it is difficult to prepare the ground truth of reflectance and shading in real scenes. For this reason, we aim at intrinsic image decomposition by unsupervised learning, and the proposed network was trained with the luminance normalization to remove the scale indeterminacy. By using the normalization, the proposed network estimated shading components more accurately than the conventional methods, as shown in Fig. 11. This is also confirmed by the fact that the proposed network provided LMSE scores as well as the conventional methods, as in Table 8.

VI. CONCLUSION
In this paper, we proposed a novel intrinsic image decomposition network considering reflectance consistency. In the proposed network, reconstruction consistency, reflectance consistency (brightness), and reflectance consistency (color) are considered by using a color-illuminant model and training the network with losses calculated from images taken under various illumination conditions. In addition, the proposed network can be trained in a self-supervised manner because various illumination conditions can easily be simulated. Experimental results show that our network can decompose images into reflectance and shading components while maintaining the reflectance consistencies in terms of both illumination-brightness and -colors.
Since the proposed network can produce robust reflectance against changes of illumination conditions, the proposed network will contribute to varous color image processing such as image enhancement while preserving object color and more realistic white balance adjustment.