{'fraction_non_alphanumeric': 0.023277392286590287, 'fraction_numerical': 0.006737130869775698, 'mean_word_length': 4.7715948777648425, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
KinD-LCE Curve Estimation And Retinex Fusion On Low-Light Image

The problems of low light image noise and chromatic aberration is a challenging problem for tasks such as object detection, semantic segmentation, instance segmentation, etc. In this paper, we propose the algorithm for low illumination enhancement. KinD-LCE uses the light curve estimation module in the network structure to enhance the illumination map in the Retinex decomposed image, which improves the image brightness; we proposed the illumination map and reflection map fusion module to restore the restored image details and reduce the detail loss. Finally, we included a total variation loss function to eliminate noise. Our method uses the GladNet dataset as the training set, and the LOL dataset as the test set and is validated using ExDark as the dataset for downstream tasks. Extensive Experiments on the benchmarks demonstrate the advantages of our method and are close to the state-of-the-art results, which achieve a PSNR of 19.7216 and SSIM of 0.8213 in terms of metrics.


Introduction
Due to the widespread use of deep learning, neural network algorithms have achieved great popularity in computer vision, and these algorithms are based on a data-driven approach to image processing and recognition. When taking photos are usually affected by the influence of the surrounding environment, such as backlighting, blurring, distortion, etc. According to the algorithms in computer vision, they are trained and predicted based on normal images, but they cannot achieve the same recognition effect as normal images for low-light images, which affects the image quality as well as industrial production efficiency. Many researchers have proposed different approaches for low-light enhancement to address the impact of low-light images on computer vision applications [1,2,3,4,5,6]. Currently, one of the mainstream schemes for designing low-light enhancement networks is based on the Retinex theory [7]. In addition, for autonomous driving technology, nighttime driving becomes a very challenging problem. To solve the problem of night driving, we need to enhance the image and then use semantic segmentation to capture the information in the image, so deploying low-light enhancement algorithms on IoT devices at the edge device is also one of the problems that need to be solved now.
Recently, many researchers have proposed algorithms to enhance the quality of low-light images. The most widely used algorithm in the early research work was histogram equalization [8]. In recent years several works [1,9,2,4,5,6,10,11] have shown that Retinex theory [7] is more effective in restoring images in low light enhancement and is widely used in image processing tasks. However, Retinex theory assumes that the illumination map is smooth, so blurring can occur at the edges leading to loss of edge details, and noise increase. Therefore, using Retinex theory requires noise suppression, recovery of smooth and non-smooth parts of the illumination map, and suppression of the effect on edge details.
In the process of extracting features from low-light images, some researchers have proposed the use of a U-shaped network structure for the compensation of features to avoid losing spatial information in the upsampling and downsampling process, which makes it possible to keep more feature information in the process of decomposing images. Although the U-shaped structure network has good generality and allows the extraction of a variety of features, the extraction will encode the feature from the image, resulting in the introduction of redundant information and additional noise, which leads to the enhancement of noise after merging.
Based on the problems described above, we designed the Retinex Fusion module to fuse the features of the reflectance map with the illumination map to solve the problem of blurring edges in the illumination map, followed by the light curve estimation module to adjust the image brightness distribution by the trainable parameter α to reduce the noise generated during upsampling and downsampling as well as to reduce the edge detail loss, and finally by using the total variation in the restoration network loss function to suppress the generation of noise.
The main contributions of this paper are as follows.
• The feature information contained in the reflectance map of the image is fused with the illumination map of the image to optimize the image quality, and the image brightness is enhanced by using the reflectance map to calculate the illumination coefficients.
• The brightness of the image is improved with the Light Curve Estimation(LCE) module, which brings out more detail in the image.
• The total variation loss function is included in the restoration network to further eliminate the noise, reduce the parts with large variation and smooth the image edges.

Related Works
With the widespread application in computer vision, working with low-light images has become particularly important.
In this section, we analyze our previous work and propose our solution.

Conventional Methods
The task of low illumination enhancement is mainly to increase the brightness of the image with the aim of improving the recognition accuracy of computer vision tasks such as image classification, object detection, and semantic segmentation and reducing the effect of environmental factors on recognition accuracy. Histogram equalization [8] is the most widely applied low light enhancement algorithm in early work, by which the histogram of the output image is constrained to enhance the contrast of the image, the noise is also amplified which is noise on the image.
Recently, inspired by Photoshop-like image editing software, some researchers [12] have tried to design a curve-based adjustment method that can adaptively adjust the light curve of an image. A low-light image is automatically mapped to a light curve, and the brightness of the image is adjusted according to the light curve. There are three goals for such curve: 1) The pixel values of the image being enhanced should be normalized to [0, 1] avoiding gradient explosion due to large loss values.
2) The curve should be monotonic to facilitate maintaining the variance between adjacent pixels. 3) Back propagation to compute the gradient should be as derivable as possible. For the method based on curve estimation, the enhancement of the original image can leads to the destruction of the features of the original image and affects the structure of the original image.

Mainstream Methods
In recent works, Retinex theory is widely used in low light enhancement. Wei et al [7] took a set of images from the real world to construct a dataset of low illumination and normal images and proposed a structure-aware image constraint on the total amount of variation, which reduces the impact on the image structure by reducing the total amount of variation at locations with large gradients. However, they found that the decomposition of the input image generated excess noise and lost some features, resulting in blurred images when fusing features.
Meanwhile, several recent studies have focused on using unsupervised, zero-shot, and few-shot learning methods to achieve low light enhancement. One of the most recent studies is the Zero-DCE network proposed by Guo et al [12]. which can be trained without using pairs of training data while using a non-referenced loss function to train the model. Zhang et al [1] proposed a multi-stage low light enhancement model designed based on the Retinex theory. The decomposition and fusion of the Retinex method is based on a neural network approach and is divided into three main stages. First, the illumination and reflection maps are segmented by Encoder-Decoder with U-shaped structure, then the reflection map is restored by the restoration network and the illumination map is enhanced by the enhancement network, and finally the enhanced and restored images are fused. Based on the problems of the mentioned Retinex theory [7]. Blurred edges lead to problems such as loss of edge details and increase in noise, and the KinD-Plus [13] network is also designed based on Retinex theory, so it will have the same problems.

Image Fusion
Image Fusion is used by most neural networks to augment the features extracted by the backbone network. The features extracted at different scales or categories are usually fused in the middle of the neural network to make the extracted features more effective. The neural network causes loss of information during downsampling, making the amount of information insufficient after upsampling resulting in loss of details. In the process of low-light enhancement, methods for image fusion and feature fusion can compensate for defects such as loss of details due to downsampling and enhance the image brightness while preserving the original image details. Ying et al [14,15] argue that there are many similarities between cameras and the human eye in that people can see high dynamic range images with clear details everywhere, while the images captured by cameras are not guaranteed to complete detail retention over the global range, the reason for this is that the brain fuses information from different brightness ranges after the eye has acquired the image, resulting in a high dynamic range image with complete detail retention. So based on this conjecture, they proposed a method to obtain high dynamic range images by first transforming the illumination of the low-light images and then fusing the images of different illumination.

Methodology
In this section, we propose a solution based on the above mentioned problems and optimize the network structure accordingly. For image quality use an illuminance map and reflectance map fusion and calculate the illumination coefficient based on the reflectance map to suppress noise generation. The total variation loss function is used to denoise; the Light Curve Estimation module reduces the effect of denoising on brightness.

Network Architecture
In this paper, TV loss [16], Light Curve estimation module and Retinex Fusion (RF) module are applied to KinD-Plus [1] network to design KinD-LCE network. We reduce the noise in the output image by using the total variation loss function to attach a normal term to the model, which acts as a constraint on the noise by matching other loss functions. The Light Curve Estimation module is to adjust the brightness of the illumination map, mapping different pixels to a continuous differentiable numerical space and adjusting the light curve using trainable parameters in back-propagation to suppress the chessboard effect and improve the image quality. The Retinex Fusion module is to fuse the reflection map and the illumination map in the illumination enhancement stage to compensate for the lost features. The network structure of KinD-LCE is represented in Figure 1.
In Figure 1, the Retinex image decomposition network structure is shown in Retinex Decomposition and Image Fusion (RDIF) on the left side of the figure. The model is divided into three parts training the low illumination image input low and the normal image input normal into the Decomposition module F decom (·) to get two sets of illumination maps (I low , I normal ) and reflection maps (R low , R normal ).
Passing I low , R low to Restoration module F restore (·) for restoring R low , which is used to generate the enhanced image and also acts as a noise reduction.
I low , I normal , R low , R normal is passed to the RDIF module F rdif (·) to restore I low , and x rf is obtained by fusing I low , I normal , R low , R normal with the Retinex Fusion module, and x rf is fed the illumination module F illum (·) to enhance I out , where F illum (·) uses an image enhancement method based on Light Curve Estimation, which improves the problem of noise and loss of edge detail due to the use of upsampling and downsampling.
x rf = F rdif (I low , I normal , R low , R normal ) (4) Finally, based on Retinex theory, the illumination map I out and reflection map R out are fused at the pixel level to obtain the enhanced image, denoted as P r = R out ⊗ I out (6)

TV loss
TV loss is a loss term that measures the total amount of change in an image. The total amount of variation of a noise-contaminated image is larger than the total amount of variation of a noise-free image [16], we can achieve the purpose of denoising by minimizing the noise. The problem is solved by reducing the TV loss by calculating the difference of neighbouring pixel values in the image. In this paper, TV loss mainly deals with the noise generated in the process of enhancing the network and reducing the noise generated in the process of fusing the illumination and reflection maps.TV loss is defined as follows.
In low-light images, there is a lot of hidden noise in the image, and upsampling and downsampling through convolutional neural networks can lead to losing some information about the image as well as introducing some noise. TV loss calculates the total change of the image and is able to preserve the edges of the image; meanwhile, Gaussian noise and Poisson noise are prevalent in the image, and Poisson noise is a kind of noise associated with illumination, based on the ability to preserve the edges The Poisson noise is an illumination-related noise and based on the property of preserving edges, the Gaussian noise and Poisson noise can be eliminated by using TV loss. The total loss function used by the method in this paper in the Illumination module is expressed as Where L grad denotes the gradient loss, L mse denotes the mean square loss, I p denotes the normally illuminated image, I denotes the adjusted image, which is the image in the process of mapping from the low illuminated image to the normal image, β denotes the weight of the loss of L grad , and the parameter β=0.01 is set in this paper.

Retinex Fusion Module
The Retinex Fusion module is divided into two parts to process low illumination images, local illumination coefficients t local are multiplied with the unit matrix E to ignore spatial features and improve the impact of illumination coefficients, and global illumination coefficients t global are multiplied with R low to focus on global detail features,which work together to make the neural network focus on more image information and take into account local features of the image. The module for image illumination attributes adjustment, which enables the edge illumination information of R low and I low at the time of combination to be complemented. It restores unsmooth areas of an image to preserve edge detail.
The global illumination matrix S global is first computed using R low , the two illumination maps I low and I normal are compared and then averaged to obtain the original illumination coefficients t global , and then the original illumination coefficients t global are multiplied by R low to recover the missing edge detail in I low due to incomplete decomposition in Decomposition, S global is computed as follows.
where N and M denote the height and width of I low .Secondly, the local illumination matrix S local is calculated using Eq. (13) , and the illumination map of the cropped low light image (I lowcrop ) and the illumination map of the normal image (I normalcrop ) are compared and then averaged to obtain the local illumination coefficient t local , and then multiplied with the unit matrix E of the same size as I lowcrop and the local illumination coefficient t local . To cope with where H and W denote the height and width of I lowcrop . After the above calculation, I lowcrop and S local and I low and S global are concatted to obtain I f local and I f global respectively, and finally I f local and I f global are concatted to obtain x rf and input to the light adjustment network module.
x rf = Concat (I f local , I f global ) The information from the reflectance map is introduced in the Retinex image fusion module to compensate for the loss of illumination map edge information due to incomplete decomposition of the decomposition network, allowing the unsmooth areas in the image to be correctly adjusted and improving the quality of the output image. The method is shown in Figure 2.

Light Curve Estimation with Illumination Map
In the study of Retinex theory, it is found that the reflectance map contains information such as feature details and edges of the original image, and the illumination map contains information such as contrast and brightness of the image. Therefore, further optimization of the decomposed reflection map and illumination map can achieve the effect of image enhancement. Inspired by the image editing software, the light curve is introduced to make the illumination curve of the illuminance map smoother and more natural, and each pixel point is adaptively enhanced, compared to Zero-DCE [12] which calculates the light curve for the original image and its adjustment has an impact on the original image colour, using Retinex theory to decompose the illumination and detail features of the image and then adjust the illumination using the light curve Reduce the effect of colour shift.
The light curve adjustment module is different from the traditional method of Encoder-Decoder network, which needs to downsample and then upsample the image, but in the process of upsampling, it is easy to produce a "checkerboard artifacts", which makes the colour of a certain part of the image will be darker than other parts of the image. The above situation occurs when the ratio of the convolution kernel size to the convolution step size is not an integer. In our work, the light curve adjustment module is used as an illumination map enhancement network to enhance the illumination map using the light curve to make the image brightness smoother and eliminate the effect of checkerboard artifacts. A schematic of the light curve estimation method is shown in Figure 3. At the beginning of the Illumination phase, the image processed by the RDIF module is used as input and finally decomposed into 8 light estimation coefficient matrices α i (i = 1, 2, · · · , 8) by four 3 × 3 convolutions and a concat operation on them respectively, which partially maps the light estimation coefficient matrices into a continuous space.
α i is a trainable parameter, the range belongs to [−1, 1], the illumination map processed by RDIF module is input to Illumination map enhancement module (Illumination), after four convolutions of 3 × 3, the feature map obtained from each layer convolution is mapped separately, and finally the feature map with the number of channels is x 7 , which represents the illumination estimation ratio for each pixel point on the image.
x 5 = Conv 3×3 (Concat (x 3 , x 4 )) , x 6 = Conv 3×3 (Concat (x 2 , x 5 )) (20) Depending on the adjustment scale factor, is decomposed into 8, which are each entered the Light Curve Estimation module to adjust the light curve, using the formula expressed as I out = sigmoid(Concat(I 1 , . . . , Where x rf (j) represents the j pixels of x rf , I out represents the adjusted image, and I i represents the image adjusted by the i th α i .

Implemental Details
The current method of building is to use LOL dataset [9] and GladNet dataset [10] implemented, which is a widely used dataset for low-light images. We use the Wang GladNet dataset proposed by et al. [10] The GladNet dataset contains 5000 pairs of 8-bit RGB image pairs of low-light/normal images as training data, and the 5000 pairs are divided into 3500 pairs as training set, 1000 pairs as validation set, and 500 pairs as test set. Using the LOL dataset [9] as the comparison set for the test set to validate the enhancement effect. LOL dataset [9] contains 500 pairs of low light/normal image pairs. All the images mentioned above were obtained from real scenes. The GladNet dataset [10] has variable image sizes with various scales; all the images in the LOL dataset are 600 × 400 × 3 images. For the downstream task, we use semantic segmentation as the downstream task, using the currently commonly used ExDark [17] semantic segmentation dataset for blackout scenes. It contains images of black night scenes taken from 13 regions, containing more than 10 categories of scenes such as buildings, streets, and cars.

Ablation Study
In this section, we perform ablation experiments using the GladNet dataset and the LOL dataset to gradually verify the validity of the improvement points proposed above, and we subsequently validate the model based on the validation sets of the GladNet and LOL datasets.

Ablation for Light Curve Estimation
In studying the KinD-Plus [13] network, we found that the illumination map enhancement module did not take into account the colour and light balance, so we chose to design its illumination map enhancement network using light curve estimation and proposed our KinD-LCE network. After our experiments, our light curve estimation module improves in PSNR metrics for the same image input. The rendering is shown in Figure 4 and the metrics table is shown in Table 1.

Ablation for Retinex Fusion
We fuse the reflectance map with the illumination map, adjust the coefficients of illumination map enhancement by the pixel values of the reflectance map, and compensate for the loss of edge details and other problems generated by the original image decomposition by fusing the reflectance and illumination maps.
After our experiments, the image fusion module improves the PSNR and SSIM metrics over the original KinD-Plus [1] network by 16.66% and 3.52%. The results are shown in Figure 5, and the metrics table is shown in Table 2.
Ablation for TV loss We add TV loss [16] to the Illumination module to calculate the total variation of the illumination map to denoise the illumination map and reduce the noise due to the decomposition network, the metrics table is shown in Table 3.

Comparison of KinD-LCE and Other Methods
After our experimental comparison and cross-validation using different methods, we can see that we can achieve relatively high metrics by adding the three modules proposed in this paper. The experimental results are shown in Table  4.
The method in this paper is compared with the best methods at this stage, and the enhancement effect is evaluated by PSNR, SSIM metrics, and the evaluation is tested using 15 validation sets of LOL dataset. The images are shown in Figure 6, And the metrics is shown in Table 5.

Comparison in Downstream Task
For the effectiveness of the method of this paper, applying the method of this paper to downstream tasks (e.g., object detection, semantic segmentation, etc.) for traditional dark images without image processing can lead to a significant degradation in its detection effectiveness. So, using the method of this paper the dark image is processed and then applied to the downstream task whether the object in the image is detected or not.    Fig. (c), for the enhancement at details the effect is more detailed than that of Fig. (b), and for the noise aspect, there is less noise compared to Fig. (b), and the brightness of the image is higher.  We experimentally compare images using different low-light enhancement networks to verify the effectiveness of different low-light enhancement networks for downstream tasks to highlight the effectiveness of the algorithms in this paper. This downstream task validation experiment uses HRNet [21] network for semantic segmentation, using low-light images in traveling scenes for enhancement, and we use the ExDark [17] dataset as the dataset for this experiment while using the Cityscape dataset-based [22] The trained model is validated against the driving scene. The following are the experimental comparison plots, shown in Figure 7.

Conclusion
In this paper, the proposed KinD-LCE network integrates light curve evaluation, light map, and reflectance map fusion, and including TV loss function is an improvement proposed for low light enhancement tasks. The combination of light curve estimation is adjusted for the light map, and the features of the reflection map are fused into the light map to improve the brightness of the image enhancement as well as to reduce the introduced noise. The approach in this paper compensates for the KinD-Plus [1] enhancement process by introducing noise and enhancements to achieve the image illumination enhancement effect. The effectiveness of this paper's method is demonstrated through ablation experiments for different methods and experiments applied to downstream tasks. However, for other metrics, such as MAE and MSE metrics performance still has shortcomings, because the image quality will still be affected during the image processing process.