{'fraction_non_alphanumeric': 0.03838359273670558, 'fraction_numerical': 0.009484435797665369, 'mean_word_length': 4.670650425189612, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Local Underexposed Photo Enhancement

Inspired by the ability of deep generative models to generate highly realistic images, much recent work has made progress in enhancing underexposed images globally. However, the local image enhancement approach has not been explored, although they are requisite in the real-world scenario, e.g., fixing local underexposure. In this work, we define a new task setting for underexposed image enhancement where users are able to control which region to be enlightened with an input mask. As indicated by the mask, an image can be divided into three areas, including Masked Area A, Transition Area B, and Unmasked Area C. As a result, Area A should be enlightened to the desired lighting, and there shall be a smooth transition (Area B) from the enlightened area (Area A) to the unchanged region (Area C). To finish this task, we propose two methods: Concatenate the mask as additional channels (MConcat), Mask-based Normlization (MNorm). While MConcat simply append the mask channels to the input images, MNorm can dynamically enhance the spatial-varying pixels, guaranteeing the enhanced images are consistent with the requirement indicated by the input mask. Moreover, MConcat serves as a play-and-plug module, and can be incorporated with existing networks, which globally enhance images, to achieve the local enhancement. And the overall network can be trained with three kinds of loss functions in Area A, Area B, and Area C, which are unified for various model structures. We perform extensive experiments on public datasets with various parametric approaches for low-light enhancement, %the Convolutional-Neutral-Network-based model and Transformer-based model, demonstrating the effectiveness of our methods.


Introduction
With the popularization of readily-available cameras on various devices, photo sharing has become very common nowadays. However, these captured photos can be underexposed due to low-and back-lighting conditions, which are unsuitable for propagation via the internet. Therefore, the technique of underexposed photo enhancement (UPE) has been proposed in recent years, which refers to the task of enhancing an underexposed image to a normally-exposed one. The traditional UPE techniques, which are not learning-based, have been integrated into commercial softwares, e.g., Adobe Photoshop. However, using these softwares require professional experiences and human efforts. With the development of deep learning, many learning-based UPE approaches have been proposed to automatically complete the enhancement. And all current learning-based UPE methods focus on global image enhancement and can not implement the local modification. Nevertheless, there is also need of local image enhancement in the real-world scenario, taking portraits as an example, for images taken in underexposed environments, or in complex lighting cases, the foreground and the background should be enhanced differently.
In this work, we introduce a new task setting called "MaskUPE", where users can complete the local enhancement for an underexposed image with an input binary mask map. The mask can indicate which area to be enhanced, and there shall be a smooth transition from the enlightened area to the unenlightened region. There are no existing strategies that can solve the MaskUPE task, since their targets are global editing. To this end, we aim to design a novel module that can serve as a play-and-plug module and be combined with the existing networks, achieving the local enhancement.
A question is: given a global enhancement network g(), how can we encode the information of mask without significant modification to its backbone, and meanwhile keeping the corresponding enhancement effects in the target region? Here comes two solutions The first solution is simultaneously taking the input image and the input mask map via the channel concatenation. We name it as MConcat Formally, For an input image I ∈ R n×m×3 and a mask map M ∈ R n×m×k (k = 2 since the mask map is formulated to distinguish the areas to be enhanced or not), we can concatenate I and M asĨ ∈ R n×m×(3+k) and then fedĨ into g for enhancement.
On the other hand, There is another way, named MNormthe mask information can be combine into hierarchical levels of the backbone in g(). The mask information is encoded to predict the parameters of spatially-adaptive normalization operations in g().
Both methods have several advantages: 1) they can be integrated into existing global enhancement networks g() with minor modifications.
2) By incorporating the mask information into the network with MConcat and MNorn, the task of MaskUPE can be implemented with satisfying results. Moreover, if the MaskUPE is combined with existing strategies that can control the enhancement degree (e.g., the Retinex-based models like Deep-UPE (Wang et al. 2019), LIME (Guo, Li, and Ling 2017), or the curve-based models like (Guo et al. 2020)), the corresponding framework can simultaneously decide spatialvarying enhancement requirements since both local editing and degree control are implemented. This further improves the flexibility for software users. The experiments on public datasets demonstrate that our methods are suitable for current approaches with different parametric outputs and various structures (either convolutional or transformer structures).
The critical challenge of training the framework is the implementation of transition between the enhanced and the unchanged area. To this, we propose a training scheme and first divide the input image into three different areas, i.e, area A, B, and C. Area A is the area to enhance; Area B is the area for transition; Area C is the area that is not required to be enlightened. Three different loss functions can be applied to these regions to achieve the corresponding purposes. For area A, we keep the loss applied on the original model g since this area should be enlightened to the normal light. In area B, we employ a novel gradient-smooth loss to minimize the second-order gradient of the illumination component. In area C, we can just apply a reconstruction loss between the predicted image and input image, since C is not required to be changed. Such a training scheme is also general for existing UPE approaches with various structures. To demonstrate the effectiveness of MConcat, MNorm and the corresponding training scheme, we conduct extensive experiments on two public datasets with the backbone of g() as Convolutional Neutral Network (CNN), e.g., DeepUPE (Wang et al. 2019), or Transformer, e.g., STAR (Zhang et al. 2021).
Proved by the experiments, for an underexposed image, the framework with MConcat and MNorm can enlighten the specified local area with different degrees.
In summary, our contribution is four-fold.
1. We introduce a novel task setting MaskUPE, which is to enhance an input image given a binary mask, satisfying the requirement of local image enhancement. And this work is the first attempt to design regional enhancement approaches with deep learning. 2. To complete MaskUPE, we design two methods MConcat and MNorm. The can be integrated into existing networks that are originally designed for global editing, to achieve the local enhancement effect. 3. We propose a novel training scheme, applying different loss components in spatial-varying areas. In particular, a novel gradient-smooth loss is proposed to achieve the smooth transition effect. 4. Extensive experiments are conducted on public datasets with representative model structures, showing that the framework with MConcat, MNorm and the corresponding training scheme can meet the requirements of MaskUPE.

Related work
To the best of our knowledge, there is no method working uder Local Underexposed Photo Enhancement Operation. The most relevant work is ABPN (Lei et al. 2022), which proposes an adaptive blend pyramid network for local photo retouching. Their method is verified under two tasks: cloth retouching and face retouching. The most significant difference is that, our method supports explicit mask input, while ABPN doesn't.
In this section, We briefly introduce the recent CNN-based and Transformer-based "Global Underexposed Photo Enhancement models", and then introduces some UPE methods including spatially-varying operations. However, note that their learning objectives and evaluation metrics are still focusing on the enhanced global result, which is different from our local learning objectives.

Global Underexposed Photo Enhancement
Underexposed photo enhancement is a vital while challenging task, since the underexposed areas are usually imperceptible, and the enhancement process is highly nonlinear. With the massive success of deep learning, a significant trend for solving this task is the CNN-based image enhancement method, and several approaches have achieved promising results. Another new trend is the Transformer-based models, starting from Vision Transformer (Kolesnikov et al. 2021), which brings Transformers (Vaswani et al. 2017) to the field of Computer Vision.
CNN models mostly use an encoder, followed by an output tail. In general, the output tail could be divided into several categories: • Retinex-theory-based methods. These models use Retinex Theory to predict the illumination map, hence generating the reflectance image. Examples are LIME (Guo, Li, and Ling 2017) and DeepUPE (Wang et al. 2019). • Parameterized models. These models predict parameters of predefined filters, e.g., DeepLPF (Moran et al. 2020), or enhancement curves, e.g., ZeroDCE (Guo et al. 2020). • Encoder-Decoder-based models. These kinds of models are prevalent via predicting the pixel value directly, e.g., SID (Chen et al. 2019), MIRNet (Zamir et al. 2020). • 3D-Look-up-Table-based approaches. Such strategies predict the index to retrieve the pixel value from 3D look-up table, like (Zeng et al. 2020)and .
Another popular direction is Transformer-based models. ViT (Kolesnikov et al. 2021) bridges the gap of Transformer between Natural Language Processing (NLP) and Computer Vision (CV). Inspired by ViT, much Transformer-based work towards low-level CV tasks appears. With over 114 Million parameters and 33 GFLOPS, IPT (Chen et al. 2021) model introduces contrastive learning to ViT, adapting to different image processing tasks, e.g., super-resolution, denoising, deraining. After that, STAR (Zhang et al. 2021) reduces model complexity to a large margin, and still improves the performance, benefiting from its Long-short Range Transformer Module.
All these models mentioned above enhance the global area, and the corresponding performances are evaluated globally. Thus, they are not capable of performing local enhancement.

Underexposed Enhancement with Spatially-varying Operations.
Tian and his fellow researcher (Tian and Cohen 2017) proposed a method using global and local enhancement for color enhancement. The algorithm includes both both global contrast adaptive enhancement, and a hue-preserving local contrast adaptive enhancement module Finally, a contrastbrightness-based fusion algorithm obtains the final result, which represents a trade-off between global contrast and local contrast. However, such a method can not enhance the local area according to the user-given inputs.
Moran et al. (Moran et al. 2020) proposed Deep Local Parametric Filters to learn spatially local filters of three different types (Elliptical Filter, Graduated Filter, Polynomial Filter).
CNNs are introduced to regress the parameters of these spatially localized filters. Nevertheless, these local filters are still designed for global enhancement.
DeepExposure (Yu et al. 2018) is an algorithm for learning local exposures with deep reinforcement adversarial learning. DeepExposure leverages image segmentation to segment the input image into sub-images based on the low-level features. Then the algorithm learns to enhance with different local exposure increments for each sub-image, i.e., a local exposure for each sub-image, and fuse the sub-images to get the final enhanced image. However, this approach can not enhance different image regions according to the user-given inputs.

Method Tasking Setting
We introduce a new task setting, namely MaskUPE: the input is the image I ∈ R n×m×3 , and a binary mask M ∈ R n×m×2 , which could be user-generated. The task is to seek a learned enhancement map f θ (I), that enhances the image in the masked area, and transits naturally to the dark area. Note that we denote the area to be enhanced with the mask value of 1 and the area to be kept with 0.
Under the MaskUPE setting, an input image can be divided into Masked Area A (encoded by the first channel of Mask M ), Transition Area B (encoded by the second channel of Mask M ), and Unmasked Area C. Their details are explained as the following. We select (some of) the normalization layers in the encoder part of g(), and replace them with our MNorm Module. Inside the module, mask information is encoded via the convolution layer for spatially-varying enhancement.

Masked Area
A is the area to be enlightened, which is specified by the first channel of M . Formally, A = {(i, j)|M i,j,0 = 1} 1 , for 1 ≤ i ≤ m, 1 ≤ j ≤ n. 2. Transition Area B is the area that are supposed to achieve smooth transition from Area A to Area C, specified by the second channel of M . Formally, B = {(i, j)|M i,j,1 = 1}. 3. Unmasked Area C is the area to keep dark. It is the area that is not covered by any channel of the mask. Formally, Fig. 2 further illustrates the task setting of MaskUPE. Under this setting, given an existing global image enhancement model g() with the parameters of θ, we want a new model f (), whose parametersθ are modified from θ with minimal efforts, to achieve local image enhancement.θ learns the transformation fθ such that the learning objective L(Ŷ , Y ) is minimized, whereŶ = fθ(I). We designate the learning objective L such that Area A is to be enhanced effectively, Area B transits smoothly, and Area C is to be kept dark. Details of loss functions will be described in the section for loss function.

Architecture Design
We introduce two methods to convert a global photo enhancement network g θ to fθ, which can achieve local enhancement effects.

Method 1 (MConcat)
A simple solution is to view the mask input as input image's extra channels, and concatenate the mask input with the RGB channel of the input image. In other words, the new model fθ has the same structure as g θ , except that we shall modify the model for it to accept a 5 channel input.
Method 2 (MNorm) MNorm simply replaces (some of) the normalization layers in g θ by mask-encoded normalization layers. Unconditional normalization methods, e.g., Batch 1 Here, we refer the first channel of image as channel 0. Normalization (BatchNorm) (Ioffe and Szegedy 2015), Instance Normalization (InstaceNorm) (Ulyanov, Vedaldi, and Lempitsky 2017), Layer Normalization (LayerNorm) (Ba, Kiros, and Hinton 2016) are not capable of encoding mask information. In contrast, we propose to modulate activations in normalization layers through a spatially adaptive and learning-based transformation. The architecture of MNorm is illustrated in detail, as shown in Fig. 3. To be specific, a MNorm module first projects the mask into an embedding space and predicts the modulation parameters γ, β via convolution. Then γ and β are multiplied and added to the normalized activation element-wise.
Formally speaking, in the context of MaskUPE, a MNorm module is as follows: for k-th convolution layer in a network g θ with batch size N with height H k and width W k , let h k be the activation of k-th layer, C k be the number of channels in layer k. The mask-encoded activation value is: where h i n,c,y,x (n ∈ N, c ∈ C k , y ∈ H k , x ∈ W k ) is the activation before normalization, µ k c , σ k c are the mean and standard deviation of the activation in channel c. Here, γ i c,y,x (M ) and β k c,y,x (M ) are learning-based parameters converting the information of Mask M to the scaling and bias values at the location (c, y, x). γ i c,y,x (M ) and β k c,y,x (M ) are implemented with a simple two-layer CNN.

Loss Function Design
, we employ different loss functions in three areas, i.e., Masked Area A, Transition Area B, and Unmasked Area C. The total loss is designed to be the weighted combination of three loss components L A , L B , and L C , as: (2) where α, β, and γ are loss weights, r a , r b , and r c are the coverage area (measured in percentage) of the Area A, B, and C, respectively. We show the details of loss components applied on different areas in the following paragraphs, by taking DeepUPE as the example of g θ . Transition area B. The transition Area B, between Masked Area A and Unmasked Area C, plays a vital role in increasing the consistency of the different regions in the whole image. In this area, we should ensure a smooth transition from darkness to brightness. Different from the smooth prior adopted in (Rother et al. 2011;Wang et al. 2019), which smooth the first-order gradient of the illumination map, the proposed Gradient-Smooth loss is working to smooth the second-order gradient of the illumination map. 2 In other words, we apply the smooth loss prior to the gradient of the illumination map L(·). Specifically, for each pixel p in Area B, we can define the smooth loss L i B as

Masked area
where s = 1.2 controls the sensitivity to the second-order image gradient; and = 0.0001 is a small constant that prevents division by zero.
Unmasked area C. For pixels in Area C, we want them to be consistent with the original dark image I. In this case, we use the reconstruction error between the dark image I and the predicted imageŶ . Taking L 2 norm as an example, the reconstruction loss can be written as

Interactive and Flexible Photo Enhancement
With MConcat and MNorm, the enhancement process could be more interactive and flexible.
Flexible mask shape. In general, arbitrary designs of the mask are feasible. The mask could be in regular shape, i.e., semantic-agnostic, like rectangular, circular; or it could be semantic-aware, and can even be user-generated strokes. For example, a mask for the human face enables face retouching. Meanwhile, we can use morphological Erosion or Dilation (Haralick, Sternberg, and Zhuang 1987) to the mask of Area A to obtain Area B without human efforts. These flexible designs enable the users to enhance according to their own preferences.
For simplicity, we implement circular masks for Area A and B in our experiment to directly compare our proposed model with baseline models. Details will be discussed in the experimental section.
Control the degree of enlightening. Underexposed Photo Enhancement is an ill-posed task-some software users may prefer darker images, while others prefer brighter ones. If we combine Mconcat and MNorm with the model g θ that adopts curve-based or Retinex-based models for enhancement, we can control the degree of enlightening by multiplying a coefficient to the predicted curve/illumination map. This makes the enhancement process more interactive, and provides the flexibility for software users to choose their own lighting.

Experiments Experiments setup
Dataset MIT-Adobe-FiveK (Bychkovsky et al. 2011) (FiveK) is used to evaluate our proposed algorithm MConcat and MNorm. We follow the settings of previous methods (Moran et al. 2020;Wang et al. 2019;Kosugi and Yamasaki 2020) to only use the retouched images by Expert C. All images are resized, with 512 pixels in the long edge, and the bit-depth is 8 bit. We select the first 4,500 images for trining, and the remaining 500 images for validation and testing. Underwater Image Enhancement Benchmark (UIEB) is an underwater image enhancement dataset including 950 real-world underwater images, 890 of which have the corresponding reference images. Most of these underwater images suffer underexposed, color casts, and blurring details. We resize with 512 pixels in the long edge during training. We randomly select 800 paired images for training, and the remaining 90 paired images for validation and testing.
Performance metrics For Area A of the enhanced image, we adopt two full-reference metrics to evaluate the image quality: PSNR (measured in dB), and SSIM (Wang et al. 2004) as our evaluation metrics in Area A. High PSNR and High SSIM suggest good image quality.
For Area B of the enhanced image, so far, no proper metric can be used to evaluate the transition effects quantitatively. Here, we performed only a qualitative analysis on the result of Area B.
Backbones To comprehensively show the effectiveness of the proposed methods, we adopt the MConcat and MNorm in both CNN-based and Transformer-based models. For CNNbased models, we choose DeepUPE as the representative one; for Transformer-based models, we choose STAR-DCE (STAR-based Zero-DCE backbone) as the representative one. Note that STAR and DeepUPE use different output tails -DeepUPE is a Retinex-theory-based model, and STAR is a parameterized model. 3 Therefore, we construct four novel models, namely DeepUPE-MConcat, DeepUPE-MNorm, STAR-DCE-MConcat, and STAR-DCE-MNorm. We trained MConcat models for 100 epochs, MNorm models for 300 epochs, for they have larger number of parameters.
Implementation details Three loss component weights α, β, γ are set to 1, 0.001, 1 respectively. As for the implementation of the mask, we use circular masks for experiments. The circle in radius r 1 indicates Area A, and a larger circle in radius r 2 indicates Area B. In the following experiments, r 1 ∼ U (m/3, m/2), r 2 ∼ U (1.1r 1 , 1.3r 1 ), where m denotes the minimum of image height and the width For DeepUPE-MNorm, we replace normalization layers with MNorm Module in the features extraction, and local features extraction part.
For STAR-MNorm, we use STAR-DCE backbone with the Mean Head tokenization Strategy. The network width is 32. We add a MNorm module in the tokenization. We keep the loss components in the original DeepUPE and STAR-DCE model and apply them in Area A of the image.

Experiment Results
We show the enhanced images on Adobe-MIT-FiveK dataset in fig. 4, the enhanced images on UIEB dataset in fig. 5  Control the degree of light. In the DeepUPE model, we could manipulate the illumination map generated by the DeepUPE backbone. In the STAR model, we could manipulate the predicted curve generated by the STAR-backbone. Fig. 6 illustrate some example results of DeepUPE-MConcat model.

Ablation Study
The gradient-smooth loss. The gradient-smooth loss is the key to ensure smooth transitions, which is also an innovation in our proposed training schemes. So we illustrate the importance of gradient-smooth loss by removing the gradientsmooth loss. In other words, in our ablation experiment, the whole image is now divided into two parts: Masked Area A = A and Unmasked Area B =Ā (complement of Area A) specified by mask M ∈ R n×m×1 , and we apply two loss components: Area A loss L A and reconstruction error L B with weight α = 1, β = 1. See Fig. 7 for details. In general, for models trained without gradient-smooth loss, we spot sharp, unnatural edges in enhancement results.

Discussions
In the designing stage of this project, we think that for MConcat, the mask information is merely provided at the pixel level for the input image, so such a concatenation strategy may not be informative for the model to achieve the target of MaskUPE. However, as the result shows, MConcat generates comparative results with MNorm methods from both the qualitative and quantitative metrics. Moreover, due to time limit, we are not able to perform experiments on semantic-aware masks, e.g. portraits. Figure 7: The ablation study for the gradient-smooth loss in STAR-DCE. Left: models without gradient-smooth loss. Right: models with gradient-smooth loss.

Conclusion
In this work, we introduce a new task setting in Underexposed Photo Enhancement, namely MaskUPE, which is to enhance the underexposed image within a given mask. We propose MConcat and MNorm to convert a global image enhancement model to enable local image enhancement by encoding mask information in (some of) the normalization layers of the model. Accordingly, we apply different loss components to different spatial areas, where the key is the novel gradient-smooth loss in Area B to achieve smooth transition effects. Experiments have shown that MConcat and MNorm are effective in the new local enhancement setting.
For the future work, we think there are still space of improvement for MConcat and MNorm, we may explore more effective methods to encode the mask.