{'fraction_non_alphanumeric': 0.03790110060455743, 'fraction_numerical': 0.023949775228646723, 'mean_word_length': 4.275143090760425, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DEMOSAICING OF REAL LOW LIGHTING IMAGES USING CFA 3.0

In CFA 2.0, there are white pixels in a color filter array (CFA) that has proven to help the demosaicing performance for images collected in low light conditions. Here, we evaluate the performance of demosaicing for images collected in low light conditions using an RGBW pattern with 75% white pixels. We term this CFA the CFA 3.0. Using a data set containing 10 images collected in low light conditions, we performed extensive experiments. Both objective and subjective evaluations were used in our experiments.


INTRODUCTION
The standard Bayer pattern [1], also known as color filter array (CFA) 1.0, has been widely used in many commercial cameras. As shown in Figure 1 (a), for each 2x2 block, there are two green, one red, and one blue pixels. In the Mastcam onboard the NASA's Mars rover Curiosity [2]- [5], Bayer pattern has been used for the RGB images. One key reason for using Bayer pattern is to reduce cost. Due to the huge success of the Bayer pattern, another pattern known as RGBW or CFA 2.0 with the aim of improving image quality in low lighting conditions was introduced by researchers at Kodak [6]. For each 4x4 block in a RGBW pattern (  Figure 1 (b)), there are 50% white pixels, 25% green pixels, and 12.5% red and blue pixels.  , there are 12 white pixels, two green pixels, one red and one blue pixels. One key advantage for CFA 3.0 is that some algorithms for CFA 2.0 can be easily modified and adapted to CFA 3.0. In that paper [8], we have also solved the problem of demosaicing for CFA 3.0. Emulated low lighting images were used in our experiments. It was observed that CFA 3.0 achieved a performance somewhere between CFAs 1.0 and 2.0.
It is worth to mention a related paper [7], which further increases the number of white pixels. The CFA is shown in Figure 1 (d). It can be seen that, for each 6x6 block, 32 pixels are white. The remaining pixels form a Bayer pattern in the center. It was claimed that this CFA can work well in low lighting conditions. However, Gaussian noise was used to emulate noise induced in low lighting conditions. As mentioned in [10], the noise generated in low lighting conditions is known as Poisson noise, which is different from Gaussian noise because Poisson noise is amplitude dependent. Poisson noise should be used instead to emulate noise in low lighting conditions.
In our recent paper on the demosaicing of CFA 2.0 (RGBW) [11], we have extensively compared CFA 1.0 and CFA 2.0 using IMAX and Kodak images and observed that CFA 1.0 has better performance. One reason for that is because IMAX and Kodak data sets were not collected in low light conditions and CFA 2.0 was designed for images in low lighting conditions. We then performed another study using emulated low lighting images in [10]. It was observed that CFA 2.0 indeed performed better than CFA 1.0.
However, in another recent paper [12], we observed that CFA 2.0 still performed better than CFA 1.0 using realistic low lighting images. In this chapter, we summarize our investigation of using CFA 3.0 for some realistic low lighting images. The goal is to see if CFA 3.0 can work well for those realistic low lighting images. Extensive experiments using more than ten methods were carried out. Both objective and subjective evaluations were performed.
Our contributions are as follows. First, we applied and compared CFA 1.0, CFA 2.0, and CFA 3.0 to actual low lighting images. This study has not been done before in the literature. Second, our experiments showed that CFA 2.0 and CFA 3.0 have lower noise than CFA 1.0 in actual low lighting images. This confirmed the observations in [11] where simulated images were used in the experiments. Third, we offered an explanation to some inconsistent findings in our experiments. Basically, the observations and conclusions in [7] are questionable because some real low lighting images containing Poisson noise were used as references in calculating the performance metrics.
In Section 2, we will briefly review the demosaicing algorithms for CFA 3.0. In Section 3, we will summarize the experiments using ten real low lighting images. Section 4 will conclude the chapter with a few remarks.

REVIEW OF DEMOSAICING ALGORITHMS FOR CFA 3.0
Different from the random color patterns in [13], the CFA 3.0 pattern in this paper has fixed patterns. One key advantage is that some approaches for CFA 2.0 can be directly applied with little modifications. As shown in Figure 2, in each 4x4 block, the four R, G, B pixels in the CFA 3.0 raw image are extracted to form a reduced resolution CFA image. A standard demosaicing algorithm can be applied. The missing pan pixels are interpolated to create a full resolution pan image. The subsequent steps will be the same as before. Details can be found in [8].
In [8], the pansharpening approach for CFA 3.0 is shown in Figure 3. Here, the four R, G, B pixels are extracted first and then a demosaicing algorithm for CFA 1.0 is applied. For the pan band, any interpolation algorithms can be applied. Afterwards, any pansharpening algorithms can be used to fuse the pan and the demosaiced reduced resolution image to generate a full resolution color image. In our experiments, we have used Principal Component Analysis (PCA) [14], Smoothing Filter-based Intensity Modulation (SFIM) [15], Modulation Transfer Function Generalized Laplacian Pyramid (MTF-GLP) [16], MTF-GLP with High Pass Modulation (MTF-GLP-HPM) [17], Gram Schmidt (GS) [18], GS Adaptive (GSA) [19], Guided Filter PCA (GFPCA) [20], PRACS [21] and hybrid color mapping (HCM) [22]- [26]. In [8], a hybrid approach combining deep learning and pansharpening [27] was adapted to CFA 3.0 as well, as shown in Figure 4. For the reduced resolution demosaicing step, the Demonet algorithm is used. In the pan band generation step, we also propose to apply Demonet [28]. After those two steps, a pansharpening algorithm is then applied.
We also have two pixel-level fusion algorithms known as F3 [29] and ATMF [30]. Figure 4. A hybrid deep learning and pansharpening approach to demosacing CFA 3.0.

Low Lighting Images
We downloaded a benchmark data set with low light images [31]. The 10 images are shown in Figure 5. In each image, there is a color checker board. The size of the images varies. Some are more than 2000 by 2000 pixels. Some of them are really dark and hence this is a good data set for comparing the different CFA patterns. We believe this data set is quite realistic. In addition to objective performance comparisons, we also carried out subjective evaluations. To visually compare the demosaicing methods, two sections were taken from the first image. See Figure 6. One section contains a tablet of colored squares that can be found in each image and the other section is a radio on the table. The color tablet is an important section because it will show how each demosaicing technique will handle a diverse image in low light conditions. The radio is a dark section of the image that tends to collect a lot of artifacts.

Performance Metrics
In this paper, five performance metrics were used to compare the different methods.
 Peak Signal-to-Noise Ratio (PSNR) [32]: A higher PSNR means better quality. A combined PSNR (CPSNR) is the mean of the PSNRs of the R, G, and B bands.  Structural SIMilarity (SSIM): This metric was defined in [33] to reflect the similarity between two images. The SSIM index is computed on various blocks of an image. The ideal value of SSIM is 1 for perfect prediction.  Human Visual System (HVS) metric [34]: HVS is claimed to resemble human perception.  HVSm (HVS with masking): This metric is similar to HVS except that visual masking effects are taken into account. The inclusion of a block containing contrast masking is the only difference between HVS and HVSm. Details can be found in [35] [36].  In addition to PSNR, SSIM, HVS, and HVSm, we also used CIELAB [36] for assessing demosaicing performance.

Results
There are 14 methods in our study. The baseline and standard methods are similar to those in CFA 2.0. The three best methods used for F3 are Demonet + GSA, PCA, and GFPCA. The ATMF uses those three methods as well as GSA, GLP, GS, and PRACS.
We first demonstrate that the pan interpolation using Demonet is better than that of a conventional method (LDI). From Table 1, we can see that interpolation using Demonet is 3 dBs better than that of LDI, which has been used in the standard approach in our studies for CFA 1.0 and CFA 2.0.
As can be seen in Table 2, the best performing method is the F3 method, which is 3 dBs better than the standard method. Moreover, F3 is 2 dBs better than that of the Demonet+GSA. Surprisingly, GFPCA also has decent performance. The bar charts in Figure 7 reveal that the blue band is worse than red and green bands. This is somewhat hard to understand, as the red and blue pixels have equal numbers. Figure 8 shows that Demonet + GSA is no longer the best performing method in CFA 3.0.
In subjective comparisons shown in Figure 11 and Figure 12, we can see the performance of different methods varies a lot. Consequently, it is easier to observe artifacts in poor performing methods. One can easily compare the pixels inside the red circles in Figure 11 and Figure 12 to see which method is good or bad. In most of the CFA 3.0 approaches, it is easy to see color distortions around the lighter colored boxes and most notably the white box in the lower left corner. There are some artifacts found in the darker panel on the color tablet especially in SFIM and HPM.     Now, we would like to compare the three CFA patterns. Since different algorithms are being used in each CFA, we think an appropriate way to compare the different CFAs is to compare the best against the best. That is, for each CFA, we select the best performing method and compare its results with best performing methods in other CFAs.

Best against the best comparison among the three CFA patterns
We took some metrics related to CFA 1.0 and CFA 2.0 from our paper [12]. In terms of the PSNR, the best performing method for CFA 1.0 is Demonet, which yielded an averaged PSNR of 43.18 dBs. The best performing method for CFA 2.0 is the combination of Demonet and GSA, which yielded an averaged PSNR of 42.36 dBs. The best performing method for CFA 3.0 is F3, which achieved an averaged PSNR of 42.09 dBs.
In short, CFA 1.0 is the best pattern for low light images. For images collected in normal illumination conditions, our earlier papers [11] and [29] also concluded that CFA 1.0 is better than CFA 2.0.

Visual comparisons
From Section 3.4, one observes that CFA 1.0 performed better than CFA 2.0 based on objective metrics. However, this is in direct contradiction with the observations in [10] in which we observed that CFA 2.0 performed better using emulated low lighting images. Here, we will first present the visual results of best against the best methods for CFA 1.0, CFA 2.0, and CFA 3.0. We then offer some explanations for the above inconsistency. From Figure 13 and Figure 14, one can see that CFA 2.0 and CFA 3.0 results have lower noise. However, from the objective results shown in Table 3, CFA 1.0 is the overwhelming favourite.
What is happening?
The reason is because, in our objective metrics, we followed the same procedures in [7] in which the realistic low lighting images are treated as ground truth. This is inappropriate because the low lighting images are not noise free. Using noisy images as ground truth can lead to erroneous conclusions.
For real low lighting images, one alternative performance metric is to adopt blind image quality assessment algorithms such as Naturalness Image Quality Evaluator (NIQE) in [25] [39].

CONCLUSIONS
In this paper, we investigate the performance of CFA 3.0 using realistic low lighting images. More than ten algorithms have been applied. Using both objective and subjective evaluations, we observed that, despite the availability of more white pixels, the demosacing performance of CFA 2.0 and CFA 3.0 was not as good as CFA 1.0 in our objective metrics. However, the performance does get better in CFA 2.0 and CFA 3.0 as compared to CFA 1.0 when using subjective evaluations. The above is a puzzling observation because we observed that CFA 3.0 and CFA 2.0 are better than CFA 1.0 using emulated low lighting images in our earlier papers [8] [10]. We offered an explanation for this phenomenon.
In the future, we plan to work on two research directions. First, since there is noise in the demosaiced images of real low lighting images, we plan to investigate various denoising algorithms to further improve the image quality. Second, for CFA 3.0, we believe that if the panchromatic band is improved, the overall quality of demosaicing will be further improved. We plan to investigate some new ways to generate the panchromatic band.