{'fraction_non_alphanumeric': 0.030737231338109544, 'fraction_numerical': 0.010515368615669055, 'mean_word_length': 4.915926179084074, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Channel-Wise Predictive Filter Flow for Demosaicking

Demosaicking is an image reconstruction process to recover a full color image from color filter array (CFA) mosaic data. Recently, deep convolutional neural network (CNN)-based demosaicking methods have been explored and have achieved state-of-the-art accuracy. In the deep-CNN-based demosaicking, output pixels are affected by a large spatial region; however, the information involved in demosaicking often only exists locally. In this paper, we propose a channel-wise predictive filter flow (PFF) for demosaicking. Since the PFF is a model that predicts a space-variant linear filter that is transformed to the target image by linearly combining it with the input image, target pixels are reconstructed only from local information. To incorporate the PFF into demosaicking, the proposed network synthesizes the filter flow corresponding to each channel by different networks that are learned independently. Experimental results demonstrate that the proposed method provides better or competitive results compared with several state-of-the-art deep-CNN-based demosaicking algorithms.


Introduction
Digital cameras acquire an image by converting light into electrical signals using an image sensor. Digital cameras are commonly equipped with a single image sensor with a color filter array (CFA) to acquire mosaic data with only one-color information at each pixel. The Bayer CFA, which consists of RGB color filters, is the most popular CFA pattern. To recover a full-color image from mosaic data, the missing twocolor information must be interpolated [1]. This process is referred to as demosaicking or CFA interpolation. Estimating each color channel separately using spatially invariant standard interpolation techniques results in high-frequency information being lost in the reconstructed image, leading to artifacts such as false colors and the zipper effect. To reduce these artifacts, many demosaicking algorithms have been proposed.
Recently, deep convolutional neural network (CNN) methods have provided superior performance for various image processing tasks such as super-resolution, demosaicking, classification, deblurring and denoising. In deep-CNN-based image processing, global features can be utilized by a wide receptive field, so that the outputs are affected by a large spatial region. However, in image reconstruction tasks such as superresolution and demosaicking, the information involved in image reconstruction often only exists locally. In this paper, we propose a channel-wise predictive filter flow (PFF) for demosaicking. The PFF predicts weights of a space-variant linear filter to obtain target pixels by linearly combining them with the input image, so that the target pixels are reconstructed only from local information. To incorporate the PFF into demosaicking, we propose a novel framework that predicts the different filter flows corresponding to each output channel.
The contributions of this paper are summarized as follows: (1) A channel-wise PFF for demosaicking is proposed and a novel framework for incorporating the PFF into the demosaicking is shown. (2) Experimental results on the Kodak and McMaster datasets demonstrate that the proposed method provides better or competitive results compared with state-ofthe-art deep-CNN-based demosaicking algorithms.

Predictive Filter Flow (PFF)
The PFF was proposed as a model that predicts the optimal filter flow using a CNN [2]. The filter flow is a space-variant linear filter used to reconstruct the target image by linearly combining it with the input image [3]. In the PFF, the filter flow is predicted through the CNN, and output pixels depend only on the local neighborhood centered on the input image at the same coordinate as the target pixel. The filter flow T is obtained by optimizing the parameter θ of the network F () from the given input image I 1 and target image I 2 .
where N indicates the number of training observations.

Channel-Wise PFF Network
To incorporate the PFF into the demosaicking, we design a novel demosaicking framework with a channel-wise PFF illustrated in Fig. 1. The proposed demosaicking framework consists of two parts, feature extraction and channel-wise filter flow synthesis. In the first part, the features are efficiently extracted by an ensemble of deep and shallow CNNs. In the second part, the channel-wise filter flows are obtained by using different networks corresponding to each output channel.

Feature extraction
As shown in Fig. 1, the feature extraction consists of a two-stream network of deep and shallow CNNs. The deep CNN extracts the global and complex features, and the shallow CNN extracts the local features and retains the spatial information of the Bayer pattern CFA image. The proposed method adopts the residual-in-residual (RIR) structure proposed in [4] as the deep CNN. The RIR (Fig. 2 (a)) consists of a long skip connection and several residual groups (RGs), and each RG consists of a short skip connection and several residual blocks (RB) (Fig. 2 (b)). The RIR can train the very deep-CNN using the residual structure with the long and short skip connections. The shallow CNN (Fig. 2 (c)) consists of 1 × 1 and 3 × 3 convolution layers with a ReLU layer. From the features obtained by concatenating the deep and shallow CNNs, filter flows for each channel are synthesized by a 3-layer network consisting of 3 × 3 convolutional, batch normalization and ReLU layers (Fig. 3). These convolution layers output a 3-dimensional array with the channel dimension of the filter size as the linear filter weight corresponding to each spatial dimension. In order for each filter to maintain the output intensity, the softmax function is applied to the last layer instead of the ReLU layer. Filter flow synthesis networks are prepared for each channel, and these networks do not share weights and are learned independently. The prediction image is obtained by multiplying the synthesized channel-wise filter flows and the input mosaicked image reshaped by using the "im2col" operation, and summing the elements in the channel direction.

Experiments
For training, we use the Waterloo Exploration Database (WED) dataset [5] consisting of 4744 color images. The training dataset is randomly cropped to 64 × 64-size patches and downsampled to the Bayer pattern CFA. The hyperparameters in the training of our proposed network are set as follows: the filter size is 25 × 25, the number of RGs in the RIR is 10, the number of RBs in an RG is 20, the mini-batch size is 10, the initial learning rate of the Adam optimizer [6] is 10 −4 and the number of epochs is 500. For testing, we use the Kodak dataset [7] consisting of 24 color images (768 × 512) and the McMaster dataset [8] consisting of 18 color images (500 × 500). Our proposed network was implemented on PyTorch with an NVIDIA 1080 GPU.

Quantitative result
As a quantitative comparison, Table 1 shows the demosaicking results of the proposed method and two state-of-theart CNN-based demosaicking methods, 3-stage CNN struc-ture (3-Stage) [9] and lightweight deep residual learning (LDRL) [10], for the color peak signal-to-noise ratio (CP-SNR) and structural similarity (SSIM) measures. As shown in Table 1, the proposed method shows a higher SSIM on the Kodak dataset and higher CPSNR and SSIM on the McMaster dataset than the other demosaicking methods. The average execution times to reconstruct the Kodak and McMaster images using the proposed method are approximately 3.587 s and 2.398 s, respectively.

Visual result
As a visual comparison, Fig. 4 and Fig. 5 show part of the demosaicking results obtained by 3-Stage, LDRL and the proposed method for the Kodak and McMaster datasets. As shown in Fig. 4, the proposed method restores object structures such as the edges of the fence more accurately than the other techniques. However, false colors occur at highfrequency edge parts. Also, Fig. 5 shows that the proposed method produces an image similar to the original one in regions where the color information is intricately mixed.

Conclusions
A channel-wise PFF for demosaicking is proposed. The proposed method synthesizes a channel-wise filter flow using different networks that are learned independently to incorporate the PFF into demosaicking. In accordance with experimental results, the proposed method provides better or competitive results on benchmark datasets compared with other deep-CNN-based demosaicking techniques. In particular, the quantitative result demonstrates a higher SSIM because the PFF reconstructs target pixels using only the local neighborhood pixels.