{'fraction_non_alphanumeric': 0.026468203405536895, 'fraction_numerical': 0.010135526468203405, 'mean_word_length': 4.841339648173207, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enhanced Image Reconstruction From Quarter Sampling Measurements Using An Adapted Very Deep Super Resolution Network

Quarter sampling is a novel sensor concept that enables the acquisition of higher resolution images without increasing the number of pixels. This is achieved by covering three quarters of each pixel of a low-resolution sensor such that only one quadrant of the sensor area of each pixel is sensitive to light. By randomly masking different parts, effectively a non-regular sampling of a higher resolution image is performed. Combining a properly designed mask and a high-quality reconstruction algorithm, a higher image quality can be achieved than using a low-resolution sensor and subsequent upsampling. For the latter case, the image quality can be enhanced using super resolution algorithms. Recently, algorithms based on machine learning such as the Very Deep Super Resolution network (VDSR) proofed to be successful for this task. In this work, we transfer the concepts of VDSR to the special case of quarter sampling. Besides adapting the network layout to take advantage of the case of quarter sampling, we introduce a novel data augmentation technique enabled by quarter sampling. Altogether, using the quarter sampling sensor, the image quality in terms of PSNR can be increased by +0.67 dB for the Urban 100 dataset compared to using a low-resolution sensor with VDSR.


INTRODUCTION
Using quarter sampling [1], the spatial resolution of an imaging sensor can be increased. This is achieved by physically covering three quarters of each pixel of a low-resolution sensor as it is illustrated in Figure 1. Effectively, this leads to a non-regular sampling of the image with respect to a higher resolution grid with twice the resolution in both spatial dimensions. Since the sampling is non-regular, it leads to reduced visible aliasing artifacts conventionally occurring for regular sampling [2,3,4]. The missing pixels need to be reconstructed from the sampled data. For a proper reconstruction, high-quality reconstruction algorithms such as the frequency selective reconstruction (FSR) [5] need to be used in combination with optimized quarter sampling patterns such as those in [6]. FSR has shown to be a successful reconstruction scheme for various inpainting and extrapolation tasks [7,8,9] and showed best results for nonregular sampling and quarter sampling in [1,5,6].
Another approach to acquire high resolution images without increasing the number of measured pixels is to upscale an image from a low-resolution sensor as it is also shown in Figure 1. Unfortunately, using standard interpolation methods leads to a blurred and degraded image. A higher image quality can be achieved using Reference image QS sensor LR sensor Masking Reconstruction Upscaling Fig. 1. Illustration of an image acquisition using a low-resolution (LR) sensor and a quarter sampling (QS) sensor. Light gray pixels indicate measured pixels whereas dark gray pixel indicate upscaled/reconstructed pixels.
super-resolution algorithms. During the last decade, great progress has been made in this field using sparse representation based approaches [10] as well as convolutional neural network based approaches [11]. Recently, the Very Deep Super Resolution (VDSR) network [12] was proposed. Other than previous neural networks, it does not use a low-resolution image as input but enhances an image that was previously upscaled with bicubic interpolation by learning the residuum between the unavailable high resolution image and the upscaled image. Using a rather simple network structure, VDSR achieves a high quality reconstruction especially at sharp edges that otherwise appear blurred by simple interpolation methods such as bicubic upscaling.
Since VDSR is capable of improving the image quality of upscaled images, the question arises whether its concepts can also be used to further enhance images that were reconstructed from a quarter sampling sensor. Though edges are typically reconstructed quite well in that case, other artifacts such as ringing may occur, which could potentially also be removed by learned filters present in VDSR. In this paper, we transfer the techniques used in VDSR to remove such undesirable artifacts from images reconstructed from quarter sampling sensors while keeping the high-resolution structures made possible by quarter sampling. Besides simply retraining VDSR for the case of quarter sampling in a first step, we further propose an adapted version (VDSR-QS) which incorporates knowledge available for the special case of quarter sampling. Moreover, quarter sampling allows us to exploit a novel degree of freedom for data augmentation which is not accessible for the conventional case of the low-resolution sensor.
This paper is organized as follows: In Section 2, we briefly introduce the VDSR network and the corresponding processing chain for the low-resolution sensor. Then, we present the processing chain for a quarter sampling sensor incorporating an adapted VDSR network and introduce a novel technique for data augmentation. In Section 3, we perform experiments that compare both processing chains and the used adaptations. We furthermore evaluate and discuss the results and provide visual examples. In Section 4, we summarize the paper and give an outlook to future work.

VDSR IMPLEMENTATIONS FOR LOW-RESOLUTION
SENSORS AND QUARTER SAMPLING SENSORS

VDSR for a Low-resolution Sensors
For a low-resolution sensor as it is shown in Figure 1, the acquisition of the image can be described as the filtering of a high resolution reference image fij with a 2×2 filtering kernel of ones followed by a twofold sub-sampling in both spatial dimensions. Effectively, each pixel measures the mean value of four high resolution pixels from a 2×2 neighborhood. Prior to the application of a super-resolution algorithm such as the VDSR [12], the image is upscaled using bicubic interpolation leading to an approximate solutionfij. In order to enhance the resolution of the upscaled image, we feed the upscaled imagefij into VDSR [12] being a convolution neural network trained to infer the residual rij = fij −fij. The resulting image is calculated by summing the input and output of VDSR, i.e., fij =fij + rij.
The full processing chain for the case of the low-resolution sensor is shown in Figure 2 (a). For the network architecture, we use a custom Tensorflow [13] and Keras [14] implementation close to the original work in [12]. For this implementation, as defined in the original paper, 20 convolutional layers with subsequent Rectified Linear Unit (ReLU) layers are used with 3×3×64 filter kernels to calculate the residual image. This results in feature maps of the size W ×H×64, with W and H representing the width and the height of the input image, respectively. Zero-padding is applied at the boundaries for the convolutions. During training, the bicubically upscaled images are separated into patches with a size of 41×41 pixels and compared against the corresponding patches from the original data. Therewith, the weights are adapted using the Adam optimizer [15] which showed superior performance than the stochastic gradient decent (SGD) method used for the original VDSR. As a loss function, the euclidean distance loss is taken. Using Adam optimizer also requires to adapt the learning rate, which we initially set to 0.0001 and decrease by a factor of 10 after every 10-th epoch. In total, the network is trained for 30 epochs. To increase the number of patches, and thus avoid overfitting during training, data augmentation is used by flipping and rotating the patches before feeding into the network. We further choose a batch size of 64 and a gradient clip value of 0.

VDSR and proposed VDSR-QS for Quarter Sampling Sensors
For the case of the quarter sampling sensor, a very similar processing chain can be followed. We describe the non-regular sub-sampling of the quarter sampling sensor as an element-wise multiplication of the reference image fij with a binary mask bij. At this point, the sampled image (fij·bij) contains as many non-zero entries as pixels of the quarter sampling sensor. In order to reconstruct the missing pixels on the high-resolution grid, we use FSR [5] leading to a first reconstruction which we also callfij due to the strong similarity with the bicubically upscaled image in the previous section. This reconstructed image is now fed into the VDSR network which we retrain for this data. This processing chain is illustrated in Figure 2 (b).
Here, an interesting observation can be made: Since FSR leaves the pixels that were actually measured with the quarter sampling sensor untouched, we know that those pixels where the condition bij = 1 is true are identical to the pixel values in the reference image. It is therefore reasonable to set the residual at the output of the neural network to zero for those positions. The final image than reads This adaptation of VDSR is named VDSR-QS and is highlighted with red color in Figure 2 (b). Due to the differences, the VDSR-QS network is also trained independently from scratch. similar, the same set of hyper-parameters are used. The only exception is the learning rate for VDSR-QS. Since 25% of the entries of the residual rij · (1 − bij) are set to zero, the loss function is 25% smaller in average. To compensate for this, we multiply the learning rate in VDSR-QS by a factor of 4/3. In terms of the quarter sampling mask, we use the optimized quarter sampling mask from [6] because it shows an improved reconstruction quality forfij. This mask is of size 32×32 pixels and is repeated periodically until the respective reference image is covered. Conveniently, such periodicity is beneficial for a future hardware implementation.

Novel Data Augmentation for Quarter Sampling Sensor
Intriguingly, using the quarter sampling sensor allows for a novel dimension for the data augmentation. From the available training images, more training data can be created by performing measurements with different sampling masks leading to different reconstructionsfij. To achieve this, the sampling mask is shifted by several pixels relative to the reference image. The different inputsfij to the neural network effectively increase the amount of training data. For the used sampling mask from [6], a total of 1024 unique shifts are conceivable. We use the first, only the first two, only the first four as well as all shifted masks shown in Figure 3 were used in independent trainings. While 7 further doublings of the number of used masks are thinkable this would also lead to an increase of the computation time for the training phase in the same manner and is therefore omitted.

SIMULATIONS AND RESULTS
In this section, we evaluate the performance of the VDSR-like networks for image measurements using a quarter sampling sensor. First, the influence of the adapted network layout in VDSR-QS is evaluated relative to VDSR and the influence from the novel data augmentation is shown. Second, the combination of both proposals is used to compare the reconstruction quality with the case of the low-resolution sensor.
In terms of the initial upscaling/reconstruction algorithms, we use bicubic upscaling and FSR with the same parameters as in [5]. For the neural networks, we use our own custom TensorFlow [13]  implementation which was tested to show comparable results with respect to the original work in [12]. One training is performed for the low-resolution sensor. Please note, that this case is slightly different from the case investigated in [12] as they concentrate on a different downscaling filter not accessible in a low-resolution sensor.
In case of the quarter sampling sensor, we performed eight separate trainings. A factor of four in the number of training arises from the different amounts of data augmentation as discussed in Section 2.3 and another factor of two arises from training the networks for the VDSR-QS, too. In terms of the training data, we use the image Set 291 as in [12] consisting of 291 images of various natural content. We simulate the monochrome low-resolution and quarter sampling sensors by converting all color images to grayscale with 8 bit depth.
To evaluate the quality of the resulting images, we calculate the mean PSNR and the mean structural similarity (SSIM) [16] for the images of the Urban 100 dataset from [17]. This dataset consists of various images of urban architecture. The content is of special interest as it is shows many of the structures where a low-resolution sensor may fail due to aliasing. Figure 4 shows the reconstruction quality in terms of PSNR using the quarter sampling sensor together with FSR and VDSR (blue squares) as well as VDSR-QS (orange circles). Four different amounts of data augmentation by shifting the quarter sampling mask were used as shown on the horizontal axis. With an increasing amount of data augmentation, the PSNR increases by roughly +0.2 dB per doubling, leading to an increase of +0.74 dB in total for the VDSR. Interestingly, no saturation is yet visible for the shown number of used masks. Using the VDSR-QS leads to an additional gain of up to +0.21 dB. Here, a similar increase is observed across all amounts of data augmentation. Overall, an increase of +0.89 dB was achieved by incorporating the knowledge that the quarter sampling sensor was used and by exploiting the novel degree of freedom in the data augmentation.
With these findings, we next compare the reconstruction quality using the quarter sampling sensor and the low-resolution sensor, both used with the VDSR networks. In Table 1 the image quality in terms of PSNR and SSIM is shown. For completeness, the intermediate image quality after the upscaling/reconstruction using bicubic interpolation/FSR is also provided. For the quarter sampling sensor, the data is given for the case of the VDSR (no adaptation, no novel data augmentation) and the VDSR-QS with 8-fold data augmentation using all masks from Figure 3 which leads to the highest PSNR and highest SSIM. From Table 1, we find that the reconstruction quality using the quarter sampling sensor with the VDSR without adaptations is 0.22 dB lower than using the low-resolution sensor with VDSR. However, using VDSR-QS and the additional 8-fold data augmentation, the quarter sampling sensor achieves a image quality of 29.29 dB in terms of PSNR, being +0.67 dB higher than for the VDSR with the low-resolution sensor. Compared to the reconstruction using only the FSR, the VDSR-QS together with an 8-fold data augmentation gains more than +2.2 dB. The SSIM values verify all these findings. Overall, a SSIM gain of +0.0266 compared to using FSR is achieved using VDSR-QS and the 8-fold data augmentation.
In Figure 5, we show visual examples for three sections of images from the Urban 100 dataset. Regions with high frequency content are affected from severe aliasing in case of the low-resolution sensor regardless of using VDSR or not. Other than that, using the quarter sampling sensor together with FSR and VDSR or VDSR-QS increases the image quality. This is especially visible for highfrequency content where the aliasing may even change the angle of parallel lines for the low-resolution sensor as can be seen in the second image. Compared to the reconstruction with the FSR, the ringing-like artifacts, e.g., occurring around some of the edges, are significantly reduced, while the correct high-frequency content is preserved.

CONCLUSION
In this paper, we transfer the concepts of Very Deep Super Resolution network (VDSR) to the special case of quarter sampling enhance the reconstruction quality of image data acquired with a quarter sampling sensor. In doing so, we propose an adapted version of VDSR called VDSR-QS to incorporate the special property of the quarter sampling measurements that the exact value of some of the highresolution pixels are known. Moreover, we examine a novel degree of freedom in the data augmentation only possible for the case of quarter sampling. Combining the VDSR-QS with such 8-fold data augmentation, we successfully increase the reconstruction quality of images measured with a quarter sampling sensor by +2.2 dB. Compared to a low-resolution sensor and VDSR, we gain +0.67 dB. Moreover, we provide visual examples showing that aliasing occurring from a measurement with the low-resolution sensor is not present for the quarter sampling sensor and the artifacts usually occurring in quarter sampling measurement reconstructed with FSR are significantly reduced.  For future work, we would like to transfer the advantages of VDSR-QS to other non-regular measurement scenarios such as three-quarter sampling, where randomly oriented L-shaped pixels are used resulting in higher reconstruction quality [18]. Even though, similar to the low-resolution sensor, a low-pass filter is effectively applied prior to the measurement, the advantages of the quarter sampling are kept for the three-quarter sampling. However, due to the filtering, adaptations such as in VDSR-QS are not straightforward.

ACKNOWLEDGMENT
We gratefully acknowledge that this work has been supported by the Deutsche Forschungsgemeinschaft (DFG) under contract number KA 926/5-3.