{'fraction_non_alphanumeric': 0.030164374704697485, 'fraction_numerical': 0.007907890483177082, 'mean_word_length': 4.708161816891413, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Light Field Super-Resolution using a Low-Rank Prior and Deep Convolutional Neural Networks

Light field imaging has recently known a regain of interest due to the availability of practical light field capturing systems that offer a wide range of applications in the field of computer vision. However, capturing high-resolution light fields remains technologically challenging since the increase in angular resolution is often accompanied by a significant reduction in spatial resolution. This paper describes a learning-based spatial light field super-resolution method that allows the restoration of the entire light field with consistency across all sub-aperture images. The algorithm first uses optical flow to align the light field and then reduces its angular dimension using low-rank approximation. We then consider the linearly independent columns of the resulting low-rank model as an embedding, which is restored using a deep convolutional neural network (DCNN). The super-resolved embedding is then used to reconstruct the remaining sub-aperture images. The original disparities are restored using inverse warping where missing pixels are approximated using a novel light field inpainting algorithm. Experimental results show that the proposed method outperforms existing light field super-resolution algorithms, achieving PSNR gains of 0.23 dB over the second best performing method. This performance can be further improved using iterative back-projection as a post-processing step.


INTRODUCTION
L IGHT field imaging has emerged as a promising technology for a variety of applications going from photorealistic image-based rendering to computer vision applications such as 3D modeling, object detection, classification and recognition. As opposed to traditional photography which captures a 2D projection of the light in the scene, light fields collect the radiance of light rays along different directions [1], [2]. This rich visual description of the scene offers powerful capabilities for scene understanding and for improving the performance of traditional computer vision problems such as depth sensing, post-capture refocusing, segmentation, video stabilization and material classification to mention a few.
Light fields acquisition devices have been recently designed, going from rigs of cameras [2] capturing the scene from slightly different viewpoints to plenoptic cameras using micro-lens arrays placed in front of the photo-sensor [1]. These acquisition devices offer different trade-offs between angular and spatial resolution. Rigs of cameras capture views with a high spatial resolution but in general with a limited angular sampling hence large disparities between views. On the other hand, plenoptic cameras capture views with a high angular sampling, but at the expense of a limited spatial resolution. In plenoptic cameras, the angular sampling is related to the number of sensor pixels located behind each microlens, while the spatial sampling is related to the number of microlenses. Light fields represent very large volumes of high dimensional data bringing new challenges in terms of capture, compression, editing and display. The design of efficient light field image processing algorithms, going from analysis, compression to super-resolution and editing has thus recently attracted interest from the research community. A comprehensive overview of light field image processing techniques can be found in [3].
This paper addresses the problem of light field spatial super-resolution. Single image super-resolution has been an active field of research in the past years, leading to quite mature solutions. However, super-resolving each view separately using state of the art single-image super-resolution techniques would not take advantage of light field properties, in particular of angular redundancy which depends on scene geometry [4]. Moreover, considering each subaperture image as a separate entity may reconstruct light fields which are angularly incoherent [5].
Assuming that the low-resolution light field captures different views of the same scene taken with sub-pixel misalignment, the problem can be posed as the one of recovering the high-resolution (HR) views from multiple low-resolution images with unknown non integer translational misalignment. A number of methods hence proceed in two steps. A first step consists in estimating the translational misalignment using depth or disparity estimation techniques. The HR light field views are then found using Bayesian or variational optimization frameworks with different priors. This is the case in [6] and [7] where the authors first recover a depth map and formulate the spatial light field super-resolution problem either as a simple linear problem [6] or as a Bayesian inference problem [7] assuming an image formation model with Lambertian reflectance priors and a depth-dependent blurring kernel. A Gaussian mixture model (GMM) is proposed instead in [8] to address denoising, spatial and angular super-resolution of light fields. The reconstructed 4D-patches are estimated using a linear minimum mean square error (LMMSE) estimator, assuming a disparity-dependent GMM for the patch structure. In [9], the geometry is estimated by computing structure tensors in the Epipolar Plane Images (EPI). A variational optimization framework is then used to spatially superresolve the different views given their estimated depth maps and to increase the angular resolution.
Another category of methods is based on machine learning techniques which learn a model of correspondences between low-and high-resolution data. In [5], the authors learn projections between low-dimensional subspaces of 3D patch-volumes of low-and high-resolution, using ridge regression. Data-driven learning methods based on deep neural network models have been recently shown to be quite promising for light fields super-resolution. Stacked input images are up-scaled to a target resolution using bicubic interpolation and super-resolved using a spatial convolutional neural network (CNN) in [10] which learns a non-linear mapping between low-and high-resolution views. The output of the spatial CNN is then fed into a second CNN to perform angular super-resolution. While the approach in [10] takes at the input of the spatial CNN pairs or 4-tuples of neighboring views, leading to three spatial CNNs to be learned, a single CNN is proposed by the same authors in [11] to process each view independently. The problem of angular super-resolution of light fields is also addressed in [12] using an architecture based on two CNNs, one CNN being used to estimate disparity maps and the second CNN being used to synthesis intermediate views.
The authors in [13] define a CNN architecture in the EPI to increase the angular resolution.
One can also cite some related work using a hybrid light field imaging system coupling a high-resolution camera with either a light field camera [14] or with multiple lowresolution cameras [15]. In [14], the HR image captured by the DSLR camera is used to super-resolve the low-resolution images captured by an Illum light field camera. The authors in [15] describe an acquisition device formed by eight lowresolution side cameras arranged around a central highquality SLR lens. A super-resolution method, called iterative patch-and depth-based synthesis (iPADS), is then proposed to reconstruct a light field with the spatial resolution of the SLR camera and an increased number of views.
In this paper, we propose a spatial light field superresolution method using a deep CNN (DCNN) with ten convolutional layers. Instead of using DCNN to restore each sub-aperture image independently, as done in [10], [11], we restore all sub-aperture images within a light field simultaneously. This allows us to exploit both spatial and angular information to restore the light field and thus generate light fields which are angularly coherent. A Naïve approach is to train a DCNN with n = P × Q inputs, where P and Q represent the number of vertical and horizontal angular views respectively. However, this will significantly increase the complexity of the DCNN which makes it harder to train and more prone to over-fitting . Instead, given that each sub-aperture image captures the same scene from a different view point, we align all sub-aperture images to the centre view using optical flow and then reduce the angular dimension of the aligned light field using a low-rank model of rank k, where k n. Results in section 3.1 show that the alignment allows us, with the considered low rank model, to significantly reduce the angular dimension of the light field. The linearly independent column-vectors of the low-rank representation of the aligned light field, which constitute an embedding of the light field views in a lower-dimensional space, are then considered as a volume and simultaneously restored using a DCNN with k input channels. This allows us to significantly reduce the complexity of the network which is easier to train while still preserving angular consistency. The restored column-vectors are then combined to reconstruct the aligned high-resolution light field. In the final stage we use inverse warping to restore the original disparities of the light field and fill the cracks caused by occlusion using a novel diffusion based inpainting strategy that propagates the restored pixels along the dominant orientation of the EPI.
Simulation results demonstrate that the proposed method outperforms all other schemes considered here when tested on 13 different light fields from two different datasets. It is important to mention that our method was not trained on the Stanford light fields and these results clearly show that our proposed method generalizes well even when considering light field structures whose disparities are significantly larger than those used for training. Further analysis in section 4 shows that additional gain in performance can be achieved using iterative back projection (IBP) as a post processing step. These results show that our method can significantly outperform existing light field super-resolution methods including the deep learningbased light field super-resolution method presented in [11].
This paper is organized as follows. After introducing the notation in section 2, we describe the proposed method in section 3. Section 4 discusses the simulation results with different types of light fields and provide the final concluding remarks in section 5.

NOTATION AND PROBLEM FORMULATION
We consider here the simplified 4D representation of light fields called 4D light field in [16] and lumigraph in [17], describing the radiance along rays by a function I(x, y, s, t) where the pairs (x, y) and (s, t) respectively represent spatial and angular coordinates. The light field can be seen as capturing an array of viewpoints (called sub-aperture images) of the scene with varying angular coordinates (s, t).
The different views will be denoted here by I s,t ∈ R X,Y , where X and Y represent the vertical and horizontal dimension of each sub-aperture image.
In the following, the notation I s,t for the different subaperture images will be simplified as I i with a bijection between (s, t) and i. The complete light field can hence be represented by a matrix I ∈ R m,n : with vec(I i ) being the vectorized representation of the i−th sub-aperture image, m represents the number of pixels in each view (m = X × Y ) and n is the number of views in the light field (n = P ×Q), where P and Q represent the number of vertical and horizontal angular views respectively.
Let I H and I L denote the high-and low-resolution light fields, respectively. The super-resolution problem can be formulated in Banach space as where η is an additive noise matrix, ↓ α is a downsampling operator applied on each sub-aperture image where α is the magnification factor and B is the blurring kernel. There are many possible high-resolution light fields I H which can produce the input low-resolution light field I L via the acquisition model defined in (2). Hence, solving this illposed inverse problem requires introducing some priors on I H , which can be a statistical prior such as a GMM model [8], or priors learned from training data as in [5], [10], [11]. Another way to visualize a light field is to consider the EPI representation. An EPI is a spatio-angular slice from the light field, obtained by fixing one of the spatial coordinates and one of the angular coordinates. Consider we fix y := y * and t := t * , an EPI is an image defined as y * ,t * := I(x, y * , s, t * ). Alternatively, the vertical EPI is obtained by fixing x := x * and s := s * . Figure 5b shows a typical EPI structure, where the slopes of the isophote lines in the EPI are related to the disparity between the views [9]. Isophote lines with a slope of π/2 rad indicate that there is no disparity across the views while the larger is the difference between the slope and π/2 rad the larger is the disparity across the views. Figure 1 depicts the block diagram of the proposed spatial light field super-resolution algorithm. Each sub-aperture image of the low-resolution light field I L is first bicubic interpolated so that both I H and I L have the same resolution. While a light field consists of a very large volume of high-dimensional data, it also contains a lot of redundant information since every sub-aperture image captures the same scene from a different viewpoint. Moreover, different light field capturing devices have different spatial and angular specifications, which makes it very hard for a learningbased algorithm to learn a generalized model suitable to restore all kind of light fields irrespective of the capturing device. The Dimensionality Reduction module tries to solve both problems simultaneously where it uses optical flow to align the light field and a low-rank matrix approximation to reduce the dimension of the light field. Results in section 3.1 show that we can reduce the dimensionality of the light field from R m,n to R m,k , where k n is the rank of the matrix, while preserving most of the information contained in the light field.

PROPOSED METHOD
The Light Field Restoration module then considers the k linear independent column-vectors of the rank-k representation of the low-resolution light field as an embedding of the light field. We then use a DCNN to recover the texture details of the light field embedding in the lower dimensional space. The super-resolved embedding gives an estimate of the aligned high-resolution light field. The Light Field Reconstruction module then warps the estimated aligned high-resolution light field to restore the original disparities. Holes corresponding to cracks or occlusions are then filled in by diffusing information in the Epipolar Plane Images (EPI) along directions of isophote lines computed, for the positions of missing pixels, in the EPI of the lowresolution light field. Iterative back-projection can be further used as a post-process to refine the super-resolved light field and assure to be consistent with the low-resolution light field. More information about each module is provided in the following subsections.

Light Field Dimensionality Reduction
Many acquisition devices have been recently designed to capture light fields, including multi-sensor approaches [2], time-sequential capture methods [18], [19] and plenoptic cameras [1], [20]. All these different light field cameras have diverse spatial and angular specifications, which makes it very hard for a learning-based algorithm to learn a generalized model suitable to restore any kind of light field independent from the source. Moreover, a light field contains a huge amount of redundant information since it represents a different view of the same scene.
The authors in [21] hypothesized that this redundancy can be suppressed by jointly aligning the sub-aperture images in the light field and estimating a low-rank approximation (LRA) of the light field. This approach has shown very promising results in the field of light field compression. In the same spirit, the RASL algorithm [22] was used to find the homographies that globally align a batch of linearly correlated images. Both methods seek for an optimal set of homographies such that the matrix of aligned images can be decomposed in a low-rank matrix of aligned images, with the latter constraining the error matrix to be sparse. The results in Figure 2 show that while both RASL and LRA methods manage to globally align the sub-aperture images, the mean sub-aperture image is still very blurred, indicating that the light field is not suitably aligned.
The authors in [5] have used the block matching algorithm (BMA) to align patch volumes. The results in Figure 2 show that BMA manages to align better the sub-aperture images, where the average variance across the n sub-aperture image is significantly reduced. This result suggests that local methods can improve the alignment of the sub-aperture images, which as we will see in the sequel, will allow us to significantly reduce the dimensionality of the light field.
In this paper, we formulate the light field dimensionality reduction problem as where u ∈ R m,n and v ∈ R m,n are flow vectors that specify the displacement of each pixel needed to align each subaperture image with the centre view, A is a rank-k matrix which approximates the aligned light field and Γ u,v (·) is a forward warping operator (which here performs a disparity compensation where the disparities maps (u, v) are estimated with an optical flow estimator). This optimization problem is computationally intractable. Instead, we decompose this problem in two sub-problems: i) use optical flow to find the flow matrices u and v that best align each subaperture image with the centre view and ii) use low-rank approximation to derive the rank-k matrix that minimizes the error with respect to the aligned light field.   Underneath each image we provide the average variance across the n sub-aperture images which was used in [5] to characterize the performance of the alignment algorithm, where smaller values indicate better alignment.

Optical Flow
The problem of aligning all the sub-aperture images with the centre view can be formulated as where j corresponds to the index of the centre view, and (u i , v i ) are the flow vectors optimal to align the i-th sub-aperture image with the centre view. There are several optical flow algorithms intended to solve this problem [23], [24], [25], [26] where Figure 2 shows the performance of some of these methods. It can be seen that the mean aligned sub-aperture images computed using [23] is generally blurred while those aligned using the methods in [25], [26] generally provide ghosting artefacts at the edges. Moreover, it can be seen that SIFT Flow [24] generally provides very good alignment and manages to attain the smallest variation across the sub-aperture images. While the SIFT Flow algorithm will be used in this paper to compute the flow vectors, any other optical flow method can be used.

Low-Rank Approximation
Given that the flow-vectors (u i , v i ) for the i-th sub-aperture image are already available, the minimization problem in Eq. (3) can now be reduced to where I L Γ = Γ u,v I L , B L ∈ R m,k is a rank-k matrix and C L ∈ R k,n is the combination weight matrix. These matrices can be found using singular value decomposition (SVD) I L Γ = UΣV T , where B L is set as the k first columns of UΣ and C L is set as the k first rows of V T , so that B L C L is the closest k-rank approximation of the aligned light field I L Γ . The error matrix E L is the error matrix which is simply computed using E L = I L Γ − B L C L Figure 3 depicts the performance of three different dimensionality reduction techniques at different ranks. To measure the dimensionality reduction ability of these methods we compute the root mean square error (RMSE) between the aligned original and the rank-k representation of the aligned light field. It can be seen that the RASL algorithm has the largest distortions at almost all ranks when compared to the other two approaches. On the other hand, it can be seen that HLRA manages to significantly outperform the RASL method, which gain is attained by the improved alignment. Nevertheless, it can be clearly observed that the proposed Sift Flow + LRA method gives the best performance, especially at lower ranks, indicating that more information is captured within the low-rank matrix.
To emphasize this point we show in figure 3 the principal basis of PCA, HLRA and Sift Flow + LRA. PCA is computed on the light field without disparity estimation and therefore can be considered here as a baseline to show that alignment allows us to get more information in the principal basis. Moreover, it can be seen that the principal basis derived using our Sift Flow + LRA manages to capture more texture detail in the principal basis than the other methods and confirms the benefit that local alignment has on the energy compaction ability of the proposed dimensionality reduction method. 1

Light Field Restoration
We consider a low-rank representation of the aligned lowresolution light field A L = B L C L , where A L ∈ R m,n is a rank-k matrix with k n. Similarly, A H = B H C H is a rank-k representation of the aligned high-resolution light field. The rank of a matrix is defined as the maximum number of linearly independent column vectors in the matrix. Moreover, the linearly dependent column vectors of a matrix can be reconstructed using a weighted summation of the linearly independent column vectors of the 1. Note that the RASL method does decompose the matrix into a combination of basis elements and therefore the principal basis of RASL could not be shown here. same matrix. This leads us to decompose A L in two submatrices:Ȃ L ∈ R m,k which groups the linear independent column vectors of A L andÂ L ∈ R m,n−k which groups the linearly dependent column vectors of A L . In practice, we decompose the rank-k matrix A L using QR decomposition (i.e. A L = QR). The index of the linearly independent components of A L then correspond to the index of the nonzero diagonal elements of the upper-triangular matrix R. We then use the same indices to decompose A H into sub-matricesȂ H andÂ H . The matrixÂ L can be reconstructed as a linear combination ofȂ L , where the weight matrix W is computed using where (·) † stands for the pseudo inverse operator. We assume here that the weight matrix W, which is optimal in terms of least squares to reconstructÂ L , is suitable to reconstructÂ H . Driven by the recent success of deep learning in the field of single-image [27], [28] and light field super-resolution [10], [11], we use a DCNN to model the upscaling function that minimizes the following objective function where f (·) is a function modelled by the DCNN illustrated in Figure 4 which has ten convolutional layers. The linearly independent sub-matrixȂ L is passed through a stack of convolutional and rectified linear unit (ReLU) layers. We use a convolution stride of 1 pixel with no padding nor spatial pooling. The first convolutional layer has 64 filters of size 3 × 3 × k while the last layer, which is used to reconstruct the high-resolution light field, employs k filter of size 3 × 3 × 64. All the other layers use 64 filters of size 3 × 3 × 64 which are initialized using the method in [29]. The DCNN was trained using a total of 200,000 random patch-volumes of size 64 × 64 × k from the 98 low-and high-resolution low-rank approximation of rank k of the light fields from the EPFL, INRIA and HCI datasets 2 . The Titan GTX1080Ti Graphical Processing Unit (GPU) was used to speed up the training process. During the evaluation phase, we estimate the superresolved linearly independent representation of the light fieldȂ H = f Ȃ L . We then estimate the super-resolved linear dependent part of the light field usinĝ The super-resolved low-rank representation of the aligned light field is then derived by the union of the two matriceŝ A H andȂ H i.e. A H =Â H ∪Ȃ H . The super-resolved light field is then reconstructed usinĝ 2. It must be noted that none of the light fields used for validation were used for training. It can be seen that using optical flow to align the light field followed by low-rank approximation attains the best performance. The images in the second row show the principal basis derived using different methods. The sharper the principal basis is the more information is being captured in the principal basis. Fig. 4: The proposed network structure which receives a low-resolution light field and restores it using the proposed DCNN.

Light Field Reconstruction
The restored aligned light fieldÎ H Γ has all sub-aperture images aligned with the centre view. A naïve approach to recover the original disparities of the restored sub-aperture images is to use forward warping. However, as can be seen in the first column of Figure 5a, forward warping is not able to restore all pixels and results in a number of cracks or holes corresponding to occlusions (marked in green). One can use either bicubic interpolation or inverse warping to fill the holes. However, in case of occlusions, the neighbouring pixels may not be well correlated with the missing information, which often results in inaccurate estimations (see Figure 5a second column). More advanced inpainting algorithms [30], [31] can be used to restore each hole separately. However, these methods do not exploit the light field structure and therefore provide inconsistent reconstruction of the same spatial region at different angular views. In this work, we use a diffusion based inpainting algorithm that estimates the missing pixels by diffusing information available in other views. Similar to the work in [32], we exploit the EPI structure to diffuse information along the dominant orientation of the EPI. However, instead of predicting the orientation of unknown pixels from their spatial neighborhood as done in [32], we exploit the similarity between the low-and super-resolved EPIs and use the structure tensor computed on the low-resolution EPI to guide the inpainting process in the high-resolution EPI.
Without loss of generality we consider the EPI where the dimensions y * and t * are fixed. The case of vertical slices is analogous. We first compute the structure tensor of the low-resolution EPI L y * ,t * at coordinates (x, s) using where ∇ stands for the single order gradient computed using the sobel kernel. The authors in [32] compute an average weighting of the columns of T (x, s) to derive the dominant orientation, where the weights are given by an anisotropy measure. Nevertheless, the anisotropy may fail in regions that are smooth and therefore the weighted average may fail in these regions to estimate the dominant direction of the EPI. This problem becomes an issue when computing these orientations on low-resolution versions of the light fields. Instead, we estimate the orientation at every pixel in the EPI by computing the eigen decomposition of T (x, s) and choose the direction θ (x, s) which corresponds to the eigen-vector with the smallest eigen-value. Moreover, driven by the observation that the disparities in a light field are typically small, and considering that the local slope in the EPI is proportional to the disparity, it is reasonable to assume that slopes which correspond to large disparities are less probable to occur. Therefore, to ensure that the tensor driven diffusion is performed along a single coherent direction per column of the EPI and reduce noise, the dominant orientation θ (x) is computed using the columnwise median of θ (x, s) whose orientation is in the range [ π 2 − α, π 2 + α] radians. In all our experiments we set α = π/4 rad. While the dominant orientation vectors are less noisy, we further reduce the noise by applying the Total Variation (TV-L1) denoising [33] on the orientation field θ (x), which searches to optimizê θ = arg minθ||∇θ|| 1 + λ||θ − θ|| 2 (11) where λ is the total variation regularization parameter and was set to 0.5 in our experiments. Figure 5b (top) shows the EPI of the low-resolution light field L y * ,t * and the dominant orientationsθ (x) marked by blue arrows.
The restored EPI H y * ,t * :=Î H (x, y * , s, t * ) (see Figure 5b (bottom)) has a number of missing pixels (marked in green). Consider that the hole we want to inpaint has coordinates (x p , s p ). The aim of the proposed diffusion based inpainting algorithm is to propagate known pixels in the orientation θ (x p ) to fill the missing pixels. The diffusion over the EPI where Tr (·) stands for the trace operator and H (x, s) denotes the Hessian of H y * ,t * at coordinates (x, s). The term θ (x p )θ (x p ) is used to enforce the diffusion to occur only in the direction of the isophote eigenvector. The missing pixels are restored iteratively by finding the solution to (12) which is closest to zero. Figure 5a (third column) shows the results attained using the proposed inpainting strategy.
In order to restore all the cracks in the light field we first fix t * to that of the center view and iteratively restore all the horizontal EPIs for all y * ∈ [1, Y ] by solving Eq. (12). This corresponds to filling the cracks for the centre row of the matrix of sub-aperture images. We then fix s * to that of the centre view and iteratively restore all the vertical EPIs for all x * ∈ [1, X] which effectively restores all cracks for the centre column of the matrix of sub-aperture images. The remaining propagations are performed row-by-row where each time we restore all pixels within t * ∈ [1, Q].

Iterative Back-Projection
One problem with the method proposed in this paper is that after we restore the aligned light fieldÎ H Γ we have to compute inverse warping to restore the original disparities. However, the inverse warping is not able to recover occluded regions and some pixels are displaced by ± 1pixel due to rounding errors. While the former problem is solved using the method described in Section 3.3, the second problem was not yet addressed. Nevertheless, the results illustrated in Tables 1, 2 and Figure 6 indicate that significant performance gains can be achieved even if we do not explicitly cater for distortions caused by rounding errors in the inverse warping process. Nevertheless, these distortions can be corrected using the classical method of iterative back projection [34], which is adopted by several single image super-resolution methods (see [35]) to ensure that the down sampled version of the super-resolved light field is consistent with the observed low-resolution light field. The IBP algorithm iteratively refines the estimated high-resolution light fieldĪ H κ at iteration κ by first backprojecting it into an estimated low-resolution light fieldĪ L κ usingĪ L κ = ↑ α ↓ α BĪ H κ (13) where ↓ α is a downsampling operator, ↑ α is the bicubic upscaling operation, α is the magnification factor and B is the blurring kernel. The deviation between the LR views found by back-projection and the original LR views is then used to further correct each HR estimated view of the light field asĪ The IBP algorithm is initiated by settingĪ H 0 =Î H and the iterative procedure terminates when κ = K. It was observed that significant improvements were achieved in the first few iterations and we therefore set K = 10 in our experiments. The restored light field following iterative back-projection is therefore set toĪ H =Ī H K .

EXPERIMENTAL RESULTS
The experiments conducted in this paper use both synthetic and real-world light fields from publicly available datasets. We use 98 light fields from the EPFL [36], INRIA 3 and 3. INRIA dataset: https://goo.gl/st8xRt HCI 4 for training. We conducted the tests using light fields from the INRIA and Stanford 5 datasets. We use the Stanford dataset in this evaluation since it has disparities significantly larger than both INRIA and EPFL light fields, which were captured using plenoptic cameras. Moreover, unlike the HCI dataset, the Stanford light fields capture real world objects. We therefore use this dataset to assess the generalization ability of the algorithms considered in this experiment to light fields which are captured using camera sensors which differ from the ones used for training. While the subaperture images of the EPFL, HCI and Stanford datasets are available, the light fields in the INRIA dataset were decoded using the method in [37] as mentioned on their website. In all our experiments we consider a 9×9 array of sub-aperture images. For computational purposes, the high-resolution images of the Stanford dataset were downscaled such that the lowest dimension is set to 400 pixels. The high-resolution images of the other datasets were kept unchanged, i.e. 512 × 512 for the HCI light fields and 625 × 434 for both EPFL and INRIA light fields. Unless otherwise specified, the low-resolution light fields were generated by blurring each high-resolution sub-aperture image with a Gaussian filter using a window size of 7 and standard deviation of 1.6, down-sampled to the desired resolution and up-scaled back to the target resolution using bi-cubic interpolation. Unless otherwise specified, the iterative back-projection refinement strategy was disabled to permit a fair comparison to the other state of the art super-resolution methods considered. We compare the performance of our system against the best performing methods found in our recent work [5], namely the CNN based light field super-resolution algorithm (LF-SRCNN) [11] and both linear subspace projection based methods, PCA+RR and BM+PCA+RR [5]. These methods were retrained using samples from the 98 training light fields mentioned above using training procedures explained in their respective papers. Moreover, given that the very deep super-resolution (VDSR) method [28] achieved state-of-the-art performance on single image super-resolution, we apply this method to restore every sub-aperture image independently. It is important to mention here that in our previous work we found that BM+PCA+RR significantly outperforms several other light field and single-image super-resolution algorithms including [8], [10], [27], [38], [39], [40], [41]. Due to space constraints we did not provide comparisons against the latter approaches.
The results in table 1 and table 2 compare these superresolution methods in terms of PSNR for magnification factors of ×2 and ×3 respectively. Moreover, the results in Figure 6 show the centre views of light fields restored using these methods when considering a magnification factor of ×3. The VDSR algorithm [28] achieves on average a PSNR gain of 0.33 dB over bicubic interpolation. One major limitation of VDSR is that it does not exploit the light field structure where each sub-aperture image is being restored independently. The PCA+RR algorithm [5] manages to restore more texture detail and is particularly effective to restore light fields with small disparities, which is the 4. HCI dataset: http://hci-lightfield.iwr.uni-heidelberg.de/ 5. Stanford dataset: http://lightfield.stanford.edu/ case of the INRIA light fields. This can be attributed to the fact that the PCA+RR does not compensate for disparities and therefore is not able to generalize to light fields containing disparities which were not considered during training, which is the case for the Stanford light fields. In fact the Stanford light fields restored using PCA+RR contain blocking artefacts. The BM+PCA+RR method [5] extends this method by aligning the patch-volumes using blockmatching. This results in a more generalized method that achieves average PSNR gains of 2.52 dB and 1.76 dB over bi-cubic on the INRIA and Stanford light fields respectively. Nevertheless, the light fields restored using this method may contain some artefacts especially near the edges.
The LF-SRCNN method [11], which uses deep learning to restore each sub-aperture image independently, was found to achieve a marginal gain over BM+PCA+RR (0.05 dB). While, light fields restored using LF-SRCNN generally contain less artefacts compared to BM+PCA+RR, they risk to restore light fields which are angularly incoherent. Nevertheless, our method achieves the best performance achieving an overall gain of 0.2 dB over the second-best performing algorithm LF-SRCNN. This performance gain is more evident in the subjective results illustrated in Figure  6 where the restored light fields are sharper and are visually more pleasing.
As mentioned in section 3.4, one problem with the proposed method is that the inverse warping is unable to perfectly restore the original disparities of the light field. Nevertheless, the results in tables 1, 2 and Figure 6 clearly show that our proposed method outperforms the other schemes even without the use of the iterative backprojection refinement strategy.
In order to fairly assess the contribution of iterative backprojection, we apply it as a post process for the two best performing methods, namely LF-SRCNN and our proposed scheme LR-LFSR. Tables 3 and 4 show the performance of the two best performing methods with and without iterative back projection as a post-processing step at magnification factors of ×2 and ×3 respectively. It can be immediately noticed that IBP significantly improves the quality of both methods. Nevertheless, our method which uses iterative back projection as a post-processing step achieves the best performance achieving PSNR gains of 0.41 dB and 0.31 dB at magnification factors of ×2 and ×3 respectively over LF-SRCNN followed by iterative back projection. It is important to notice that while the performance gain of our method over LF-SRCNN without IBP is around 0.23 dB and 0.12 dB at magnification factors of ×2 and ×3 respectively, this gain roughly doubles when both use IBP as a post processing step. This indicates that since LF-SRCNN processes each view independently, IBP only corrects inconsistencies between the low-resolution and restored light fields. Apart from this distortion, LR-LFSR-IBP corrects the distortions caused by the inverse warping process which provides light fields which are more visually pleasing and with smoother transitions across views. Supplementary multimedia files uploaded on ScholarOne show some sample restored light field. Supplementary material is available on the project's website 6 while the code of the LR-LFSR will be made 6. https://goo.gl/8DDsDi available upon publication.     the aligned light field gives an embedding in a lower dimensional space which is super-resolved using deep learning. All aligned views of the high-resolution light field can be reconstructed from the super-resolved embedding by simple linear combinations. These views are then inverse warped to restore the disparities of the original light field. Holes corresponding to dis-occlusions or cracks resulting from the inverse warping are filled in using a novel diffusion based inpainting algorithm which diffuses known pixels in the EPI along dominant orientations computed in the lowresolution EPI. Extensive simulations show that the proposed method manages to generalize well, i.e. manages to successfully restore light fields whose disparities are considerably different from those used during training. These results also show that our proposed method is competitive and most of the time superior to existing state-of-the-art light field super-resolution algorithms, including a recent approach which adopts deep learning to restore each sub-aperture image independently. One major limitation of the proposed scheme is that the inverse warping process is not able to restore the original disparities and produces some distortion caused by rounding errors. We proposed here to use the classical iterative back-projection as a post processing step. Simulation results clearly show the benefit of using IBP as a post processing of the super-resolved light field and demonstrate that the proposed method with IBP achieves the best performance, outperforming LF-SRCNN followed by IBP by 0.4 dB.