{'fraction_non_alphanumeric': 0.02934658005349637, 'fraction_numerical': 0.010049675200611388, 'mean_word_length': 4.791325514494357, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Hierarchical Coding Scheme for Glasses-free 3D Displays Based on Scalable Hybrid Layered Representation of Real-World Light Fields

This paper presents a novel hierarchical coding scheme for light fields based on transmittance patterns of low-rank multiplicative layers and Fourier disparity layers. The proposed scheme learns stacked multiplicative layers from subsets of light field views determined from different scanning orders. The multiplicative layers are optimized using a fast data-driven convolutional neural network (CNN). The spatial correlation in layer patterns is exploited with varying low ranks in factorization derived from singular value decomposition on a Krylov subspace. Further, encoding with HEVC efficiently removes intra-view and inter-view correlation in low-rank approximated layers. The initial subset of approximated decoded views from multiplicative representation is used to construct Fourier disparity layer (FDL) representation. The FDL model synthesizes second subset of views which is identified by a pre-defined hierarchical prediction order. The correlations between the prediction residue of synthesized views is further eliminated by encoding the residual signal. The set of views obtained from decoding the residual is employed in order to refine the FDL model and predict the next subset of views with improved accuracy. This hierarchical procedure is repeated until all light field views are encoded. The critical advantage of proposed hybrid layered representation and coding scheme is that it utilizes not just spatial and temporal redundancies, but efficiently exploits the strong intrinsic similarities among neighboring sub-aperture images in both horizontal and vertical directions as specified by different predication orders. Besides, the scheme is flexible to realize a range of multiple bitrates at the decoder within a single integrated system. The compression performance analyzed with real light field shows substantial bitrate savings, maintaining good reconstruction quality.


I. INTRODUCTION
The market of glasses-free displays is growing rapidly in recent years. Current generation autostereoscopic display techniques are seen as an alternative to stereoscopic 3D as it supports both stereopsis and motion parallax with many views for different viewing directions. However, technology has still not matured enough to simultaneously provide direction-dependent outputs without sacrificing the resolution in reconstructing the dense light fields. Emerging multilayered light field displays offer solution that provides continuous motion parallax, greater depth-of-field, and wider field-of-view [1]- [3]. A typical structure of a multi-layered display is shown in Fig. 1a layers stacked in front of a backlight. On each layer, the transmittance of pixels can be controlled independently by carrying out multiplicative operations on the layered patterns. Fig. 1b illustrates the light rays which pass through different combinations of pixels in stacked layers depending on the viewing directions. Layered displays accurately reproduce multi-view images or light field simultaneously with high resolution using just few light attenuating layers. Efficient representation and coding of light rays in multi-layered 3D displays is critical for its adaptation on different autostereoscopic platforms.
Existing light field coding algorithms are broadly categorized into approaches using the lenslet image [4]- [6] or subaperture images (SAIs). We can further classify compression methods exploiting geometry information [7]- [11], scene content information [12], view prediction based learning schemes [13]- [16] or methods considering light field data as a pseudo video sequence [17]- [20]. The approaches arranging SAIs as pseudo-temporal sequence are constrained to one-dimensional coding structure. Nonlinear correlation in horizontal and vertical directions among adjacent SAIs are not carefully considered in their coding models. The geometry-based or learning-based methods usually demand accurate mathematical model for parameter estimation or suffer from the primary limitation of low quality reconstruction. Their suitability is questionable for compactly encoding light fields with multiple bitrates dealing with layered structure of transmittance patterns in multi-layered displays.
In this paper, we propose a novel hierarchical coding scheme for light fields based on multiplicative layers [21] and Fourier disparity layers representation [22]. The input light field is divided into two subsets based on predefined Circular and Hierarchical prediction orders (Fig. 4). A data-driven convolutional neural network (CNN) is optimized to learn three multiplicative layers for each view subset. The key idea is to reduce dimensionality of stacked multiplicative layers using a randomized block Krylov singular value decomposition (BK-SVD) procedure [23]. Factorization derived from the BK-SVD efficiently exploits the high spatial correlation between multiplicative layers on Krylov subspaces and approximate light fields for varying low ranks. Encoding of these approximated layers using HEVC codec [24] further eliminates intra-view and inter-view redundancies. In the first stage, the proposed scheme approximates multiplicative layers of a target light field for multiple ranks and the quantization parameters (QPs). The view subsets are reconstructed from the decoded layers.
In the second stage, the entire approximated light field is processed in the Fourier domain following a hierarchical coding procedure. A Fourier Disparity Layer (FDL) calibration estimates disparity values and angular coordinates of each light field view [25]. These parameters provide additional information for FDL construction and view prediction. These parameters are transmitted to the decoder as metadata. We then split the approximated light field into subsets as identified with scanning and prediction orders. The first set of views is encoded and used for construction of the FDL representation. This FDL representation is used to synthesize the second subset. The remaining correlations between the prediction residue of synthesized views is further eliminated by encoding the residual signal using HEVC with inter coding mode. The set of views obtained from decoding the residual are employed in order to refine the FDL representation and predict the next subset of views with improved accuracy. This hierarchical procedure continues iterating until all light field views are coded. The critical advantages of the proposed hybrid layered representation and coding scheme are: • It efficiently exploits the strong intrinsic similarities in light field structure by approximating multiplicative layer patterns with varying low ranks. Moreover, it ensures inter-view prediction from adjacent views (both horizontally and vertically) that exhibit higher similarity. The results with different scanning orders demonstrates high compression performance on real world light field data. • It is flexible to realize a range of multiple bitrates within a single integrated system trained using few (two) convolutional neural networks. This critical characteristic of the proposed model complements existing light field coding systems or methods which support only specific bitrates during the compression. This would offer an adaptation of layered displays to support a variety of computational or multi-view auto-stereoscopic platforms by optimizing the bandwidth for a given target bitrate.

II. PROPOSED SCHEME
The overall workflow of our proposed representation and coding scheme is illustrated in Fig. 2. The scheme is divided into three primary components. In COMPONENT I, input light field images are divided into subsets, specified by different view prediction orders. The COMPONENT I is further expanded into three blocks as depicted in Fig. 3. This component performs low-rank approximation of different subsets of the input light field. The BLOCK I represents a CNN that converts the light field views of the input subsets into three multiplicative layers [21]. In BLOCK II, the intrinsic redundancy present in subset views is removed by exploiting the hidden low-rank structure of multiplicative layers on a Krylov subspace for various ranks [23]. HEVC [24] encoding of low-rank approximated multiplicative layers further eliminates the inter-frame and intra-frame redundancies in low-rank approximated layers. In BLOCK III, the approximated views of the subset are reconstructed from the decoded layers. The entire approximated light field is obtained at the end of COMPONENT I.
The approximated subset of decoded views obtained from COMPONENT I following multiplicative layer representation is used to construct Fourier disparity layer (FDL) [22] representation in COMPONENT II. The processing of approximated light field in COMPONENT II of our scheme is shown in Fig. 2a. There exist strong intrinsic similarities between neighboring sub-aperture images in both horizontal and vertical directions as specified by different predication orders. The main functionality of COMPONENT II is to further eliminate inter-view redundancy from low-rank approximated adjacent views as possible determined from different scanning orders. Following a hierarchical procedure, the FDL representation reconstructs the light field iteratively.
The angular coordinates and disparity values of each view of the low-rank approximated light field (at different ranks and QPs) are determined through FDL calibration and directly transmitted to the decoder (COMPONENT III) as metadata [25]. The approximated light field is divided into two subsets specified by prediction order. Subset 1 aids in constructing the FDL representation, which is further used to synthesize the second subset of views. The encoding of the residual signal is performed to account for the correlations in the prediction residue of synthesized Subset 2. A more accurate FDL representation is built from previously encoded subsets. Thus, there is an iterative refinement of the FDL representation in COMPONENT II until all the approximated light field views are encoded. Our decoding scheme is depicted in the COMPONENT III (Fig. 2b). The angular coordinates and disparity values of the low-rank approximated light field, along with the encoded bitstreams of approximated Subset 1 and Subset 2, are utilized for the final reconstruction.
A. Approximation of light field at different ranks COMPONENT I illustrates the division of input light field into two subsets and their low-rank approximation in the Block-Krylov subspace. As shown in Fig. 3, this step involves three blocks that are described in the following subsections.
1) View Subsets of Light Field: Dib et al. [25] recommend two best scanning orders for synthesizing and coding view subsets, Circular-2 (C 2 ) and Hierarchical-2 (H 2 ). For a 9×9 light field, the C 2 pattern contains two subsets. The first subset forms a circular pattern, and the remaining views constitute the second subset. The H 2 configuration follows a more simple division of subsets, where each subset comprises of alternate views. The exact coding orders of each subset of the C 2 and H 2 patterns are shown in Fig. 4. In both patterns, the views are ordered beginning with the central view and spiraling out towards the periphery for each subset. Our proposed workflow ( Fig. 2) begins with partitioning of the input light field into two subsets based on the C 2 or H 2 pattern, which are then approximated by BK-SVD.
2) Light Field Views to Stacked Multiplicative Layers: The four-dimensional light field L(u, v, s, t) is parameterized by angular coordinates (u, v) and spatial coordinates (s, t) [26], [27]. If the (s, t) plane is located at depth z, the light ray will have coordinates (u + zs, v + zt) (Fig. 5).
Multiplicative layers [21] are light attenuating panels that are stacked in evenly spaced intervals in front of a backlight as shown in Fig. 1. The transmittance of a multiplicative layer M z is given by M z (u, v) and a light ray emitted from these layers will have intensity where, z is the disparity among directional views and Z denotes a set of z. In our work, we considered the two subsets of the light field as L 1 (u, v, s, t) and L 2 (u, v, s, t). Three multiplicative layers located at Z = {−1, 0, 1} for each of the two input subsets are obtained. These multiplicative layer patterns are enough to efficiently represent each subset entirely. We performed the optimization of the three layer patterns for each light field by training two CNNs. The final trained network can covert each of the the light field subsets into three optimized output multiplicative layers as depicted in BLOCK I of Fig. 3.

3) Low Rank Representation and Coding of Stacked Multiplicative Layers on Krylov Subspace:
The key idea of the proposed scheme is to remove the intrinsic redundancy in light field multiplicative layers by compactly representing them on a Krylov subspace. The randomised Block-Krylov SVD introduced by Cameron Musco and Christopher Musco [23] can optimally achieve the low-rank approximation of a matrix within (1 + ) of optimal for spectral norm error. The algorithm quickly converges inÕ( 1 √ ) iterations for any matrix.
In our proposed scheme, we denote each multiplicative layer pattern produced by the CNN as M z ∈ R m×n×3 , where z ∈ {−1, 0, 1}. The red, green, and blue colour channels of the layer z are denoted as M r z , M g z , and M b z respectively. We constructed three matrices B ch ∈ R 3m×n , ch ∈ {r, g, b}, as The intrinsic redundancies in multiplicative layers of the light field can be effectively removed by following low-rank BK-SVD approximation in a Krylov subspace for each B ch , ch ∈ {r, g, b}. For simplicity, we will denote B ch as B henceforth. Our present formulation of low-rank light field layer approximation seeks to find a subspace that closely captures the variance of B's top singular vectors and avoid the gap dependence in singular values. We target spectral norm low-rank approximation error of B, defined as where, W is a rank k matrix with orthonormal columns. In a rank k approximation, only the top k singular vectors of B are considered relevant and the spectral norm guarantee ensures that W W T B recovers B up to a threshold . We start with a random matrix Π ∼ N (0, 1) d×k and perform Block Krylov Iteration by working with the Krylov subspace, We constructed p q (B)Π for any polynomial p q (·) of degree q by working with Krylov subspace K. The approximation of matrix B done by projecting it onto the span of p q (B)Π, is similar to the best k rank approximation of B lying in the span of the Krylov space K. Thus, the nearly optimal low-rank approximation of B can be achieved by finding this best rank k approximation. To ensure that the best spectral norm error low-rank approximation lies in the span of K, we orthonormalized the columns of K to obtain Q ∈ R c×qk using the QR decomposition method [28]. We take SVD of matrix S = Q T BB T Q where, S ∈ R qk×qk , for faster computation and accuracy. The rank k approximation of B is matrix W , which is obtained as where,Ū k is set to be the top k singular vectors of S. Consequently, the rank k block Krylov approximation of matrices B r , B g and B b are W r , W g , and W b respectively. The W ch ∈ R x×y for every colour channel ch.
To obtain the approximated layers M z , z ∈ {−1, 0, 1}, we extract their colour channels from the approximated W ch matrices by sectioning out the rows uniformly as The red, green, and blue colour channels are then combined to form each approximated layer M −1 , M 0 , and M 1 . Thus, factorization derived from the BK-SVD exploits the spatial correlation in multiplicative layers of the subset views for varying low ranks. The three block Krylov approximated layers for each subset are subsequently encoded into a bitstream using HEVC [24] for various QPs to further eliminate inter and intra layer redundancies in the low-rank approximated representation. We have shown this low-rank representation and coding of stacked multiplicative layers on Krylov subspace in BLOCK II of Fig. 3.

4) Decoding and Reconstruction of the Light Field Subsets:
The decoding procedure of compressed layers and the reconstruction of light field subset is illustrated in BLOCK III of Fig. 3. We decoded three layers from the bitstream and converted them into the RGB format. The decoded layers are denoted asM z , where z ∈ {−1, 0, 1} depicts the depth of the layers. To reconstruct the views of each subset, we considered the integers s * , t * (which depend on the number of views in the subset). The view I (s * ,t * ) is obtained by translating the decoded layers toM z and performing an element-wise product of the colour channels of the translated layers. For a particular view (s * , t * ), the translation of every z th layerM z , toM z is carried out as Thus, the three translated layers for every viewpoint (s * , t * ) areM An element-wise product of each colour channel ch ∈ {r, g, b}, of the translated layers gives the corresponding colour channel of the subset view.
The combined red, green, and blue colour channels output the reconstructed light field subset at the viewpoint (s * , t * ) as I (s * ,t * ) . Subsets 1 and 2 are then merged according to the Circular-2 (C 2 ) or Hierarchical-2 (H 2 ) ordering. COMPONENT I of the proposed scheme, hence, exploits the spatial correlation in multiplicative layers of the subset views for varying low ranks. Inter and intra layer redundancies in the low-rank approximated representation are removed as well. We further compress the light field by eliminating intrinsic similarities among neighboring views of the C 2 or H 2 patterns (in both horizontal and vertical directions) using the Fourier Disparity Layers representation.

B. Fourier Disparity Layers (FDL) Representation
The Fourier Disparity Layers representation [22] samples the input BK-SVD approximated light field in the disparity dimension by decomposing it as a discrete sum of layers. The decomposition is carried out in the Fourier domain and FDL representation is constructed by a regularized least square regression performed independently at each spatial frequency. COMPONENT II in Fig. 2a summarises the use of FDL and encoding of low-rank approximated subsets. We highlight the FDL calibration, subset view synthesis and prediction in the following subsections.
1) FDL Calibration: Construction of the FDL requires the angular coordinates u j of the input approximated light field views and the disparity values d k of the layers. These parameters are found by minimizing over all frequency components f s of each view [25]. The view L uo of the approximated light field at angular coordinate u o can be defined as L uo (s) = L(s, u o ) and by using n disparity values {d k } k∈ [1,n] , the Fourier transform of L uo can be written aŝ where, f s is the spatial domain frequency. The Fourier transform of the central light field view obtained by only considering a specific spatial region of disparity d k is given by eachL k . By computing the Fourier transforms of all m approximated light field views asL uj (j ∈ [1, m]), the FDL representation can be learnt by solving a linear regression problem for each frequency f s . The problem is formulated by Ax = b with Tikhonov regularization where A ∈ R m×n , x ∈ R n×1 and b ∈ R m×1 . Elements of matrix A are A jk = e +2iπuj d k fs , x contains the Fourier coefficients of disparity layers x k =L k (f s ) and Fourier coefficients of the j th input view, b j =L uj (f s ) are contained in b.
2) FDL View Synthesis and Prediction: Any subset view L uo at angular coordinate u o can be synthesized in the Fourier domain with the FDL and their disparities d k using Eq. 9. The C 2 and H 2 prediction orders divide the lowrank approximated light field into two subsets. The first approximated subset is encoded and aids in constructing the FDL representation. The second approximated subset views are then synthesized using the FDL and the residual signal further refines the representation through iterative encoding of each view. We send the bitstreams of each approximated subsets along with the encoded metadata of parameters to the decoding end, COMPONENT III of Fig. 2b.

III. RESULTS AND ANALYSIS
The performance of the proposed compression scheme was evaluated on real light fields captured by plenoptic cameras. We experimented with Bikes, Fountain & Vincent 2, and Stone pillars outside light field datasets from the EPFL Lightfield JPEG Pleno database [29]. Fig. 6 shows the central views of the chosen light field images. The raw plenoptic images were extracted into 15×15 sub-aperture views using the Matlab Light field toolbox [30]. The C 2 and H 2 coding patterns were constructed from the inner 9×9 views for our tests.

A. Experimental Settings and Implementation Details
The proposed scheme was implemented on a single highend HP OMEN X 15-DG0018TX system with 9th Gen i7-9750H, 16 GB RAM, RTX 2080 8 GB Graphics, and Windows 10 operating system. We optimized the multiplicative layer patterns for each subset using CNN with twenty 2-D convolutional layers stacked in a sequence. The CNN models were trained for 25 epochs at a learning rate of 0.001 and a batch size of 15. The entire network was implemented using the Python-based framework, Chainer (version 7.7.0).
The resultant output multiplicative layers for each subset of the C 2 and H 2 patterns were obtained from the trained CNN models. We rearranged the colour channels of these multiplicative layers as described in section II-A.3 and then applied BK-SVD for ranks 4,8,16,28,44, 52 and 60. The approximated matrices were then arranged back into layers and compressed using HEVC HM 11.0 (BLOCK II of COMPONENT I). We used quantization parameters 2, 6, 10, 14, 20, 26, and 38 to test both high and low bitrate cases of HEVC. The decoding and reconstruction of BK-SVD approximated subsets for both C 2 and H 2 patterns was performed.
Low-rank approximated subsets were then utilized to form the FDL representation of the light fields. The number of layers in the FDL method was fixed to n = 30. Views in approximated Subset 1 construct the initial FDL representation. Views of approximated Subset 2 are predicted from this FDL representation and the residues iteratively refine the FDL. We used HEVC HM 16.0 to perform the encoding in COMPONENT II, for quantization parameters 2, 6, 10, 14, 20, 26 and 38.

B. Results and Comparative Analysis
We compared the proposed scheme with the direct compression of light fields using FDL by Dib et al. [25]. The proposed coding scheme outperforms the anchor by large margins. The total number of bytes taken by our approach is comparatively far lesser than the work by Dib et al. [25] for all ranks and QPs. These corresponding results are highlighted in Table I for the (C 2 ) pattern and in Table II for (H 2 ) pattern. The bitrate vs YUV-PSNR graphs of the three datasets are illustrated in Fig. 7. For both the Circular- 2 and Hierarchical-2 patterns, the proposed scheme has significantly better rate-distortion results. Further, we analysed bitrate reduction (BD-rate) of the proposed scheme with respect to Dib et al. [25] using the Bjontgaard metric [31]. The average percent difference in rate change was estimated over a range of QPs for the seven chosen ranks. A comparison of the percentage of bitrate savings of our proposed coding scheme with respect to the anchor method for the three chosen light field datasets is shown in Table III

IV. CONCLUSION
We have proposed a novel hierarchical coding scheme for light fields based on transmittance patterns of low-rank multiplicative layers and Fourier disparity layers. Typical pseudo sequence-based light field compression schemes [17]- [20] do not efficiently consider the similarities between horizontal and vertical views of a light field. Our proposed scheme not only exploits the spatial correlation in multiplicative layers of the subset views for varying low ranks, but also removes the temporal inter and intra layer redundancies in the low-rank approximated representation of the view subsets. The approximated light field is further compressed by eliminating intrinsic similarities among neighboring views of the Circular-2 or Hierarchical-2 patterns using Fourier Disparity Layers representation. This integrated compression achieves excellent bitrate reductions without compromising the quality of the reconstructed light field.
Our scheme offers flexibility to cover a range of multiple bitrates using just two trained CNNs to obtain a layered representation of the light field subsets. This critical feature sets our proposed scheme apart from other existing light field coding methods, which usually train a system (or network) to support only specific bitrates during the compression. Besides, existing coding approaches are not explicitly designed to target layered displays and are usually only classified to work for lenslet-based formats or sub-aperture images based pseudo-sequence light field representation. Our coding model can complement existing layered light field displays and is also adaptable to other mobile, head-mounted [32], tabletop [33] or autostereoscopic displays.
We plan to extend the proposed idea to light field displays with more than three light attenuating layers in our future work. Proof-of-concept experiments with our scheme also pave the way to form a more profound understanding in the rank-analysis of a light-field using other mathematically valid tensor-based models [2], [34] and coded mask cameras [35].
We also aim to verify our scheme with physical light field display hardware. Further, we wish to adapt the proposed scheme with display device availability and optimize the bandwidth for a target bitrate. This would enable deploying the concepts of layered displays on different display platforms that deliver 3D contents with the limited hardware resources, thus, best meeting the viewers' preferences for depth impression or visual comfort.