{'fraction_non_alphanumeric': 0.02957304133774722, 'fraction_numerical': 0.012545624310330192, 'mean_word_length': 5.116926272066459, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CapsField: Light Field-based Face and Expression Recognition in the Wild using Capsule Routing

Light field (LF) cameras provide rich spatio-angular visual representations by sensing the visual scene from multiple perspectives and have recently emerged as a promising technology to boost the performance of human-machine systems such as biometrics and affective computing. Despite the significant success of LF representation for constrained facial image analysis, this technology has never been used for face and expression recognition in the wild. In this context, this paper proposes a new deep face and expression recognition solution, called CapsField, based on a convolutional neural network and an additional capsule network that utilizes dynamic routing to learn hierarchical relations between capsules. CapsField extracts the spatial features from facial images and learns the angular part-whole relations for a selected set of 2D sub-aperture images rendered from each LF image. To analyze the performance of the proposed solution in the wild, the first in the wild LF face dataset, along with a new complementary constrained face dataset captured from the same subjects recorded earlier have been captured and are made available. A subset of the in the wild dataset contains facial images with different expressions, annotated for usage in the context of face expression recognition tests. An extensive performance assessment study using the new datasets has been conducted for the proposed and relevant prior solutions, showing that the CapsField proposed solution achieves superior performance for both face and expression recognition tasks when compared to the state-of-the-art.


I. INTRODUCTION
Face is the most common characteristic used by humans to perform personal identity [1], [2] and expression recognition [3]. Following the emergence of the first automatic facial image analysis system in the late 1960's [4], this field has attracted considerable research efforts leading to incredible progresses. Despite the recent advances in face and expression recognition, mostly thanks to the advancements in deep neural networks [5], [6], [7], highly accurate recognition results are still not achievable for some specific conditions. This is mostly the case when poor quality data is available, for instance due to the capture of face images in uncontrolled settings, commonly referred to as face recognition in the wild, where facial images are affected by multiple variations, notably in resolution, background, expression, pose, illumination, and occlusions, among others [8].
The emergence of new imaging sensors has opened new frontiers and also brought new challenges for face image analysis systems [9]. Lenslet light field cameras, hereafter referred only as Light Field (LF) cameras, have recently come into prominence as they are able to simultaneously capture the intensity of light rays coming from multiple directions in space [10], [11]. An LF camera "sees" the visual scene from multiple angles or perspectives, allowing to render a set of 2D Sub-Aperture (SA) images, each corresponding to a 2D image for a different viewpoint of the observed scene. The set of rendered 2D SA images forms a multi-view SA array which offers intraview/spatial (within each view) and inter-view/angular (across views) correlations that can be effectively exploited for various visual analysis tasks, notably facial image analysis [12]. In this context, the main challenge of LF-based face and emotion recognition is how to exploit both the intra-view and inter-view relationships to improve the performance of the recognition solutions. This is the recognition paradigm considered in this paper. The LF-based face and emotion recognition can also be formulated as an image set based recognition problem [13], where matching is performed between image sets instead of single images.
Despite the significant success of the LF representation for constrained face recognition [14], [15], [16], [17], [18], [19], [20], [21] and expression recognition [22], [23], this type of visual representation has never been used for face and expression recognition in the wild, where the added value of the LF information may be more critical since the conditions are more extreme and unconstrained. In this context, this paper proposes to exploit the richer spatio-angular representation acquired with a LF camera, to improve the performance of face and expression recognition systems in the wild. In this paper, 'in the wild' refers to facial images captured under several unconstrained acquisition conditions such as random lighting conditions, shading, backgrounds, environments, poses, distances, etc., with minimal or no user cooperation. The key contributions and technical novelty of this paper in the context of LF face and expression recognition in the wild can be described as follows: • CapsField Solution: A new deep learning solution, called CapsField, is proposed for both face and expression recognition in the wild. CapsField combines convolutional and capsule networks [24] for exploiting not only spatial information but also the angular information available in LF images. A few solutions based on capsule networks have recently been proposed in the context of facial image analysis, including age and gender [25], and expression [26] recognition, showing that capsule networks are well suited to deal with face pose variations. CapsField goes one step further, for the first time exploiting dynamic routing between capsules for learning the angular variations across the different viewpoints captured in LF images. Additionally, CapsField assigns higher weights to the more relevant features while ignoring the spurious dimensions, to increase its robustness against face appearance and environmental changes. • In the wild face data collection: The captured dataset, named Light Field Faces in the Wild (LFFW), addresses a major weakness in the face and expression recognition research domains, which is the lack of publicly available LF facial images captured in the wild. To the best of the authors' knowledge, no similar dataset is available, even for private usage. The proposed LFFW dataset consists of LF images captured under several unconstrained acquisition variations, in both indoor and outdoor environments, at different locations, and from different distances. During the acquisition, there were no pre-defined protocol or any stimulus used to induce the desired expression. Additionally, a subset of the LFFW dataset includes facial images with different expressions, which has also been annotated to be used in the context of LF-based facial expression recognition. • Constrained face data collection: This paper also makes available a second LF dataset, called Light Field Face Constrained (LFFC), acquired in constrained conditions, with the same subjects present in LFFW. The complementary LFFC dataset was acquired between 1 day and 3 years prior to the LFFW images, allowing to study the constrained versus in the wild cross-dataset generalization ability of face recognition solutions. The two complementary face datasets (LFFW and LFFC) offer a unique set of facial data, facilitating a wide range of research possibilities related to face analysis. Both datasets, along with the annotation metadata, will be publicly available to the research community. • Benchmarking: Finally, this paper provides an extensive performance assessment of the proposed face and expression recognition solutions regarding the most competitive state-of-the-art solutions. As new datasets along with novel evaluation protocols are proposed in this paper, the authors had to re-implemented all the benchmarking solutions. The obtained results show that the proposed CapsField solution outperforms the state-of-the-art solutions by a significant margin for both face and expression recognition tasks, for the new face datasets. The rest of this paper is organized as follows: Section II briefly reviews the basic concepts and the added value of LF imaging for face recognition and provides a comprehensive review of recent advances in LF-based face and expression recognition solutions and datasets to better understand the technological landscape in this field. The proposed CapsField solution is presented in Section III and the new face datasets are described in Section IV. Section V presents an extensive performance evaluation for the proposed and state-of-theart solutions using a novel evaluation framework addressing varied and challenging recognition tasks. Finally, Section VI concludes the paper.
II. BACKGROUND AND RELATED WORK This section starts with an overview of light field imaging basics, followed by a LF-based face and expression recog- nition state-of-the-art review, and finally overviews the main face datasets available, including the LF ones.

A. Light Field Imaging Basics
LF imaging has been researched for more than a century [27]. In 1908, Lippmann used small and closely spaced circular lenses to record different perspectives of a scene, so that an image could be observed from a selected perspective through an array of lenses [28]. In 1936, Gershun coined the term light field, referring to the amount of light traveling in all directions through every point in space [29]. In 1991, Adelson proposed the Plenoptic function P(x,y,z,t,λ,θ,φ), describing the information carried by light rays at any point in 3D space (x,y,z), for all directions (θ,φ) and wavelengths (λ), along time (t). The static 4D LF [11], L(x,y,u,v), also known as Lumigraph [30], proposed a representation based on the intersection points of the light rays with two parallel planes [31].
For capturing LF images, the type of visual data adopted in this work, two main practical setups are currently available, notably using a high density array of cameras or a lenslet LF camera. In a lenslet LF camera, as illustrated in Figure 1, a micro-lens array, consisting of a set of micro-lenses, is placed at the focal plane of the main lens at a given distance from the photo sensor [10]. This allows for the incoming light to be split based on its direction, and projected onto the photo sensor area. Each micro-lens can be thought of as a small camera, capturing light rays from a specific point in the scene, thus acquiring a so-called micro-image [32]. A sample of a full set of LF color demosaiced micro-images is shown in Figure 2(a). A set of SA images, corresponding to different viewpoints of the scene can then be rendered from the micro-images. The set of rendered SA images can be considered as a multiview SA array. Examples of a multi-view SA array and the corresponding 2D central SA image are shown in Figure 2(b) and Figure 2(c), respectively. Since each 2D SA image 'sees' the scene from a slightly different viewpoint, the full set of 2D images provides angular variations of the scene, which is a distinctive characteristic of LF imaging. The LFFW dataset proposed in this paper was captured with a Lytro Illum camera [33], which multi-view SA array includes 15 × 15 SA images, each SA image with a spatial resolution of 625 × 434 pixels. In this context, there are two different types of correlation available within a multi-view SA array: i) the spatial intraview correlation within views; and ii) the angular inter-view correlation between different views. Therefore, LF images open the possibility for spatio-angular information to be exploited, allowing to boost the face and expression recognition performance by: i) a posteriori refocusing to improve the quality of the face region(s) that may be out-of-focus during acquisition; ii) exploiting the disparity for representing variations associated with different light directions captured in LF images; and iii) exploiting depth for providing geometric information about the position and shape of facial components.

B. Light Field-based Face Recognition Solutions
Recently, several face recognition solutions using LF images have been proposed. This section reviews this category of face recognition solutions, which are able to exploit the availability of LF sensors and data. The main characteristics of the most notable recognition solutions in this area are summarized in Table I, including the adopted feature extraction approach, classifier, exploited LF capability (as introduced in Section II), the type of LF data, and the datasets used by each solution for reporting performance results. For comparison, Table I also  includes the characteristics of the CapsField solution proposed  in this paper. The solutions summarized in the table are  reviewed here, notably grouped based on the feature extraction  approaches considered. 1) Visual Descriptor-based Solutions: The first group of visual descriptor-based solutions relies on the a posteriori refocusing capability to improve the quality of out-of-focus facial regions. In [14] a wavelet energy method is proposed for selecting the best focus plane. Then, the Local Binary Pattern (LBP) descriptor is applied to extract features and a Nearest Neighbor (NN) classifier is used for performing recognition. Another solution uses a resolution enhancement scheme [15] based on a discrete wavelet transform, to capture the components with highest frequencies from different refocused images. This solution creates after an all-in-focus face image to be input to an LBP descriptor. In [34], the recognition of multiple faces placed at different distances with an all-in-focus image rendered from the LF is investigated. An LBP descriptor is then applied for feature extraction and a Sparse Reconstruction Classifier (SRC) is used for classification. In [16], the LFbased face recognition solution renders a set of refocused images using two different approaches. The first selects the best refocused image for recognition while the second combines the refocused images for creating a super-resolved image. Different local descriptors including Histogram of Oriented Gradient (HOG), LBP, Center-Symmetric LBP (CSLBP), and Binarized Statistical Image Features (BSIF) are used to extract features for classification.
The second group of visual descriptor-based solutions exploits the depth information that can be estimated from an LF image. These solutions explore the geometry of the facial components, notably their positions and shapes. In [35], a depth map is exploited to extract discriminative HOG features used as input to a linear Support Vector Machine (SVM) for classification.
The last group of visual descriptor-based solutions relies on exploiting the LF disparity information. In [17], a face recognition solution is proposed based on a visual descriptor named Light Field LBP (LFLBP), exploiting the richer spatioangular LF information. LFLBP has two main components, the spatial LBP and the angular LBP, capturing not only the usual spatial information but also the angular information available in a set of SA images. Another solution is proposed in [18] based on a the Light Field Histogram of Gradients (LFHG) descriptor, which fuses the conventional HOG with a Light Field Histogram of Disparity Gradients (LFHDG) descriptor, to consider both the orientation and magnitude information.
2) Deep Learning-based Solutions: Recognizing the importance of deep learning in biometric recognition, an LF face recognition solution has been designed based on a VGG 2D+Disparity+Depth (VGG-D3) fused deep descriptor [19]. The VGG-D3 description is formed by concatenating descriptions extracted from 2D images as well as disparity and depth maps using the VGG-16 descriptor trained on the VGG-Face 1 dataset [36]. This was the first solution to adopt a fused deep CNN network to exploit the complementary LF information, including 2D texture and the corresponding disparity and depth maps. Another face recognition solution is based on a doubledeep descriptor, so-called VGG + Conventional Long Short Term Memory (Conv-LSTM) [20]. This solution exploits the multi-perspective LF information While the VGG-D3 [19] processes only the information available in the 2D central SA image. The double-deep descriptor extracts spatio-angular LF dependencies using VGG and LSTM networks in sequence, to provide a more powerful description for face recognition. The solution in [21] proposes three novel LSTM cell architectures to jointly learn from LF horizontal and vertical parallaxes. The three cell architectures perform: i) Gate-Level Fusion LSTM (GLF-LSTM); ii) State-Level Fusion LSTM (SLF-LSTM); and iii) Sequential Learning LSTM (SeqL-LSTM). These architectures create richer spatio-angular LF descriptions for face recognition and have been integrated into an end-to-end framework, where a VGG network feeds the LSTM networks composed of the novel LSTM cell architectures.
In summary, given the nature of the available face recognition solutions and the way they were assessed, notably using different datasets and evaluation protocols, it is difficult to compare the various solutions and even more difficult to estimate how these solutions would perform in unconstrained operational conditions. Since more recent research has shifted towards in the wild face recognition, this paper addresses this  [34] 2013 Visual Descriptor LBP SCR A Posteriori Refocusing LF 2D Rendered LiFFID Face-Iris MF LF [16] 2016 Visual Descriptor HOG; LBP; BSIF SRC A Posteriori Refocusing 2D Rendered LiFFID DM LF [35] 2016 Visual Descriptor LFHOG SVM Depth Computation M-V SA Array Private LFLBP [17] 2017 Visual Descriptor LFLBP SVM Disparity Exploitation M-V SA Array LFFD LFHG [18] 2018 Visual Descriptor HOG; LFHDG SVM Disparity Exploitation M-V SA Array LFFD VGG-D3 [19] 2018 shortcoming by presenting an extensive benchmarking study, performed on the LFFW dataset proposed in this paper.

C. Light Field-based Expression Recognition Solutions
There are currently two solutions available for exploiting additional LF information for face expression recognition. Inspired by [20], a deep learning spatio-angular fusion framework was adopted in [22], to model both the spatial and angular information with VGG-16 and LSTM recurrent networks. In [23], the deep framework proposed in [22] was extended by exploring both forward and backward angular relations using bidirectional LSTM. Additionally, an attention mechanism selectively focuses on the most important spatio-angular features, thus learning the final features more effectively.

D. Sampling Schemes and Fusion Strategies
In order to exploit the angular dependencies between LF views, the views should be arranged in the form of a pseudovideo sequence to feed an LSTM network. In [20], 11 different sampling schemes were considered to select and scan the set of SA images, notably varying their number, position, and scanning order. The results showed that capturing angular information along the horizontal and vertical directions leads to better average performance when compared to other sampling schemes, notably due to the consideration of larger disparities. The results also showed that the score-level fusion of horizontal and vertical angular classification models leads to additional improvements.
An alternative approach to the score-level fusion strategy is to jointly learn the horizontal and vertical view sequences, as done by the GLF-LSTM and SLF-LSTM cell architectures proposed in [21] (introduced in Section II.B). Additionally, there are other alternative solutions, not originally designed for the same task, but which are flexible enough to be adopted for simultaneously learning from the horizontal and vertical view sequences. Examples are the Spatio-Temporal LSTM (ST-LSTM) [37] and the Dual-Sequence LSTM (DS-LSTM) cell architecture [38], which were originally designed to fuse different sequences for activity and speech recognition, respectively. These alternative LSTM cell architectures will be considered for benchmarking purposes, when comparing to the proposed solution in this paper.

E. Face Datasets: Status Quo
There are currently over 150 publicly available face datasets [39]. The characteristics of the more notable face datasets are presented in Table II, highlighting variations in terms of acquisition conditions, expressions, poses, and occlusions. For easier reading, the datasets are sorted according to their release date. For comparison, Table II also includes the characteristics of the LFFW and LFFC datasets proposed in this paper.
Since LF imaging is a relatively new technology, to the best of the authors' knowledge, only two face datasets have been made publicly available. The Light Field Face and Iris Database (LiFFID) [16] was the first face dataset to include images acquired with a LF camera for facial recognition purposes. It includes a set of 2D greyscale images, focused at different depths, rendered from the LF content acquired using a first generation, lower resolution, Lytro lenslet camera. However, LiFFID does not include the raw LF images, which is a major limitation for many research applications. The IST-EURECOM Light Field Face Database (LFFD) [50] was the second LF face dataset made available. This dataset includes raw and rendered data from 100 subjects, with 20 LF samples per person, where each LF sample includes 225 2D SA images, captured by a second generation Lytro ILLUM lenslet camera, in two separate acquisition sessions. The images are captured in a controlled acquisition setup with different facial variations, referring to application scenarios where the subjects present themselves to a fixed camera with a uniform background.
As can be observed from Table II, the more recent face datasets have shifted towards the 'in the wild' scenarios to consider more challenging variations. Nevertheless, since the two previously available LF face datasets were both collected in constrained conditions, it is not possible to assess the performance of LF-based face recognition solutions for unconstrained setups. The proposed LFFW dataset addresses this shortcoming by including 1908 LF images, corresponding to 429,300 2D images, from 53 subjects, thus providing a largescale publicly available LF face dataset captured in the wild. Since this dataset will be publicly available, it may be used as the basis for the future design, validation, and assessment of LF-based face recognition systems. The proposed LFFC dataset complements the LFFW dataset by including 1060 LF and 238,500 facial 2D images from the same 53 subjects,  [40] 1998 ≈ 4k No Color 2D FERET [41] 2003 ≈ 14k No Gray/Color 2D LFW [42] 2007 ≈ 13k Yes Color 2D Multi-PIE [43] 2009 ≈ 750k No Color 2D MOBIO [44] 2010 ≈ 30k Yes Color 2D YouTube Faces [45] 2011 ≈ 606k Yes Color 2D Video SCface [46] 2011 ≈ 4k No Color/Infra. 2D; Infra. BU-3DFE [47] 2013 ≈ 370k No Color 2D+Depth Kinect Face DB [48] 2014 ≈ 2k No Color 2D+Depth PIPA [49] 2015 ≈ 63k Yes Color 2D VGG-Face 1 [36] 2015 ≈ 2.6 M Yes Color 2D LiFFID [16] 2016 ≈ 107k No Gray LF LFFD [50] 2016 ≈ 900k No Color LF MegaFace [51] 2017 ≈ 4.7M Yes Color 2D DFW [52] 2018 ≈ 11k Yes Color 2D VGG-Face 2 [53] 2018 Color LF captured between 1 day and 3 years before the corresponding LFFW acquisition. It must be highlighted that while the LFFC acquisition conditions are similar to the previously available IST-EURECOM LFFD dataset [50], the subjects are partly different. The newly proposed LFFW and LFFC datasets offer from now on a unique publicly available data platform as this dataset combination provides precious information for the face recognition community, for instance to study aging effects and cross-dataset generalization performance of different analysis solutions.

III. CAPSFIELD SOLUTION
This section presents the proposed face and expression recognition solution, exploiting the spatio-angular information available in LF images by using a combination of two deep learning networks.

A. Problem Formulation
Given a dataset D of LF face images, where each LF image LF i,j has been captured from a known face/expression with label x i ∈ {x 1 , x 2 , . . . , x N } and a set of viewing angles θ j ∈ {θ 1 , θ 2 , . . . , θ V }, the tasks of face or expression recognition for a probe face sequence LF probe can be defined as: where P r(x i |LF probe , D) is the probability of LF image LF probe belonging to label x i .

B. Solution Intuition and Overview
An LF image offers intra-view/spatial (within each view) and inter-view/angular (across views) information that can be used for improving face and expression recognition performance. Spatial features can be learned from each view using powerful CNN architectures. The inter-view information can be explored by combining the spatial features of individual views, for instance using feature level fusion, and feeding the result to a fully connected layer and then a classifier. However, the fully connected layer merely applies a weight vector to the concatenated feature vector, applying a non-linear function to the scalar output of a linear filter. This approach considers the concatenated features as a whole, not allowing to learn the relative positions of relevant face components in each view or across the different viewpoints, which is the main problem addressed in this paper.
A capsule network includes layers of capsules, each consisting of a group of neurons representing different properties of the input. The mechanism to learn interactions between capsules is performed with routing algorithms such as dynamic routing [24]. Capsules can share the knowledge across locations using a pose matrix that represents the relation between parts of an object and the whole object, thus making capsules capable of dealing with viewpoint changes. Accordingly, the activation of a capsule is also based on a comparison between multiple incoming pose predictions. Capsule networks have recently been adopted in the context of face analysis tasks, most notably in age and gender classification [25], and action detection [26]. This paper proposes the combination of a capsule network with a CNN to, respectively, learn the available inter-view and intra-view relations available in an LF image for face and expression recognition. In this context, the novel CapsField solution not only learns viewpoint-invariant relations between different parts of a face and the whole face in each view, but also it learns the relations between the different views along the whole multi-view sequence. A capsule network is also capable of learning feature importance by assigning higher weights to the more relevant features, while ignoring the spurious dimensions [54]. This functionality can act as an attention mechanism in the proposed CapsField solution, selectively focusing on the most important angular features making the solution more robust to changes in appearance and environment.
Following the motivation and intuition described above, the proposed CapsField solution integrates three sub-networks that are applied to the horizontal and vertical views to predict a classification lable, L, as follows:  where f H CN N and f V CN N are CNN feature extraction subnetworks, respectively, applied to horizontal and vertical spatial SA sequences; these sub-networks are composed of multiple convolutional, pooling, and fully connected layers. f H CAP S and f V CAP S are capsule routing sub-networks that learn deeper part-whole relations between the spatial features and then selectively assign higher weights to the more discriminative features while ignoring misleading ones. The last pair of sub-networks, f H CLS and f V CLS , respectively, perform classification for horizontal and vertical learned features using a softmax activation function. The results of the these three sub-networks, independently applied to horizontal and vertical views, are then fused using a sum-rule strategy, thus obtaining the predicted label, l. Figure 3 represents the architecture of the proposed Caps-Field solution composed by a pre-processing step and three sub-networks: CNN, capsule, and classification, whose descriptions are provided in the following.

C. Network Architecture and Walkthrough
1) Pre-processing: This paper uses the Light Field Toolbox v0.4 software [55] to render the lenslet Light Field Raw (LFR) images, thus creating a multi-view SA array with 15×15 SA images, as illustrated in Figure 3-left. At this stage, the face region is cropped within each 2D SA image using the provided face bounding box locations available in the datasets. Next, the middle row and the middle column SA images from the multi-view SA array are scanned into two separate SA image viewpoint sequences (as shown by red and green rectangles in Figure 3-left), thus capturing the viewpoint changes along the horizontal and vertical directions, respectively. These two separate SA image viewpoint sequences are used as inputs to the proposed deep learning network.
2) CNN Feature Extraction: The proposed CapsField solution first utilizes CNNs to independently extract spatial features from the two input sequences, i.e., horizontal and vertical SA sequences. To this end, the VGG-16 [36] and Resnet50 [56] CNN architectures have been, respectively, used for the face and expression recognition tasks. As both CNNs have been trained on the large-scale VGG-Face 2 dataset [53], the proposed solution will not suffer from overfitting and there is no additional training performed at this stage. More details about the used pre-trained models are provided in Section V-D.
3) Capsule Routing: A capsule network includes layers of capsules where each capsule consists of a group of neurons representing the information being learned. Capsule networks [24] are composed of primary and secondary layers.The first layer encapsulates the input using convolutional, reshaping, and squashing operations to represent the instantiating parameters, such as positional attributes, in the form of a vector. The secondary layer then learns deeper part-whole relations between the sub-parts of an object and the whole object. In the CapsField solution proposed in this paper, the convolution operation has been removed from the primary capsule layer, so that only reshaping and squashing functions are applied to the spatial features coming from the previous sub-network. This modification is due to the fact that the previous CNN feature extraction sub-network already encodes the spatial information through several convolutional layers.
The output generated by the CNN feature extraction subnetwork, i.e., the sequence of multi-view features, is first reshaped to a N c ×C s tensor where N c and C s are the number of primary capsules and their size, respectively. This operation is followed by a non-linear squashing activating function, creating an output vector in which each vector element takes a value in the [0,1] range: where v j is the j th vector output of the primary capsule layer and s j is the j th capsule input formed by reshaping the output of the CNN feature extraction sub-network. The orientation of the output vector, v j , represents the positional properties of the input. This means that when the viewpoint of a section of a face changes, the orientation of the vector also changes while its length remains unchanged. The secondary capsule layer is constructed by N c capsules, where each capsule is updated by a dynamic routing process as originally proposed in [24]. The iterative dynamic routing process first learns the coupling coefficients, C i,j , between two capsules, i and j, available in two subsequent layers. After the degree of agreement between them is computed using the routing softmax function: where b i,j is the log probability indicating whether capsules i and j should be coupled. Next, the input to capsule j, s j , is computed as follows: where W ij is a trainable pose matrix that encodes the relation between the view and the views' sequence, thus establishing an intrinsic capacity for learning angular information. Eq. (5) can also be considered as an attention layer, where the coupling coefficients play the role of attention weights. Finally, a squashing function (Eq. (3)) is used again to create the output vector of capsule j. The iterative dynamic routing process is repeated between the primary and secondary layers for N r iterations, and the final outputs are used to feed the classifier.

4) Classification:
The proposed network uses two independent dense layers with softmax activation as the classifier for the horizontal and vertical capsule features, thus computing the corresponding classification scores. Score-level fusion is finally applied, averaging the horizontal and vertical classification scores to compute the final class probability vector, determining the face or expression recognition result.

D. Implementation and Training Details
The parameter values for training the proposed network are empirically tuned to achieve the best performance for both the face and expression recognition tasks, as presented in Table III. This table presents the optimal parameter values for each subnetwork as well as for the whole deep network, CapsField. As can be observed, the network settings for both face and expression recognition tasks are rather similar except for: i) the choice of CNNs, as the two tasks use different CNN models for spatial feature extraction; and ii) the training batch size, as the number of input samples is different for the face and expression recognition tasks. Concerning the number of routing iterations, different values ranging from 2 to 6 have been considered, where increasing the value to more than 3 did not improve the performance while increasing the computational time. The entire architecture has been implemented using TensorFlow [57] with Keras backend and is trained using a GeForce GTX 1080 GPU.

IV. A NOVEL LIGHT FIELD FACES IN THE WILD DATASET
The proposed LFFW dataset fills the gap regarding the lack of availability of a LF face dataset captured in the wild. It includes data from 53 subjects, 42 males and 11 females, with 36 LF image shots per person, covering several unconstrained variations, totaling 1908 LF face images in the dataset. As each LF image consists of 225 2D SA images captured from 15×15 different perspectives, the LFFW dataset includes 429,300 (53×36×225) 2D facial images in total. The participants were between 20 and 62 years old, from 10 different countries across the globe. The unconstrained variations considered in the proposed LFFW dataset can be categorized into two types: i) acquisition variations, notably different dates, environments, distances, and lighting conditions; and ii) facial variations, notably facial poses, random expressions, occlusions, and actions with no pre-defined protocols or any stimuli used to induce the desired variations. All LF images were taken under 'in the wild' conditions, meaning that no restrictions were imposed on the subjects when capturing their photos. The LF images have been captured using a Lytro ILLUM LF camera [33], with a 40 Megaray sensor and a 30-250 mm lens with 8.3x optical zoom and f/2.0 aperture.
In addition to the LFFW dataset, this paper also proposes a complementary, constrained dataset with the same 53 LFFW subjects, named LFFC, which includes their facial images captured in a controlled acquisition setup. The LFFC LF images have been acquired between 1 day and 3 years prior to acquiring the corresponding in the wild images. The LFFC dataset includes 20 LF shots per person, with different facial variations including expressions, actions, poses, illuminations and occlusions, thus providing 238,500 (53×20×225) 2D facial images in total. This dataset provides additional information essential for studying aging effects and the constrained versus in the wild cross-dataset generalization ability.

A. LFFW Dataset Structure
The hierarchical structure designed for the LFFW dataset is illustrated in Figure 4. The LF image acquisition has been performed in both indoor and outdoor environments, at different locations, with different backgrounds. The acquisition has also been performed in different lighting conditions, e.g., from gloomy to extremely sunny weather, allowing images to be captured in the wild with different illuminations and in the presence of uncontrolled shadows, and backgrounds. As shown in Figure 5(a), for each environment, LF images were   captured from three distance ranges: close, moderate, and far, respectively captured from around 1 to 2, 3 to 4, and 5 to 6 meters from the subjects, thus providing facial LF images with different face spatial resolutions. Finally, the LF images have been captured with several facial variations, including different poses, notably frontal, half-profile, and full-profile, random expressions, occlusions, and actions. This process has been repeated for each environment-distance pair, which means each subject acted each facial variation 6 times. Examples of the variations considered for a subject in the LFFW dataset are illustrated in Figure 6.

B. LFFW Dataset Elements
The LFFW dataset is composed by the following elements:

C. LFFC Dataset
This paper also proposes an LF Face Constrained (LFFC) dataset, complementary to the LFFW dataset; it includes 1060 LF or 238,500 2D SA facial images captured in a controlled environment, from the same 53 subjects available in LFFW. This pair of complementary datasets has considerable research potential. Image acquisition was performed in an indoor environment, using the same Lytro ILLUM lenslet camera [33] similar to the LFFW dataset. An illustration of the LFFC acquisition setup is presented in Figure 5(b); the acquisition setup included a white backdrop background behind a chair for the subject, placed at a fixed distance of 1.25 m from the camera. The scene was illuminated with a three-point lighting kit, including key, fill, and back lights, placed to limit shadows and allow easy segmentation of the subject from the background.
The LFFC dataset includes 20 LF shots per person (each including 225 2D SA images), with different facial variations including: i) one neutral image; ii) three facial expressions: happiness, anger and surprise; iii) two facial actions: closed eyes and open mouth; iv) six poses: looking up, looking down, right half-profile, right profile, left half-profile, left profile; v) two different illuminations: low and high illumination levels; and vi) six facial occlusions: eye and mouth occluded by hand, and face occluded by glasses, surgical mask, and hat. Examples of the various face variations considered in the LFFC dataset are illustrated in Figure 7.
The availability of the LFFC dataset enables constrained versus in the wild cross-dataset studies. To the best the authors' knowledge, this type of study has not been yet performed for LF-based recognition due to the difficulties in obtaining the required data. Additionally, the temporal gap between the  acquisition of the two datasets also allows to study the impact of aging. In order to illustrate how challenging cross-dataset studies can be, t-Distributed Stochastic Neighbor Embedding (t-SNE) visualization [60] has been used to summarize the distribution of images available in the two LFFW and LFFC datasets in a two-dimensional space (see Figure 8). t-SNE maps the multi-dimensional input data to a lower dimensional feature space. The figure shows that considerable difference exists between the two datasets, further indicating that generalization across the two datasets (training on one and testing on the other) may be difficult.

D. Ethics, Access, and Usage Conditions
The participants in the capture of both datasets all signed a consent form before the acquisition, allowing the collection of LF images of their faces, to become part of the proposed datasets, naturally, to be used for research purposes. All access to the dataset images is controlled and supervised, and images will be used anonymously. The generic information asked from the subjects (such as age or gender), to be included as metadata, will also be made available anonymously. Only images from some specific subjects (shown in Figure 9) may appear in publications as per received consent.

E. Facial Expression Annotation
The LFFW dataset includes facial images that are captured with random expressions. To support their usage for expression recognition algorithms, the random expressions are carefully analyzed by the authors of this paper in order to validate and label the acted expressions. In this context, 832 face LF images have been manually annotated with different expression labels including: neutral, happiness, anger, surprise, sadness and disgust, as illustrated in Figure 9. As can be seen, some face expressions are captured in very challenging acquisition conditions, e.g., at very high or very low illumination levels or with occlusions by hand or glasses, making the expression recognition task more difficult.

V. PERFORMANCE ASSESSMENT
This section presents the proposed experimental evaluation protocols and the corresponding performance results and analysis for the proposed and benchmarking recognition solutions. This section also performs ablation experiments, investigating the effect of the individual sub-networks on the overall recognition performance.

A. Face Recognition Evaluation Protocols
To assess the overall face recognition performance for in the wild conditions, five evaluation protocols using the LFFW and LFFC datasets are proposed. These protocols divide the whole data into training and testing sets as described below, where 10% of the training set images have been used for validation in all the test protocols.
1) Cross-environment Evaluation Protocol: This protocol uses the 18 indoor LFFW LF images of each subject for training, whereas the testing phase uses the LFFW outdoor LF images. Then, the process is repeated using the outdoor images as training data and the indoor images as test data. For this protocol, performance results are reported with respect to the various facial variations. Moreover, the sensitivity of the face recognition solutions to the unconstrained environmental conditions in the wild, such as lighting, shadows, occlusions, and backgrounds, is analyzed.
2) Cross-distance Evaluation Protocol: In this protocol, the training stage uses the 12 LFFW LF images of each subject captured from one of the three distances (close, moderate and far), whereas the testing phase independently uses the 24 captured LF images from the two other distances. This process is repeated, changing the training and testing (distance) sets, for a total of six different iterations. This protocol evaluates the face recognition solutions' sensitivity to various details in the face region resulting from capturing images at different distances and in the wild.
3) Cross-pose and expression Evaluation Protocol: This protocol considers practical scenarios where only frontalneutral LF images are available in the training dataset, and LF facial images captured in the wild are used for testing. Therefore, the training set contains only the 6 frontal-neutral LFFW LF images of each subject, while the testing set considers all the other facial variations.
4) Cross-dataset Evaluation Protocol: This protocol performs the training with the LFFC dataset and the testing with the LFFW dataset, and vice-versa. The performance results are reported with respect to the different facial variations. This protocol evaluates the generalization ability of the face recognition solutions, and to some extent the robustness to aging effects. This type of assessment is rarely found in the literature as it requires the availability of appropriate complementary datasets as available in this work. This type of assessment is performed here, for the first time, with LF images.

5) Subject-Independent Evaluation Protocol:
This protocol performs the training with all the images from the first 28 subjects (ID No. 1 to 28) in the LFFW dataset. The trained model is then used as a feature extractor for the testing set including all the images from the remaining 25 subjects. Then, in the testing set, the frontal images have been used as the gallery samples and the rest of the images have been used as probe samples to perform the recognition process. In this protocol, the testing identities are disjoint from the training ones, thus making the face recognition process more challenging and closer to real-world application scenarios.

B. Expression Recognition Evaluation Protocols
To assess the performance of the face expression recognition solutions using the annotated face images, two evaluation protocols are proposed. It should be noted that the number of evaluation protocols designed for expression recognition is less than for face recognition since the expression samples are comparatively limited, both in terms of numbers and variations.
1) Subject-Independent 4-fold Cross Validation: The first evaluation protocol performs subject-independent 4-fold Cross Validation (CV) on the LFFW dataset. Accordingly, the proposed expression recognition solutions are trained with 1/4 of the annotated images available in each expression class, while the remaining 3/4 of the LF images are used for testing. This process is repeated 3 more times with the next folds and the average results are reported.
2) Cross-dataset Evaluation Protocol: This protocol considers cross-dataset evaluation by using expression samples from the LFFC dataset for training and the LFFW dataset for testing, and vice-versa. The in the wild and constrained samples were, respectively, categorized into 6 and 4 expression classes. The two additional expression classes for the in the wild dataset not available in the constrained dataset, i.e., sadness and disgust, are naturally removed from the experiments for this evaluation protocol.

C. Benchmarking Recognition Solutions
Since this paper proposes new evaluation protocols for the two developed datasets, the authors have re-implemented all the benchmarking solutions for both face and expression recognition tasks since otherwise no comparative results would be available. To do so, the best configuration of parameters reported in the original papers have been used to report the rank-1 performance results in this paper.

D. Comparative Face Recognition Performance
This section reports the rank-1 face recognition rates, for the four evaluation protocols proposed earlier, when using CapsField and 12 state-of-the-art face recognition solutions, with the best results highlighted in bold.
1) Cross-environment Performance: The results in Table IV refer to the cross-environment protocol and are independently reported for different facial variations, showing that the environment variations, including different lighting conditions, shadows, occlusions, and non-uniform backgrounds can negatively impact the face recognition performance in the wild. The proposed CapsField solution delivers very impressive performance results when trained with indoor and tested with outdoor samples. The proposed solution achieves less impressive results when tested with indoor samples, showing that it is sensitive, notably to low lighting conditions. Overall, the results show that, on average, the proposed CapsField solution is superior to the other benchmarking solutions for this evaluation protocol.
2) Cross-distance Performance: The cross-distance protocol results are presented in Table V. Since the Lytro ILLUM LF camera used in this research has a small baseline, it is expected that more disparity is captured at closer distances. Additionally, the cropped face images have different resolutions, due to the different capturing distances. As the CNN feature extractor first resizes all the images to 224×224 pixels, the images with larger or smaller resolutions were, respectively, down-sampled or up-sampled to be then processed by the network; this means the images captured at far distances may include up-sampling artefacts. Hence, the overall performance results for all benchmarking solutions are better when testing with the images captured from close and moderate distances, as these LF images feed the recognition solutions with more details, thus further contributing to an improved performance. The proposed CapsField solution obtains very good results, notably achieving a performance gain of 2.95% when compared to the best performing benchmarking solution, i.e., VGG+SeqL-LSTM [21].
3) Cross-pose and expression Performance: The cross-pose and expression protocol results are presented in Table VI for the several facial variations considered, notably including different face orientations, random expressions, occlusions, and actions. The overall performance results for the benchmarking solutions show poor performance when trained with only frontal faces and tested with different poses, notably fullprofile images. The performance is also poor for random action variations as the type of actions during training can be very different from those used for testing. Overall, the performance of the proposed CapsField solution is considerably better than all other benchmarking solutions, notably showing a gain of 15.85% over the best benchmark.

4) Cross-dataset Performance:
The results in Tables VII and VIII refer to the cross-dataset evaluation protocol and correspond to two situations: i) LFFC is used for training and LFFW is used for testing; and ii) LFFW is used for training and LFFC is used for testing. As expected, better performance results are obtained when testing on the less challenging LFFC dataset. The results show that the proposed solution outperforms all the benchmarking solutions for both cases by a large margin, achieving performance gains of 11.05% (when testing with LFFC) and 14.62% (when testing with LFFW), compared to the best performing benchmark. These results, showing a better generalization ability of the proposed CapsField solution, clearly illustrate the advantage of exploring the richer information available in LF images with the consideration of a more robust model created by capsule networks.

5) Subject-independent Performance:
The subjectindependent protocol results are presented in Table IX for the facial variations considered, notably different poses, random expressions, occlusions, and actions. Overall, the average performance of the proposed CapsField solution is better than all the benchmarking solutions, even for this protocol, where the face recognition process is more challenging and closer to real-world application scenarios.

E. Comparative Face Expression Recognition Performance
This section reports the expression recognition accuracy obtained for the two proposed evaluation protocols. The performance results are reported in Table X for the proposed and 6 benchmarking solutions, with the best results highlighted in bold.
The performance results for both evaluation protocols clearly show that the proposed CapsField solution achieves considerably better performance than the non-LF-based deep recognition solutions, presented in the first four rows of Table  X. This improvement is due to the fact that the proposed solution exploits the available spatio-angular information in the LFs for the face expression recognition task. The results also show the superiority of the proposed CapsField solution over the LF-based ones, notably VGG+LSTM [22] and VGG+BiLSTM+Attention [23] solutions. As the comparative LF-based and the proposed solutions use the same spatial CNN model, i.e., VGG-16 pre-trained with VGG-Face 2, the performance results reveal that the capsule network better models the richer angular features than the LSTM and BiL-STM recurrent networks used in the benchmarking solutions for the LF expression recognition task.

F. Benefits of LF Images for Face Recognition
An additional experiment has been performed to evaluate the added value of LF images in comparison with single-view images for face recognition. This considers the comparison between four LF-based face recognition methods including LFLBP [17], HOG+HDG [18], VGG-16 + LSTM [20], and the proposed ResNet-50 + Capsule, and their corresponding single-view versions, including LBP [62], HOG [63], VGG-16 [36], and ResNet-50 [53], applied to the central SA image. The results presented in Table XI clearly demonstrate the superiority of the LF-based methods over their corresponding single-view baselines for the four subject-dependent test protocols, thus highlighting the added value of LF images for face recognition.

G. Angular Feature Space Exploration
To study the impact of different strategies for dealing with the available LF angular information, t-SNE [60] is used again to plot the feature spaces produced by the different strategies, thus showing the different discrimination ability of the various LF angular features in a two dimensional space. Figure 10 offers a visualization of the features that are used as inputs to the classifier, when four different strategies are used to deal with the angular information, notably: i) only the 2D central SA image is used, thus ignoring the available angular information (Figure 10-a) ; ii) the concatenated features extracted from all the views are used (without capsule network) to feed the classifier (Figure10-b); iii) LSTM RNN [20] is used to exploit the angular information (Figure10-c); and iv) capsule network, as proposed in this paper, is used to exploit the angular information (Figure 10-d). To make the visualisation legible, the t-SNE analysis is only performed for the first 10 LFFW subjects using the cross-pose and expression face recognition protocol. The t-SNE plots show that the proposed CapsField solution forms denser and more effective clusters that can facilitate distinguishing between the various classes; this validates the superiority of the capsule network in learning angular information.

H. Ablation Study
Ablation experiments have been performed based on the proposed evaluation protocols for the adopted datasets by systematically removing individual components of CapsField one by one and evaluating the performance of the 'reduced' models. It must be noted that the CNN sub-network must always be kept in order to extract the spatial features.  the classification scores of both LF sequences, average performance gains of 0.80% and 1.04% are obtained, respectively, when compared to a CapsField version considering only the horizontal or vertical SA image sequences.
2) Impact of Capsule Sub-network: Finally, the capsule component is removed, and the CNN spatial features obtained from different LF SA images are directly concatenated for classification. It is worth noting that the SE-ResNet-50 [53] and VGG-16 [53] results presented in Tables IV through VIII are applied to a single 2D central SA image, while the features extracted from all the views are concatenated here. Removing the capsule sub-network decreases the recognition performance of the proposed CapsField solution as it limits the ability to learn the inter-view/angular relations between the spatial features. Moreover, in the absence of the capsule component, it is not possible to assign higher weights to the more relevant features. These observations reveal the added value of the capsule sub-network in learning a model that fully exploits the angular features available in LF images.

I. Complexity Analysis
The computational time and embedding size for the proposed and benchmarking recognition solutions are studied in this section. This analysis has been done by measuring the execution times on a 64-bit Intel PC with a 3.20 GHz Core i7 processor, 48 GB RAM, and a GeForce GTX 1080 Ti GPU, running TensorFlow with Keras backend. Table XIII shows the training and testing times (in seconds) per each 2D/LF image as well as the final embedding size in terms of the number of embedding elements. It can be observed from Table XIII that the proposed CapsField solution offers the most compact embedding when compared to the benchmarking solutions, thus simplifying the retrieval and transmission of the embeddings as well as reducing the computational time for the testing phase. The required training time for the CapsField solution is higher than other benchmarking solutions, which is the tradeoff for extracting more discriminative features. It is also worth noting that the very high training and testing times for the VGG-D3 [19] solution derives from the necessary disparity and depth maps extraction processes that are computationally very expensive.

VI. CONCLUSION
This paper proposes a new solution for both face and expression recognition tasks, called CapsField, based on the combination of convolutional neural and capsule networks. In order to analyze the performance of CapsField in the wild, this paper proposes the first unconstrained LF face dataset, An extensive benchmarking study has been performed for challenging evaluation protocols on the developed datasets. The performance results have shown the superiority of the CapsField solution, compared to the state-of-the-art in LFbased face and expression recognition, due to its ability in learning inter-view and intra-view relations available in an LF image.

VII. ACKNOWLEDGMENT
This work is partly funded by FCT/MCTES under the project UIDB/50008/2020.  Since January 2018, he is the SPS Vice-President for Conferences. Since 2013, he is also a EURASIP Fellow for "contributions to digital video representation technologies and standards". Since 2015, he is also a IET Fellow. He is/has been a member of the Scientific and Program Committees of many international conferences and workshops. He has been participating in the MPEG standardization activities, notably as the head of the Portuguese delegation, MPEG Requirements Subgroup Chair, and chair of many Ad Hoc Groups related to the MPEG-4 and MPEG-7 standards. Since February 2016, he is the JPEG Requirements Subgroup Chair. He has contributed more than 300 papers in international journals, conferences and workshops, and made several tens of invited talks at conferences and workshops. His areas of interest are visual data analysis, coding, description, adaptation, quality assessment, and advanced multimedia services. He has been guest editor of several special issues of scientific journals and cooperated in many conference organizing committees, as general chair, program chair, finance chair or special sessions chair. He is a founding member of the Advisory Board of EURASIP, and he is the elected chairman of EURASIP's Technical Area Committee on Biometrics, Data Forensics and Security. He has coauthored more than 135 conference and journal papers. Research interests include image and video analysis and processing, with emphasis on biometric recognition.