{'fraction_non_alphanumeric': 0.0391025397669033, 'fraction_numerical': 0.015261379598344785, 'mean_word_length': 4.637063984592339, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Unified Approach to Kinship Verification

In this work, we propose a deep learning-based approach for kin verification using a unified multi-task learning scheme where all kinship classes are jointly learned. This allows us to better utilize small training sets that are typical of kin verification. We introduce a novel approach for fusing the embeddings of kin images, to avoid overfitting, which is a common issue in training such networks. An adaptive sampling scheme is derived for the training set images to resolve the inherent imbalance in kin verification datasets. A thorough ablation study exemplifies the effectivity of our approach, which is experimentally shown to outperform contemporary state-of-the-art kin verification results when applied to the Families In the Wild, FG2018, and FG2020 datasets.


INTRODUCTION
The goal of kin verification [1], [2], [3], [4] is to verify whether or not two people are related by a particular kin relationship given their face images. The kinship classes are depicted in Fig. 1, where given a pair of face images {φ i , ϕ i } and a kinship class λ i ∈ Λ ={B-B, F-S, M-D, S-S, F-D, M-S, SIBS}, we aim to verify whether φ i and ϕ i are related by λ i . In contrast to face recognition, kin faces are non-identical, and their visual similarity is often unintuitive, even for human observers, as kins might differ by gender and notable age differences. The facial similarity of kins varies significantly between different families and even within the kins of a particular family, implying significant intraclass variability. As face recognition schemes aim to identify individual subjects, kins will be identified as different individuals. Following the common approach in face recognition, classical kin verification schemes encoded the input faces {φ i , ϕ i } using handcrafted features φ i , ϕ i [5], [6], [7] such as LBP, HOG, and Gabor. The representation was refined using metric-learning [8], [9], [10], and the kinship was classified by applying KNN and SVM classifiers. With the emergence of deep learning, it was applied to kin verification. Facial features were derived by training face recognition CNNs on large-scale face recognition datasets [11] and using transfer learning to apply them to kin verification [11]. Recent results by Wang et al. [12] applied a Generative Adversarial Net (GAN) to mitigate the agedifferences by synthesizing face images depicting intermediate ages. Small-scale datasets were initially utilized in kin verification [5], [13], [14], most of which were shown [15], [16] to be biased, as the faces of kins were often cropped from the same photo, implying that the kinship could be inferred from the similar chromatic properties of the photo crops. The large-scale RFIW dataset [17] was the only one shown to be unbiased, consisting of kin pairs cropped from different photos. The novel FIW in Multimedia (FIW-MM) database, recently introduced by Robinson et al. [18], is a large-scale multi-modal kin verification dataset, consisting of of video, audio, and contextual transcripts In this work, we propose a deep learning-based approach for kin verification using a siamese CNN. We apply multi-task learning to train a single unified CNN for all kinship classes, allowing to leverage their training samples to refine the face embedding backbone CNN. We show that kin verification, in contrast to face recognition, is prone to overfitting and derive a novel CNN architecture for the fusion of face embeddings based on transposed 1 × 1 convolutions. We also show that kin verification datasets are inherently imbalanced in terms of the number of samples per family, and samples per kin (father, mother, etc.), implying that the training process might be biased. For that, we propose an adaptive sampling scheme for the training set, which is shown to improve the kin verification accuracy. Some kin verification classes, such as B-B and D-D, are symmetrical, allowing to utilize siamese CNNs. In contrast, other kinship classes, such as F-D, F-S, etc., are asymmetric in terms of age and gender, requiring different CNNs for each class. For that, we applied an asymmetric weighting layer that improves accuracy.
In particular, we propose the following contributions: (1) We present a unified approach for kin verification that applies multi-task learning to jointly utilize the training sets of all kinship classes. (2) We show that kin verification is prone to overfitting and suggest a novel subnetwork for the fusion of feature maps based on cascaded 1×1 convolutions.
(3) To overcome the imbalanced training set, we introduce an adaptive approach for sampling the training pairs, that is shown to improve the kin verification accuracy. (4) The proposed scheme is experimentally shown to outperform contemporary state-of-the-art approaches, when applied to the RFIW [17], FG2018, and FG2020 [19] datasets.

RELATED WORK
Kin verification is related to face verification, and similar approaches were often used [17]. Thus, early kin verification schemes utilized handcrafted image descriptors for encoding the face images. Puthenputhussery et al. [6] used SIFT descriptors and genetic fisher vectors to optimize the similarity between true and false kins. Similarly, Fang et al. [5] applied LBP, HOG, and Gabor descriptors to encode the face images and trained KNN and Kernel SVM classifiers. LBP was used as a local face descriptor by Dander and Limbate, [7]. The informative patches in the face images were detected by Quin et al. [20] by applying sparse regularized group lasso regression to a SIFT-based face descriptor. The resulting face embeddings were classified by a linear SVM.
Mahpod and Keller [21], [22] proposed a metriclearning-based approach, applying symmetric and asymmetric metric learning to face embeddings using handcrafted features. The resulting representations were classified by an SVM classifier, and the scheme was applied to the KinFaceW-I and KinFaceW-II datasets [14]. A color histogram was found to be the most informative image feature, corresponding to the later findings by Lopez et al. [15] and Damson et al. [16] who showed these datasets, among others, to be color-biased. A neighborhood repulsed correlation-metric-learning approach was used by Yang [10], while Xn and Shan proposed an ensemble of bilinear models [23], where each model was trained by metric-learning using multiple face descriptors. Handcrafted image descriptors, such as LBP and Fisher vectors, were used in the Triangular Similarity Metric-Learning (TSML) approach by Pilei et al. [8], where the dimensions of the descriptors were reduced by metric learning. Optimal weights per feature were computed by Hairpin et al. [9] in the Discriminative Multimetric-Learning (DML) scheme, and the weighted features were used within a metric learning formulation.
With the emergence of deep learning, CNNs were also applied to kin verification. Lu et al. [24], proposed discriminative deep metric-learning (DML) to train a CNN to jointly learn multiple metric-learning networks, and use ensemblelearning to optimize the results. An asymmetric scheme, where each image is processed by a different CNN that is adjusted to the input (Father, Son, etc.), was proposed by Avignon et al. [25], where metric-learning was applied to the face embeddings. A coarse-to-fine domain adaptation approach was derived by Duad et al. [11] to utilize the deep features learned by a face recognition network as kin verification features.
A large-scale kin verification evaluation by human reviewers was conducted by Kohl et al. [26], who proposed a hierarchical verification scheme, using representation learning to encode the face regions. The Kinnet CNN was introduced by Li et al. [27], where the face embeddings were learned using a large-scale face recognition dataset. A triplet loss was applied to learn the kin verification, and the number of images per family member was balanced by image augmentation. An ensemble of four CNNs was applied to improve the accuracy. Kohl et al. [28] applied autoencoders to detect kin similarity in unconstrained videos, using a supervised mixed-norm autoencoder to compute the sparse embeddings of the kin pairs. Deep autoencoders were also studied by Deadman et al. [29] to detect informative facial features for kin verification, where a Sparse Discriminative Metric Loss (SDM-Loss) was derived to utilize the positive and negative training pairs. A review and evaluation of contemporary schemes were detailed in the RFIW 2020 challenge [19], organized by Robinson et al..
A Generative Adversarial Net (GAN) was proposed by Wang et al. [12] to mitigate the age differences by synthesizing younger and older face images of parents and their siblings, respectively. Thus, a conditional GAN was used to synthesize cross-generation kins in an intermediate age domain, to improve the face similarity. Ozkan and Ozkan [30] proposed a GAN-based approach for synthesizing an image of a child's face using the face images of a single parent. The synthesis is derived by a generator network in an autoencoder architecture. An adversarial loss was applied using large-scale unsupervised data, to mitigate the overfitting, due to the small annotated training sets. The stability of the approach was improved using cycle-consistency. Gao et al. [31] introduced a novel biologically-motivated kinship generation approach, where the facial appearance of a child is inherited from both parents by a random genomic fusion process. For that, a three-step approach was derived, where first a conditional auto-encoder (CAD) is trained on a largescale face dataset to encode the facial appearance given age and gender. Second, a CNN denoted as DNA-Net is trained to further encode the face embeddings into an embedding denoted as genes. Random subsets of male and female genes are combined and decoded by the CAD decoder to synthesize the child's face.
A fundamental flaw in most contemporary kin and family verification image datasets was detected by Lopez et al. [15], who noticed that the KinFaceW datasets were created by cropping the kin face images from the same family photographs. Hence, learning-based schemes in general, deep learning-based in particular, might learn the chromatic ("from same photo") cues instead of the kinship similarity. Dawson et al. [16] extended these results by training the deep learning-based From Same Photo (FSP) image classifier that was trained to detect whether two image patches were cropped from the same photo. The FSP was applied to most contemporary kin face datasets, achieving accurate kinship classification in all, but for the RFIW dataset. Thus, in this work, we evaluate the proposed scheme by training it, using only the RFIW dataset [17]. A thorough survey by Robinson et al. [ Fig. 2. The proposed kin verification system. The logical components are color-coded. The face embedding CNN is shared by all kinship classes and is initially pretrained on the CASIA-WebFace face image dataset [33]. The embeddings ϕ i and φ i of the input face images are fused and classified by separate subnetworks, one per kinship class. The weightsharing FC-2 layer is used to refine the face embedding on the RFIW dataset [17].

UNIFIED MULTI-TASK KIN VERIFICATION
In this work, we propose a unified multi-task kin verification approach, where all kinship classes are jointly learned, using the CNN depicted in Fig. 2. It consists of a siamese network for learning the face embeddings used in the classification of all of the kinship classes, followed by multiple kinship classification subnetworks, one per kin verification task, e.g., B-B, SIBS. The embeddings ϕ i , φ i of {φ i , ϕ i } are computed using the siamese CNN detailed in Section 3.1. ϕ i , φ i are jointly trained end-to-end using all kinship classes and corresponding images in the training set. In contrast, in previous schemes [17], a separate classification CNN was trained per kinship class, utilizing an order of magnitude less training images. Each kin verification triplet ϕ i , φ i , λ i is classified by the embedding-fusion and classification subnetwork depicted in Fig. 2. There are |Λ| = 7 such subnetworks that are detailed in Section 3.2. A novel embedding-fusion approach is introduced in Section 3.2, based on channelwise 1 × 1 convolutions, to avoid overfitting. The overfitting issue was previously reported by Robinson et al. [17] when applying distance learning to face embeddings computed using the Sphereface CNN, resulting in little classification accuracy improvement. Kin verification datasets in general, and the RFIW dataset [17] in particular, are inherently imbalanced, with respect to the number of images per family, and the number of images per kin, in each family. For that, we propose in Section 3.3 an adaptive image sampling scheme that is shown to improve the verification accuracy. The proposed scheme is trained using multiple losses. The siamese face embedding CNN is refined using the Sphere losses L φ Sphere and L ϕi Sphere , that are applied to classify the embeddings ϕ i , φ i of the 10,676 identities in the RFIW dataset. The kin verification is classified using the |Λ| = 7 subnetworks trained using L λi BCE λi∈Λ Binary Cross-Entropy (BCE) losses, such that L λi BCE verifies the kinship λ i . Given a particular training triplet ϕ i , φ i , λ i , all other BCE losses are zeroed Thus, the overall loss is given by and α ∈ R. We tested a range of values of α, and the kin verification accuracy was robust to its choice. Hence, α = 1 was used.

Face Embedding
The backbone of the face embedding CNN is based on the Sphereface CNN [34], which was used by Robinson et al. [17] to achieve state-of-the-art kin-verification results. It is detailed in Table 1 where the embedding of an input image ϕ is given by the output of the FC-1 layer ϕ∈R 512 . The output of FC-2 is used to refine the face embedding CNN using the RFIW dataset [17] and is optimized by the SphereLoss [34]. Each training sample consists of a pair {φ i , ϕ i } of input images, and both CNNs are jointly trained via weight sharing.

The Fusion of Face Embeddings
The face embeddings ϕ i , φ i ∈R 512 are fused to compute the kin verification scores. For that, we propose a novel fusion subnetwork, whose core attribute is avoiding overfit that is common in kin verification CNNs due to the small training sets, and significant intraclass variability. An overview of the proposed fusion subnetwork is shown in Fig. 3. Concatenation and bilinear pooling [35] are com-  Table 2. Average pooling and a Softmax activation are applied to the fused embedding, before the BCE loss. (b) The input layers of the proposed embedding fusion subnetwork. The embeddings ϕ i ∈R 512 and φ i ∈R 512 are concatenated channel-wise, resulting in a w ∈R 512×2 activation map, that is processed by 512 1 × 1 transposed (channel-wise) convolutions, resulting in an R 512×512 activation.
mon embedding fusion approaches that are followed by corresponding FC layers. This requires a significant number of training parameters, resulting in overfitting, as exemplified in Section 4.3. Hence, we propose a novel lowcapacity fusion subnetwork, that utilizes significantly fewer parameters. It is shown to provide better generalization and improves the verification accuracy. The core of the proposed fusion approach is depicted in Fig. 3b, where the inputs ϕ i , φ i ∈R 512 are concatenated and processed channel-wise, such that the concatenated embedding is w ∈R 512×2 . 1 × 1 channel-wise convolutions are then applied to w and the succeeding activations maps as detailed in Table 2, such that their initial dimensionality (d = 512) is retained. As both embeddings ϕ i , φ i are computed by the same weightsharing siamese CNN, and are the output of an FC layer, the corresponding entries in ϕ i and φ i encode the same semantic content. The spatial locality in such embeddings is uninformative, implying that they can be processed by either an FC layer or 1 × 1 convolutions, as in the proposed scheme.
The face embeddings ϕ i , φ i are normalized elementwise before applying the fusion scheme φ n i = φ i . * w φ where (·. * ·) denotes pointwise multiplication and w ϕ , w φ ∈R 512 . Some of the kinship classes are symmetric in terms of gender (i.e. B-B, S-S, F-S, M-D), while others are asymmetric (F-D, M-S, SIBS). We utilize Eq. 4 to induce asymmetry in the kin verification CNN for the asymmetric classes, by learning symmetric weights w ϕ = w φ = w for the same-gender classes, and asymmetric weights w ϕ , w φ for the asymmetric ones.

Adaptive Sampling of Imbalanced Training Sets
Kinship verification datasets are inherently imbalanced with respect to multiple attributes. First, families differ significantly by the numbers of their images and the images per kin [17]. Second, following Table 3

EXPERIMENTAL RESULTS
The proposed scheme was experimentally evaluated using the RFIW (Families In the Wild) dataset [17], that was found by Lopez et al. [15] and Damson et al. [16] to be the only unbiased kin verification dataset. The RFIW is the largest kin verification dataset depicting 1, 000 different families, for a total of 11, 932 images. There are 654, 304 positive and negative pairs overall, and the images contain gender metadata. We followed the evaluation protocols detailed in [17], by applying the restricted protocol where the identities of the subjects in the dataset are unknown, and we are given predefined pairs of training images, per kinship class. Thus, the adaptive sampling (Section 3.3), and the embedding fine-tuning can't be applied. In the unrestricted protocol, both the identities and the kinship classes (F, M, B, S, etc.) are given, allowing to train using a significantly larger set of pairs. Following Robinson et al. [17], we used a pre-trained VGG-Face face embedding [36] CNN, to compare with previous schemes based on that backbone. The fusion embedding subnetwork follows Section 3.2. In the unrestricted protocol, we used all of the CNN components detailed in Section 3, and the Sphereface was finetuned using the RFIW dataset. The five-fold cross-validation split, as in [17], was used, with no family-overlap between the different folds. Negative pairs were sampled randomly, the same number as the positive ones.

Network Parameters and Training
The proposed approach was trained in two steps, where only the first step was applied in the unrestricted protocol. First, the face embedding CNN was trained using the CASIA-WebFace image dataset [33]. A SphereLoss was attached to the output of the layer FC-1 and applied using l ∈ [5,1500]. The learning rate was gradually reduced from 0.01 to 0.0001 in 30 epochs, using an SGD optimizer.
In the second training step, applied for the unrestricted protocol, the RFIW dataset was used to jointly train a refined face embedding and a multi-task kin verification classifier. For that, the complete CNN, as depicted in Fig. 2, was used. Two SphereLosses were connected to the outputs of the FC-2 layer, for additional face embedding refinement, by classifying the faces in the RFIW dataset [17], and the multitask kin verification subnetworks were jointly trained.
The MTCNN [37] was used to detect and align the faces in the training and test images. The face images were scaled to 108 × 108 pixels and augmented by changing the image gamma factor, scalings s ∈ [0.5, 1, 2], horizontal flipping, and random jitter of dx, dy ∈ [−2, 2]. All of the augmentations were applied jointly and randomly. The weighting in Eq. 4 is applied differently to the symmetric (B-B, F-S, M-D, S-S) and asymmetric (F-D, M-S, SIBS) pairs of kins. The training pairs of the symmetric classes are randomly input to either the left or right side of the CNN. In contrast, each kin in the asymmetric classes is input to a predefined side of the CNN. We trained the network using the SGD optimizer with a learning rate ∈ 10 −3 , 10 −5 for 30 epochs, where the learning rate was reduced by a factor of 10 when the loss decay flattened.

Kin Verification Results Using the RFIW Dataset
The proposed scheme was evaluated by applying it to the RFIW Dataset using both the restricted and unrestricted test protocols. We present the results of the FG2018 challenge [38] using the restricted protocol in Table 4. Our method is shown to be the most accurate, outperforming the secondbest by 13.8%. In the unrestricted case, we compare to contemporary state-of-the-art results of the FG2018 [38] challenge. The unrestricted test protocol is more suitable for training deep CNNs, such as ours, as it provides a larger training set.
The FG2018 results are shown in Table 5. In the upper part of the table, we compare with kin verification schemes based on the VGG-Face [36] backbone. We cite the results of methods in which distance learning was applied on top of the VGG-Face backbone, that were implemented by Robinson et al. [17]. Our approach was also implemented using the VGG-Face backbone to allow fair comparison, and outperforms all previous methods, in all categories, but for the B-B class. Compared to the mDML scheme [43] having the second-highest average accuracy among all methods, we achieve an average accuracy improvement of 2.59%.
In the mid-segment of Table 5 ("Face embedding"), we compare with methods based on face embedding CNNs. Compared to the Sphereface CNN [17], our approach shows an average improvement of 8.34%, exemplifying the upside of the proposed embedding fusion and classification subnetwork. Average improvements of 10.88% and 6.39% are achieved for the cross and intra-generation classes, respectively. Our approach is thus more accurate in verifying the more difficult cross-generation classes. Comparing our scheme implemented using the VGG-Face [36], to those based on the Sphereface CNN [17] backbones, it follows that the Sphereface-based ones are the more accurate. Last, we compared with recent state-of-the-art results [12], [27], that similar to our method, utilize both face embedding and metric learning. While these schemes outperform all of the "Face embedding" methods, they are outperformed by our approach in all categories.
The results for the FG2020 challenge [19] are reported in Table 6 and compared with the leading entries submitted to the FG2020 challenge, as reported on the challenge website. We could not apply our approach to the GFGD, GFGS, GMGD, and GMGS 1 categories, whose training sets are smaller by an order of magnitude (Table 3). It follows that our scheme outperforms all previous results on average, and in five out of eight kinship categories.
Qualitative kin verification examples of images from the RFIW dataset are depicted in Fig. 4. The reference images were chosen randomly and depict the common results. For the S-S and B-B kinship classes, the face similarity might seem intuitive. But the similarity of other kinship classes such as F-S, F-D, etc., seems to be unintuitive, In particular, comparing the True Positives with the False Positives

Ablation Study
We conducted an ablation study reported in Table 8, wherein each experiment, we modified a single algorithmic component. Our approach was applied following the unrestricted protocol using the RFIW dataset, as in Section 4.2. Adaptive sampling. We first studied the adaptive sampling scheme presented in Section 3.3. The statistics of the RFIW dataset [17], with and without applying the adaptive sampling, are reported in Table 7 and exemplify the balancing effect. The variations in the mean and standard deviation of the balanced RFIW are notably smaller. To compare the corresponding kin verification accuracy, we trained a network following our approach, without the adaptive sampling. Instead, we used the common random sampling. The proposed adaptive sampling is shown to improve the classification accuracy by a significant 4.44%. Multi-task training. The multi-task training was evaluated by training a different CNN for each kinship class instead of the unified approach, as in Section 3. Each of the multiple single-task face-embedding CNNs was refined as in the first phase in Section 4.1. For that, we used the RFIW dataset and enabled one of the Sphere losses L ϕi, φ Sphere , as in Fig. 2. The fusion and classification subnetworks were then trained separately per kinship class. The results show that the proposed multi-task training provides an average accuracy improvement of 1.1%, implying that different kinship classes share a joint structure utilized by the proposed scheme to improve accuracy.
Embedding-fusion scheme. The embedding-fusion subnetwork, introduced in Section 3.2, was studied by replacing it with the common fusion approach where the embeddings ϕ i , φ ∈ R 512 were concatenated, followed by an FC layer. This results in a significant accuracy degradation of close to 25%, compared with our scheme. We attribute that to overfitting, as kin verification is characterized by significant intraclass variability. We also applied our approach without using a fusion scheme, such that the face embeddings were used for kin verification, as in [17]. In such a scheme, there is no overfit, but the results are inferior by 5.9%.
Asymmetric normalization. We studied the use of the asymmetric normalization in Eq. 4 by applying the proposed scheme without the normalization, and by using symmetric weights w ϕ = w ϕ . Using the asymmetric weights improves the average accuracy by 0.65% and 0.57%, compared to not applying the weighting, and using symmetric weights, respectively.

CONCLUSIONS
In this work, we proposed a multi-task deep learning-based approach for kin verification, where all kinship classes are jointly trained, utilizing all of the training data. We also introduced a novel embedding fusion scheme to fuse the face embeddings of the input subjects, that is shown to avoid the overfitting issues common in kin verification, and provides notable accuracy improvement. An adaptive sampling of the training pairs in the RFIW dataset allows creating a balanced training set that further improves the kin verification accuracy. Our scheme is experimentally shown to achieve state-of-the-art accuracy when applied to the RFIW, FG2018, and FG2020 datasets in both the restricted and unrestricted test protocols.