{'fraction_non_alphanumeric': 0.03170839307378405, 'fraction_numerical': 0.01058784603507224, 'mean_word_length': 4.774039482063256, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prepended Domain Transformer: Heterogeneous Face Recognition without Bells and Whistles

Heterogeneous Face Recognition (HFR) refers to matching face images captured in different domains, such as thermal to visible images (VIS), sketches to visible images, near-infrared to visible, and so on. This is particularly useful in matching visible spectrum images to images captured from other modalities. Though highly useful, HFR is challenging because of the domain gap between the source and target domain. Often, large-scale paired heterogeneous face image datasets are absent, preventing training models specifically for the heterogeneous task. In this work, we propose a surprisingly simple, yet, very effective method for matching face images across different sensing modalities. The core idea of the proposed approach is to add a novel neural network block called Prepended Domain Transformer (PDT) in front of a pre-trained face recognition (FR) model to address the domain gap. Retraining this new block with few paired samples in a contrastive learning setup was enough to achieve state-of-the-art performance in many HFR benchmarks. The PDT blocks can be retrained for several source-target combinations using the proposed general framework. The proposed approach is architecture agnostic, meaning they can be added to any pre-trained FR models. Further, the approach is modular and the new block can be trained with a minimal set of paired samples, making it much easier for practical deployment. The source code and protocols will be made available publicly.


I. INTRODUCTION
F ACE recognition (FR) systems offer a convenient way for access control. Most of the state-of-theart FR methods achieve excellent performance in 'in the wild' conditions and human parity in face recognition performance [1], thanks to convolutional neural networks. Typical FR systems operate in the homogeneous domain, meaning the enrollment and matching are performed with the same modality, typically with face images obtained from an RGB camera (visible spectrum).  However, in several scenarios, matching in a heterogeneous setting could be advantageous. For example, nearinfrared (NIR) cameras, ubiquitous in mobile phones and surveillance cameras, offer superior performance irrespective of the illumination conditions [2]. However, an FR system operating in a homogeneous setting will require enrollment samples obtained under the same NIR camera. Heterogeneous face recognition (HFR) systems try to alleviate this limitation, by enabling ways to perform cross-domain matching. For example, with an HFR system, enrolled RGB images could be matched with NIR images, obviating the requirement for enrollment of different modalities [3]. This also opens up the possibility of using additional channels for recognition, such as thermal and shortwave-infrared, without having enrollment samples in the corresponding modalities. Thermal images can be acquired without using active illumination and hence could be used robustly in day and night conditions. In short, the HFR task is very useful in practical applications and could provide a way to extend FR to challenging and uncontrolled environments.
Though HFR is very useful, it is very challenging for several reasons. First, there is a large domain gap between the images captured by various sensors. The arXiv:2210.06529v1 [cs.CV] 12 Oct 2022 performance of the networks trained using RGB images deteriorates when used with images captured with other sensing modalities [4]. Further, there are not many largescale heterogeneous datasets to train models specifically for HFR [5]. Leveraging pre-trained FR models, which are trained using large-scale face recognition datasets, appears to be a reasonable strategy to follow in this limited data domain. Furthermore, special emphasis should be put on developing approaches that work with a limited amount of paired data as this type of heterogeneous data is costly to acquire and not readily available [5].
In this work, we propose a novel, yet surprisingly simple approach for HFR. We leverage a pre-trained FR model as one of the key components in our framework. Our approach doesn't depend on the selection of architecture of the FR model giving it maximum flexibility in deployment. We achieve this by prepending a new network module, called Prepended Domain Transformer (PDT), to a pre-trained FR module to transform the target domain images. The only learnable component is the new prepended module, which is very parameter efficient and obtains excellent performance with few paired samples. This method is very practical in deployment scenarios since one just needs to prepend a new module to convert a typical FR pipeline to an HFR pipeline. The approach is generic and can be retrained easily for any pair of heterogeneous modalities. Through extensive evaluations, we show that this simple addition achieves state-of-the-art results in many challenging HFR datasets. The framework's design is intentionally kept simple to demonstrate the effectiveness of the approach and to allow for future extensions. Moreover, the parameter and computational overhead added by the framework is negligible, making the proposed approach suitable for real-time deployment.
The main contributions of this work are listed below: • We propose a heterogeneous face recognition framework, leveraging a pre-trained FR model together with a Prepended Domain Transformer block. The approach is architecture agnostic and can be used in many heterogeneous recognition scenarios. The method has minimal computational overhead and a very minimal number of learnable parameters, which makes it easy for deployment. • We perform an extensive set of experiments on public datasets and validate the effectiveness of the proposed approach in different heterogeneous face recognition settings. • We introduce a new multi-channel heterogeneous face recognition (MCXFace) dataset which consists of both homogeneous and heterogeneous protocols with channels such as color, thermal, depth, stereo, infrared, short-wave infrared, and synthesized 3D maps. We provide standard protocols and baselines for the heterogeneous protocols in this dataset. Finally, the source codes are made available publicly to make it easier to extend the work further 1 .
The rest of the paper is organized as follows. Section II presents recent literature on HFR. Details of the proposed approach are described in Section III. Extensive evaluation of the proposed approach, along with comparisons with the state-of-the-art, and discussions, are presented in Section IV. Conclusions and future directions are described in Section VI.

II. RELATED WORK
The task in HFR is matching face images collected in different sensing modalities. The challenge lies in the fact that the image of the same subject appears very different in different modalities (domain gap). This difference in appearance increases the within-class variance making matching difficult, and a direct comparison of these heterogeneous images degrade the performance. Several approaches have been proposed in the literature to address this limitation.

A. Common-space projection method
Common-space projection methods [6], [7] aim to learn a mapping to project different face modalities into a common shared subspace in an effort to reduce the domain gap. Lin and Tang [8] developed a common discriminant feature extraction method for extracting features from cross-modal images and projecting them onto a common feature space. Canonical correlation analysis (CCA) was proposed as a way to match face images between NIR and VIS by Yi et al. [9]. To learn mapping functions that connect cross-modality domains and common spaces, regression-based approaches were proposed by authors in [10], [11]. Sharma and Jacobs [12] proposed a partial least squares-based method to learn a linear mapping for face image across different modalities so that the mutual covariance is maximized. Klare and Jain [3] proposed a way to represent face images in terms of their similarity to a set of prototype face images. The prototype-based face representation was then projected onto a linear discriminant subspace, which was used to perform the recognition. In [13], the authors proposed a novel approach to the HFR task called Domain-Specific Units (DSU). Essentially, they suggest that high-level features of convolutional neural networks trained on the visible spectrum are domain-independent, and they can be used to encode images captured in other sensing modalities. Subsequently, they proposed adapting the initial layers (DSUs) of a pre-trained FR model to make it suitable for different heterogeneous scenarios.
Essentially, adapting the lower layers reduces the domain gap, and the whole pipeline is trained in a contrastive setting. However, the number of layers that need to be adapted is a hyper-parameter that has to be found with an extensive set of experiments. Furthermore, this approach requires the model to be adapted for each different architecture of pre-trained FR models. Recently, Cheema et al. [14] proposed a Cross Modality discriminator network (CMDN) for HFR. The architecture of CMDN follows a standard ResNet50 model with the squeeze and excitation module (SENet-50) [15]. Their approach uses a Deep Relational Discriminator (DRD) module to learn cross-domain matching. This DRD module is essentially a multi-layer perceptron (MLP) supervised by binary cross entropy loss (BCE). The learning of the CMDN module is supervised by the unit class loss which is a combination of triplet loss and a modified version of triplet loss that uses class means. The CMDN module (SENet-50) is initialized from a pre-trained face recognition backbone trained on the VGGFace2 dataset [16]. The CMDN network is further trained on another Visible-Thermal face dataset (IRIS face dataset [17]). The tuned CMDN network is further trained for the HFR task using a variant of triplet loss (Unit class loss). Now the embeddings obtained from two modalities (gallery and probe modalities) are concatenated for positive and negative pairs and an MLP model (DRD module) is trained on top of these concatenated embeddings with the BCE loss. The scoring is performed with probe-gallery pairs, and three different strategies were used for scoring 1) Using the embeddings and cosine loss, 2) using the output of the DRD module, and 3) score fusion of 1 and 2. Their evaluation strategy uses pairs of samples instead of probes against the gallery. They reported that the fusion model achieves better results.

B. Invariant feature based methods
Invariant feature-based methods aim to extract modality invariant features to match heterogeneous face images. Liao et al. [18] propose to use Difference of Gaussian (DoG) filters to highlight the structure of the images. Then, they use multi-scale block local binary patterns (MB-LBP) [19] as features and train a subspace-based face recognition system jointly on the samples of source and target domains. In [20], Klare et al. proposed a local feature-based discriminant analysis (LFDA) for the HFR task. They extracted scale-invariant feature transform (SIFT) [21] and multi-scale local binary pattern (MLBP) [22] feature descriptors as a patch unit from the sketch and VIS images for the HFR task. Zhang et al. [23] proposed a coupled information-theoretic encoding (CITE) extraction method to maximize the mutual information between the heterogeneous modalities in the quantized feature spaces. The local maximum quotient (LMQ) was proposed to extract invariant characteristics in crossmodality facial images in [24]. Several works have also used convolutional neural network (CNN) based methods [4], [7] to extract invariant features for the HFR task.

C. Synthesis based methods
Synthesis-based HFR methods [25], [26] attempt to synthesize the source domain (VIS in most of the cases) from the target modality, after synthesizing the source images typical face recognition networks can be used to perform the biometric matching. Authors in [27] proposed a patch-based synthesis approach to generate VIS to sketches and reverse using Multi-scale Markov Random Fields. The approach was evaluated using several face recognition methods such as Eigenfaces, Fisherfaces, dual space LDA, and so on. The work in [28] used Locally Linear Embedding (LLE) to learn a pixel-level mapping between VIS images and viewed sketches. Authors in [29] use CycleGAN [30] to transform images from the target domain to the source domain. Contrastive loss using a Siamese network is added during the training of CycleGAN to preserve the identity of faces. Moreover, the images are pre-processed before inputting to CycleGAN using [31] to reduce the domain gap further. Authors in [32], proposed a Generative Adversarial Network-based Visible Face Synthesis (GAN-VFS) method to synthesize photo-realistic visible face images from polarimetric images. Identity loss was combined with a perceptual loss in the training process. The synthesized visible images were further used by a VGG network to extract the embeddings. Their method was evaluated on the Polathermal dataset and achieved an average Equal Error Rate of 34.58%. With the advancement in the development of stable methods to train GANs, several recent approaches have been proposed using GANs for the synthesis of VIS images from another modality. The work in [26] treated HFR as a dual generation problem and proposed a Dual Variational Generation (DVG-Face) framework. A Dual generator was designed to learn the joint distribution of heterogeneous pairs and to generate heterogeneous pairs to address the lack of adequate data to train the HFR model. A pairwise identity preserving loss on the generated images was employed to ensure identity consistency. The generated images are used to train the HFR network in a contrastive setting. This approach achieved state-ofthe-art results in many challenging HFR benchmarks.

D. Limitations of current approaches
The majority of recent HFR methods proposed in the literature [26], [32] utilize synthesis-based methodologies. GAN-based synthesis methods have become popular due to their ability to generate high-quality images combined with breakthroughs in training them [33]. Furthermore, synthesis-based HFR approaches can benefit from the use of a pre-trained FR model, which eliminates the requirement to train the model with a huge amount of training data.
However, this technique falls short when it comes to actual use cases. The synthesize-based HFR must first generate an RGB image from the target modality image before passing it through a FR model to retrieve the embeddings at inference time. The synthesis process adds a significant amount of computing cost, which may restrict its usefulness in practical deployment scenarios. Moreover, synthesis-based (generative) algorithms are often trained to generate RGB images that are optimized for both perceptual and identity loss [26] (along with several other metrics). However, because the generated pictures are utilized in conjunction with a neural network, the perceptual quality of the generated images may not be significant in the context of the HFR task. In other words, it is critical to retain discriminative traits that potentially match those in the source class rather than generating high-fidelity images. This is particularly important as the generation process is often a much harder problem to solve when the amount of paired training data is limited.

A. Formal definition of HFR
Consider a domain D with samples X ∈ R d and a marginal distribution P (X) (with dimensionality-d). The task of an FR system T f r can be defined by a label space Y whose conditional probability is P (Y |X, Θ), where X and Y are random variables and Θ defines the model parameters. In the training phase of an FR system, P (Y |X, Θ) is typically learnt in a supervised fashion given a dataset of faces X = {x 1 , x 2 , ..., x n } together with their identities Y = {y 1 , y 2 , ..., y n }. Now consider the following heterogeneous face recognition (HFR) problem. Here we assume that we have two domains, source domain D s = {X s , P (X s )} and target domain D t = {X t , P (X t )} sharing the labels Y .
In a broad sense, the task in the HFR problem T hf r is to find aΘ, where P (Y |X s , Θ) = P (Y |X t ,Θ). The form and the nature in whichΘ is estimated varies with different HFR approaches.

B. Proposed approach
In the proposed approach, let us first assume that the samples from both domains, X s = {x 1 , x 2 , ..., x n } and X t = {x 1 , x 2 , ..., x n } from D s and D t with the shared set of labels Y = {y 1 , y 2 , ..., y n } are available. Also, assume that the parameters of an FR model Θ (we denote it as Θ F R in the following discussion) for the (VIS) model is available from D s . In our case, Θ F R is essentially the parameters of a pre-trained FR model trained using visible spectrum images. Following the discussion on the synthesis based HFR, we hypothesize that a module with a learnable set of parameters θ P DT can transform the target domain image to a new representation (X t = F PDT (X t )) to reduce the domain gap while retaining discriminative information. This new representation (X t ) can be used together with a pretrained FR model to achieve the HFR task.
To accomplish this task, we propose to prepend a small network module called "Prepended Domain Transformer " (PDT) to a pre-trained FR model. A schematic diagram of the proposed framework is depicted in Fig.  2. Essentially, we apply this module as a transformation to the target modality images, which generates a trans- This can also be viewed as an extension of the DSU [13] approach. Instead of adapting the lower layers as in DSU, we prepend a neural network block in front of a pre-trained FR model to handle domain-specific features. However, one of the main restrictions of DSU is that the local features that could be modified are restricted by the design of the face recognition (FR) network. One can only try to adapt or freeze layers already present in the FR network. It should also be noted that the architecture of the low-level layers in the FR network is designed for optimizing the performance in visible spectrum face recognition, and as such the lower layers (and their architecture) may not be optimal for the HFR task. This creates a bottleneck that limits the possibility of learning. In Prepended Domain Transformer there is more flexibility by changing the architecture of the PDT block, one can even change the local receptive field by changing the architecture of the PDT block. One could even perform a neural architecture search [35] to optimize the architecture of the PDT block for a specific heterogeneous use case. The architecture we use is more general which fits a larger set of heterogeneous tasks. In this sense, PDT is more flexible compared to DSU. Moreover, in PDT, the transformation is performed in the pixel space itself, meaning this approach can be applied to several FR architectures as a plug-in module. We have reimplemented DSU heterogeneous face recognition approach with the recent Iresnet100 pre-trained model as an additional baseline (DSU-Iresnet100).
The transformed image can then be passed to a pretrained FR model to get the embeddings for the HFR task. With the proposed approach, we can express the HFR problem in the following way: The parameters of PDT block (θ P DT ) can be learned in a supervised setting using back-propagation. In the forward pass for a tuple (X s , X t ), the X s image directly passes through the shared pre-trained FR network to produce the embedding. The target image (X t ) first passes through the PDT module (X t = F PDT (X t )), and then the transformed image passes through the shared pre-trained FR model to generate the embedding. Contrastive loss [36] is employed as the loss function in the training phase, to reduce the distance between these cross-modal embeddings when the identities are the same and increase the distance when the identities are different. The Contrastive loss is given as: Where Θ denotes the weights of the network, X s , X t denote the heterogeneous pairs and Y p the label of the pair, i.e., whether they belong to the same identity or not, m is the margin, and D W is the distance function between the embeddings of the two samples. The label Y p = 0, when the identities of subjects in X s and X t are the same, and Y p = 1 otherwise. The distance function D W can be computed as the Euclidean distance between the features extracted by the network.
The parameters of the shared FR model are kept frozen during the training and only the parameters of the PDT module are updated in the backward pass. At the end of the training, the model corresponding to minimum validation loss is selected which is used for the evaluations.

C. Architecture of the Prepended Domain Transformer (PDT) block
The Prepended Domain Transformer block is parameter-efficient and generic, allowing it to be applied to a wide range of HFR scenarios. The input and output of the PDT block are 'three-channel' images with the same size. This makes it easy to visualize the output of the proposed PDT module and to pass the modified images on to pre-trained FR models during inference. This module can also be readily "plugged in" to any pre-trained FR pipeline to convert it to an HFR pipeline. The architecture of the proposed PDT module is shown in Fig. 3. The initial part of the PDT bock utilizes the design principles inspired from the inception architecture [37]. We follow the idea of multi-scale processing by using different parallel branches with different kernel sizes. Parallel branches were necessary since the receptive field required for various heterogeneous settings differs, and having multi-scale features at the input level aids in a generic design with minimal computational complexity. There are four parallel paths from the input image, 1) a 1 × 1 filter, a 3 × 3 branch with two sequential filters, a 5 × 5 branch, and an average pooling branch. In each of these branches, 1 × 1 convolutions are used to reduce the number of output channels. We use ReLU activation after each of the convolution operations. A feature map is formed by concatenating the outputs from each of these branches, which consist of features obtained using filters with different receptive fields. The addition of an attention mechanism helps the network in deciding "what" and "where" to focus. A Convolutional Block Attention Module (CBAM) [38] attention module was added which achieves this in a simple and parameter efficient way. The CBAM block acts on a feature map along the channel as well as the spatial dimension in a sequential manner. The attention maps obtained are multiplied by the input feature map. The addition of the CBAM module helps in focusing on meaningful features along the channel and spatial dimensions making the proposed architecture robust to a wide variety of HFR scenarios. After the CBAM block, the channel dimension of the output feature map is still high and a 1 × 1 convolutional layer is added to reduce the channel dimension to three.
Overall the number of parameters to learn is merely 1.4K. The minimal design enables the network to focus on important features with a minimal parameter overhead. It is to be noted that, this module can be further optimized for specific heterogeneous scenarios.

D. Pre-trained FR backbone
As described earlier, the PDT module can be prepended to any pre-trained FR model. Though we perform most of the experiments with Iresnet100 model [39], this can be extended to many publicly available pre-trained FR models. For the sake of reproducibility, we used publicly available pre-trained face recognition models available from [40] 2 . These models were trained on MS-Celeb-1M-v1c 3 with 72,778 identities and about 3.28M images. In most cases, the pre-trained FR model accepts three-channel images with a resolution of 112×112. Faces are first aligned and cropped ensuring eye center coordinates fall on pre-fixed points. In the case of single-channel inputs (such as NIR, thermal, etc.), we replicate the same channel to three channels without making any changes to the network architecture. This was necessary since the pre-trained networks were designed to accept three-channel RGB images, and we didn't want to alter the layers/weights of the FR network that would change the performance of the pre-trained network in the RGB images in any manner.

E. Implementation details
The proposed framework is trained with Contrastive loss in a standard Siamese network setting [36]. The margin parameter is set as 2.0 in all the experiments. We used Adam Optimizer with a learning rate of 0.001 and trained the model for 20 epochs with a batch size of 90. The framework was implemented in PyTorch using the Bob library [41]. Except for the new PDT module added to the target channel branch, the whole pre-trained FR model is shared between source and target modalities in the Siamese network. Only the parameters of the PDT module are adapted during training, while the weights of the FR model remain frozen. The experiments are reproducible and the source code and the protocols will be made available publicly 4 .
The proposed method may be applied to a variety of HFR scenarios, including VIS-Thermal, VIS-SWIR, and VIS-Low resolution VIS, and so on. Furthermore, components of the proposed framework and the training routine are kept simple to demonstrate the efficacy of the proposed approach.

IV. EXPERIMENTS
An extensive set of experiments and ablation studies with the proposed approach are presented in this section. Primarily, we have evaluated VIS-Thermal HFR performance in four different datasets. We also compare the performance of the proposed approach against other heterogeneous settings such as VIS-NIR, VIS-SWIR, and so on. We further perform several studies evaluating the amount of data required for training the models and also the performance with different FR architectures.

A. Databases and Protocols
The datasets used in the evaluations are described in the following section.
Polathermal dataset: Pola Thermal dataset [42] -Polarimetric and Thermal Database is an HFR dataset collected by the U.S. Army Research Laboratory (ARL). The dataset contains polarimetric LWIR (long-wave infrared) imagery together with color images collected synchronously for 60 subjects. The dataset contains thermal imagery collected for conventional thermal images as well as polarimetric images. In this work, we use the conventional thermal images for our experiments using the reproducible protocols introduced in [13]. We follow the same five-fold partitions in which 60 subjects were split into a training set with 25 identities and 35 identities for testing. To compare different methods, the average Rank-1 identification rate is reported from the evaluation set of the five folds.
Tufts face dataset: The Tufts Face Database [43] provides face images captured with different modalities for the HFR task. Specifically, we use the thermal images provided in the dataset to evaluate VIS-Thermal HFR performance. Overall, there are a total of 113 identities comprising of 39 males and 74 females from different demographic regions. For each subject, images from different modalities are available. For comparison purposes, we follow the procedure followed by authors in [26], 50 identities are randomly selected from the data as the training set and the remaining subjects were used as the test set. We report the Rank-1 accuracies and Verification rates at false acceptance rates (FAR) 1% as well as 0.1% for comparison.
ARL-VTF dataset: In [5], authors made available the DEVCOM Army Research Laboratory Visible-Thermal Face Dataset (ARL-VTF). The dataset contains heterogeneous data from 395 subjects with three visible spectra as well as one thermal (long-wave infrared-LWIR) camera, with over 500,000 images altogether. The dataset contains variability in terms of expressions, pose, and eyewear. We evaluate the models with the protocols originally provided with the dataset. The dataset also provides annotations for face landmarks. Several protocols evaluating the effects of the pose, expressions, and eyewear are also provided with the dataset. The test set for each setting is fixed to enable direct comparisons with state-of-the-art methods. The naming of each protocol is as follows: Gallery and Probe protocols are designated "G" and "P" respectively. "V" and "T" denote the visible and thermal images. Categories such as "B", "E", and "P" denote baseline, expression, and pose sequences. Presence of the " * " symbol indicates any or all sequence categories in the protocol. A subject who does not possess glasses has the tag 0, andand + tags are present for subjects who have their glasses removed or worn respectively.
CASIA NIR-VIS 2.0 dataset: The CASIA NIR-VIS 2.0 Face Database [44] provides images of subjects captured with both visible spectrum as well as nearinfrared lighting, with a total of 725 identities. Each subject in the dataset has 1-22 visible images and 5-50 near-infrared (NIR) images. The experimental protocols provided uses a 10-fold cross-validation protocol with 360 identities used for training. The gallery and probe set for evaluation consist of 358 identities. The train and test sets are made with disjoint identities. We perform experiments in each fold and the mean and standard deviation of the performance metrics are reported.
SCFace dataset: The SCFace [45], dataset contains high quality mugshot for enrollment for FR. The probe samples correspond to surveillance scenarios coming from different cameras and are of low quality. Depending on the distance and quality of probe samples, four different protocols are present. They are close, medium, combined, and far. The "far" protocol is the most challenging one. The dataset contains 4,160 static images (in the visible and infrared spectrum) from 130 subjects.
MCXFace Dataset: We present a new HFR dataset named Multi-Channel Heterogeneous Face Recognition dataset (MCXFace). The dataset is derived from the HQ-WMCA dataset we have created earlier [46], [47]. The dataset contains images of 51 subjects collected in different channels under three different sessions and various illumination conditions. The channels available are color (RGB), thermal, near-infrared (850 nm), shortwave infrared (1300 nm), Depth, Stereo depth, and depth estimated from RGB images using 3DDFA [48] method. All the channels are registered spatially and temporally across all the modalities. The details about the sensors and data collection sessions can be found in our earlier work [46], [47]. In the MCXFace dataset, only bonafide samples are present. Further, the files are divided into train and dev sets with a disjoint set of identities to make experiments in different homogeneous and heterogeneous settings possible. For each of the protocols, we have created five different folds, by randomly dividing the subjects in train and dev partitions. Each of the protocol names is of the following form: < SOU RCE > − < T ARGET > −split < split >. In addition to the images, annotations for left and right eye centers for all the images are also provided. The dataset will be available publicly in the following link 5 .
CUFSF dataset: The CUHK Face Sketch FERET Database (CUFSF) [23] contains 1194 faces from the FERET dataset [49] and each image in the FERET dataset has a corresponding sketch image drawn by an artist. The dataset is challenging since the sketches have more shape exaggerations compared to the source photos. We follow the same protocol as reported in [50], where 250 identities were used for training the model, and the rest of 944 identities are reserved as the testing set. The Rank-1 accuracies are reported for comparison.

B. Metrics
To evaluate the models we follow several different metrics corresponding to previous literature. We have used a subset of metrics from the following performance metrics, Area Under the Curve (AUC), Equal Error Rate (EER), Rank-1 identification rate, Verification Rate with different false acceptance rates (0.01%, 0.1%, 1%, and 5%).

C. Experimental results
The experiments performed in the different datasets and the results are discussed in this section.
1) Experiments with Polathermal dataset: Here we perform experiments in thermal to visible recognition scenarios. The results in Table. I shows the average Rank-1 identification rate in the five protocols of the Polathermal 'thermal to visible protocols' (using the reproducible protocols in [13]). The reimplemented DSU-Iresnet100 baseline achieves better results compared to the original model from [13], indicating that the use of a better pre-trained model improves the results. It can be seen that the proposed approach achieves an average Rank-1 accuracy of 97.1% with a standard deviation of 1.3%, which is much greater than the results from other baselines reported in the literature.  [42] 78.72 % (-) Base-PLS in [42] 53.05% (-) ines LBPs + DoG features in [18] 36.8% (3.5) Repro-ISV in [51] 23.5% (1.1) ducible GFK in [52] 34.1% (2.9) baselines DSU(Best Result) [13] 76.3% ( 2) Experiments with Tufts face datasets: Table. II shows the performance of the proposed approach against other state-of-the-art methods in the VIS-Thermal protocol of the Tufts face dataset. The Tufts face dataset is very challenging due to pose and other types of variations. The challenging pose variations in Tufts face dataset are depicted in Fig. 4. Performance of even visible spectrum face recognition systems degrade in such extreme yaw angles, so it is expected that the performance of HFR would also degrade. Nevertheless, it can be seen that the proposed approach achieves the best results in verification rate, and is only second to DVG-Face [26] in Rank-1 accuracy, showing the effectiveness of the proposed approach.
3) Experiments with CASIA-VIS-NIR 2.0 dataset: Though most of the focus is given to Thermal-VIS HFR, we perform experiments in the CASIA-VIS-NIR 2.0 dataset to showcase the effectiveness of the proposed approach in other heterogeneous scenarios, specifically  NIR-VIS recognition. As it can be seen from the baselines, the domain gap appears to be less in this scenario and even some of the pre-trained FR model trained using VIS modality achieves reasonable performance. Due to this, the evaluations are done with even tighter thresholds and VR@FAR=0.1% and VR@FAR=0.01% are used for comparison. There are 10 sub-protocols in the dataset, and we report the mean and standard deviation of all ten folds in the comparison. The results are compared in Tab. III. From the results, it can be seen that the proposed approach achieves the best performance compared to other state-of-the-art methods. The superior performance indicates that our framework generalizes across several heterogeneous scenarios.

4) Experiments with SCFace dataset:
We have performed a set of experiments on the SCFace dataset using the protocols provided for visible images in the dataset. In this dataset, the heterogeneity arises from the quality difference between gallery and probe images, i.e., gallery images are high-resolution mugshots and probe images and low-resolution images coming from a surveillance camera. The results are tabulated in Table. IV, the values reported corresponds to the "evaluation" set of the protocols. The baseline model is a pre-trained Iresnet100 model as used in other experiments, and the rows with PDT denotes the proposed model where the PDT model is trained using contrastive training. Results with reimplemented DSU-Iresnet100 model is also added for comparison. It can be seen that with the proposed approach the performance of the baseline model improves. The improvement is obvious in the "far" protocol where the quality of the probe images is very poor. It can be seen the PDT module, in this case, helps in learning quality and resolution invariant features improving the results.

5) Experiments with MCXFace dataset:
The experiments in the MCXFace dataset gives a unique opportunity to evaluate the performance of the models in several heterogeneous scenarios, including VIS-Thermal, VIS-Depth, VIS-SWIR, VIS-NIR, and so on. The results containing the average performance among five folds are shown in Table. V. For each modality, (eg. VIS-Thermal), the values reported are aggregated over the five folds in the dataset. In each protocol, we perform a baseline evaluation which is essentially using the pretrained Iresnet100 FR model on the new modality. The lower baseline performance indicates a large domain gap. For example, running a vanilla FR model itself achieves excellent performance for NIR and SWIR channels, whereas the performance is poor for depth and thermal channels. In the Table. V, the rows with PDT show the results with our proposed approach. In addition, the reimplemented DSU-Iresnet100 model is also added as a baseline. It can be seen that the performance improves greatly in the case of the thermal channel. However, for the depth channel, even though the performance improves, the final performance is not satisfactory, it indicates that the depth channel needs a different treatment, and using the depth data as range images may not be the optimal choice. Using representations like normals or point clouds could be well suited for the depth modality.

VIS-NIR
Baseline 100.00 ± 0.00 0.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 DSU-Iresnet100 100.00 ± 0.00 0.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 PDT 100.00 ± 0.00 0.00 ± 0.00 100.00 ± 0.00 100.00 ± 0.00 6) Experiments with ARL-VTF dataset: The ARL-VTF offers a large-scale VIS-Thermal dataset with different variations such as pose, expressions, and eyewear making it possible to perform experiments to evaluate the effect of different factors. We have used the standard protocols shipped with the datasets for our evaluations. The test sets of different protocols are fixed and the best model selected from the cross-validation was used to evaluate the performance. The performance of the proposed approach against the state-of-the-art methods is shown in Table. VI. It can be seen that the proposed approach achieves state-of-the-art performance in most cases. The performance improvement is very clear in challenging protocols with pose variations (P TP-). In the P T B0 − G V B0− protocol, where all the images are frontal, the proposed method achieves a verification rate of 98.57% for FAR1% (and a corresponding Rank-1 accuracy of 99.14%), whereas in the P T P 0−G V B0− protocol where the samples contain pose variations the verification rate drops to 60.80% ( and a corresponding Rank-1 accuracy of 60.23%). The Rank-1 accuracy for just frontal faces is 99.14% whereas with pose variations it is 60.23%. It can be seen that this drop is present in all other methods as well, indicating that HFR also suffers performance degradation with pose variations as with homogeneous face recognition. 7) Experiments with CUFSF dataset: Here we perform experiments with the challenging sketch to photo recognition task. The Rank-1 accuracies obtained with the baseline and the other methods are shown in Table. VII, using the protocols in [50]. It can be seen that the proposed approach obtains a Rank-1 accuracy of 71.08%. Nevertheless, the absolute accuracy in the sketch to photo recognition is low. Several methods in literature have shown (on a different protocol) that the sketch recognition performance could be improved with specifically designed features [68] and specially designed neural network models [69]. The sketch modality is very different compared to other heterogeneous imaging modalities we have considered so far like thermal, nearinfrared, and so on. Though the CUFSF dataset contains viewed hand-drawn sketch images [70] which appears to be holistically similar for humans, there exists a domain gap in the context of an automatic face recognition system as evidenced from the baseline performance. The pretrained model achieves a Rank-1 accuracy of 56.57%, indicating the domain gap. The modalities like thermal, NIR, and SWIR, were all the "imaging" modalities and it shares the same high-level representation of the face, but a different aspect of the face. In sketch recognition, the sketch images contain exaggerations depending on the artist, and may not optimally preserve the discriminative information which a face recognition network might be looking for. This could explain the larger performance gap of sketch-photo recognition compared to the performance in other imaging modalities.

D. Analysis of the Framework
To further understand the effectiveness of the proposed approach, we have performed additional experiments on the ARL-VTF dataset due to the large number of subjects present. Specifically, we performed experiments to understand the effect of the amount of training data, the type of supervision used in training, and the performance of different FR architectures. All these experiments were performed on the G V B0 − P T B− protocol of the ARL-VTF dataset. 1) Performance with limited amount training data: The amount of paired training data available to train HFR model is often limited and expensive to acquire. In this regard, we perform a set of experiments to understand how the amount of training data available to train the model affects its performance. We used the ARL-VTF dataset for this set of experiments because it had a larger number of subjects. The test samples are kept the same for this set of experiments, with the only difference being the amount (or percentage) of training and validation samples. We start with using 100% of the training data for training the PDT module and gradually lower the number of samples in the intervals of 10%, and finally 1% intervals. For context, we also note the number of subjects in the training set for these scenarios. The results of this set of experiments are tabulated in Table. VIII. Remarkably, the proposed approach achieves a Rank-1 accuracy of 94.67% with just 2% percentage of the training data, for context, just with data from 4 subjects. This could be due to the parameter efficiency of our approach. The learnable component of the PDT block contains approximately just 1.4K parameters, and hence requires a very minimal amount of data to achieve good performance. This is an important observation, as HFR datasets are often small in size.
2) Experiments with unpaired images: So far in the experiments, the networks were trained using contrastive loss in a supervised manner. We assume that paired heterogeneous samples are available at the training time.
Here we try to emulate an unpaired setting, meaning we do not have paired heterogeneous samples, and we do not have the identity information of the samples. In this scenario, we supervise our framework to match the feature distributions of source and target modalities using Maximum Mean Discrepancy (MMD) [71] loss. We have experimented with applying the MMD loss in three different ways, 1) the MMD loss was applied to both the transformed and source images, thereby attempting to match the FR network's inputs.(ip), 2) The MMD loss is applied to the embeddings obtained from the FR network (op). and 3) where both output embeddings and inputs were supervised by MMD loss (ip+op). The results obtained from this set of experiments are shown in Table. IX. The first row shows the baseline with a pre-trained Iresnet100 and the last row shows the results with supervised learning with the PDT approach. From the results, it can be seen that even in unpaired settings the proposed framework performs reasonably well. The best results are obtained when MMD is used on both the input and output. This indicates that the proposed framework can be employed even if paired samples aren't available. It was possible to achieve reasonable performance just by matching the distributions of the source and target modality features. However, as previously discussed, having a small number of labeled samples improves performance significantly when compared to using unpaired samples in training. 3) Experiments with different Face Recognition models: We have used Iresnet100 as the pre-trained FR model for all the experiments discussed in the previous section. In this section, we investigate whether the proposed approach generalizes to other FR architectures. We again perform these experiments in the ARL-VTF dataset using the G V B0 − P T B− protocol. For each We first run these pre-trained models without the PDT module to get a baseline performance. The results with the trained PDT module are also shown in Table. X. From the table, it can be seen that the proposed approach works well with all the architectures used. The complexity and performance of each of these models are different. From these results, it is clear that the proposed approach can be used with any FR model architecture, given the PDT module is trained together with it. Pretrained FR models can thus be chosen for HFR based on a tradeoff between accuracy and computational complexity.
4) Generalizability of the PDT weights across architectures: In the previous sub-section, it has been shown that the proposed approach works with different FR models despite the differences in architecture and complexity. It is to be noted that, the architecture of the PDT remains the same even though we used different FR architectures. We further investigated whether the PDT module learned for one FR architecture would work for another. These experiments could provide insights into the learned transformations and could also indicate whether PDT modules could be used with black-box models. For example, in Table. XI, AttentionNet(C) (column)-EfficientNet(A) (row), means that the PDT was added to AttentionNet model during training, the weights of the PDT module trained with AttentionNet (checkpoint) was used in conjunction with EfficientNet architecture for evaluation. From the results in Table. XI, it can be seen that the proposed approach works 6 https://github.com/JDAI-CV/FaceX-Zoo well for most of the scenarios with the notable exception of GhostNet [72], which could be due to GhostNet's unique architecture. GhostNet modules try to reduce the computational complexity of a network by reducing the filters needed to compute redundant feature maps. This is achieved by creating additional feature maps using cheap linear operations such as depth-wise separable convolutions on the output feature maps. The original feature maps and the newly created (ghost) features are concatenated in the ghost module to maintain the feature map sizes. We suspect that this operation makes the PDT learned together with GhostNet specifically optimized to the filters in the pre-trained backbone. This results in the low performance of the PDT module trained with the GhostNet module when it is used with other architectures. In summary, while the PDT module works in most cases, the parameters of the PDT module are ultimately learned in conjunction with a specific architecture, and it may not always generalize when used with black-box models. However, using multiple FR models to learn the PDT weights in a distillation framework could improve the performance in the case of black-box FR models. This is left as future work and is beyond the scope of the current contribution.

5) Visualization of the intermediate features:
The t-SNE plots of thermal and visible images at various stages in the pipeline are shown in Fig. 5. The thermal and visual modalities are readily distinguished in the t-SNE plots at the input stage. The heterogeneous samples appear to form different distributions even after the PDT stage. As we progress in the layers in the feature maps of the CNN, the distribution of thermal and visible features becomes more aligned. Towards the last layers and the final embedding output, the modalities align very well, i.e., the embeddings of the same identity in the thermal and visible spectrum grow closer in the embedding space, which is exactly what we need for the HFR task. One important observation is that even after the PDT block the distribution of features is disjoint.Despite the distribution disparity, the transformed image following the PDT block helps align the embeddings in the final layers.
Since the feature maps after the PDT module are also three-channel images with the same resolution, they can be used to inspect the PDT module's output. For example, Fig. 6, shows the thermal, visible, and transformed-thermal image of the same person. In a broad sense, the face embeddings extracted from the transformed images and visible spectrum images have a higher match score. Intuitively, one would expect the transformed image to resemble the VIS image in order for this to occur; however, the results show that this is not the case; instead, the transformed image only needs to preserve discriminative information for the HFR task .  TABLE XI  RANK-1 ACCURACIES FOR THE CROSS-TEST BETWEEN DIFFERENT ARCHITECTURES, ROWS ARE THE ARCHITECTURE USED FOR  EVALUATION (A), AND COLUMNS ARE THE ARCHITECTURE USED FOR TRAINING THE PDT MODULE (C), RESULTS FOR THE SAME  TRAIN-TEST ARCHITECTURE

V. DISCUSSIONS
As opposed to the computationally expensive synthesis based methods, the proposed approach introduces a simple, computationally simpler, plug and play module for heterogeneous face recognition that can be trained with a minimal number of samples. The proposed approach can be used even in scenarios where paired samples are not available. The PDT approach achieves this by learning a parameter efficient transformation in the pixel domain that reduces the domain gap while retaining the discriminative information.
The proposed approach achieves state-of-the-art performance in many challenging benchmarks, as demonstrated by experiments in different thermal to VIS as well as other heterogeneous protocols. The new PDT module added is parameter efficient and is generic enough for several heterogeneous scenarios. As evidenced from the experiments in Table. VIII, the amount of training data required for our framework is very small, this makes it suitable for the real-world data starved heterogeneous settings. The approach was also found to perform well even in the absence of paired samples, this is of practical importance in heterogeneous scenarios where paired data is not readily available. The approach itself is modular and can be added to any pre-trained FR model. To summarize, we show that prepending a learnable neural network module to a pre-trained FR model yields stateof-the-art performance in a variety of challenging HFR scenarios.

A. Limitations and Future directions
Currently, the architecture of the PDT block is designed to be general so as to serve a wide range of heterogeneous scenarios. The current design of PDT provides flexibility in terms of the receptive field due to the multi-branch architecture and CBAM module. This simple approach achieves comparable, and in many cases outperforms many computationally expensive state-ofthe-art models. The receptive field of the branches in PDT is a design parameter that can be optimized even further. It could be possible to optimize the architecture of PDT for a specific heterogeneous scenario, which might boost the performance even further. Since the Prepended Domain Transformer framework depends on the performance of the pre-trained network used in the framework, the overall performance would be impacted by the performance of the pre-trained network. For example, most of the standard pre-trained FR models struggle with extreme yaw angles and profile faces, meaning such a network would result in lower HFR performance when there is extreme yaw angles present (as we see from the results in Tufts face dataset). Nevertheless, the proposed approach is generic enough to be applied to newer and robust face recognition models. Further, the PDT approach is better suited for heterogeneous "imaging" modalities, as they share more structural similarities with visible face images. Heterogeneous modalities like drawn sketches do not satisfy this criterion and may be harder to adapt in the image domain. Methods like generative approaches might be better suited to these scenarios. Though the PDT blocks trained with one architecture do not necessarily work well with other architectures, this could be enhanced to work with blackbox models by employing multiple models in a teacherstudent fashion. The proposed approach can also be combined with GAN-based generation methods, and can also be improved by using triplet or quadruplet loss functions. The proposed approach can be further extended with more tuning of the PDT architecture with the likes of neural architecture search [35], training schedules, data augmentation, triplet training, and so on. However, as the title suggests, we hope that this approach will serve as a simple yet strong baseline motivating further research in heterogeneous face recognition.

VI. CONCLUSIONS
In this work, we have proposed a simple yet effective framework for heterogeneous face recognition. Essentially, to convert a pre-trained FR model to an HFR network we just prepend a novel neural network module for the target modality. The new module, called Prepended Domain Transformer (PDT) is parameter efficient and does not require a lot of samples to train. Since the method performs well across a variety of FR architectures, it can be used to convert any face recognition model to a heterogeneous one. The design of the framework, training schedule, loss functions, and parameter selection are intentionally left simple to demonstrate the efficacy of the proposed approach. These can be further tuned to improve the results further. The proposed approach was found to outperform state-of-theart methods in many challenging heterogeneous datasets. Further, we also introduce MCXFace heterogeneous face recognition dataset which contains multiple modalities which can be used for HFR evaluations. Lastly, the source code, protocols, and datasets used are publicly available to make further extensions of the work possible.