{'fraction_non_alphanumeric': 0.02817281728172817, 'fraction_numerical': 0.01107110711071107, 'mean_word_length': 5.255912162162162, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MiDeCon: Unsupervised and Accurate Fingerprint and Minutia Quality Assessment based on Minutia Detection Confidence

An essential factor to achieve high accuracies in fingerprint recognition systems is the quality of its samples. Previous works mainly proposed supervised solutions based on image properties that neglects the minutiae extraction process, despite that most fingerprint recognition techniques are based on detected minutiae. Consequently, a fingerprint image might be assigned a high quality even if the utilized minutia extractor produces unreliable information. In this work, we propose a novel concept of assessing minutia and fingerprint quality based on minutia detection confidence (MiDeCon). MiDeCon can be applied to an arbitrary deep learning based minutia extractor and does not require quality labels for learning. We propose using the detection reliability of the extracted minutia as its quality indicator. By combining the highest minutia qualities, MiDeCon also accurately determines the quality of a full fingerprint. Experiments are conducted on the publicly available databases of the FVC 2006 and compared against several baselines, such as NIST's widely-used fingerprint image quality software NFIQ1 and NFIQ2. The results demonstrate a significantly stronger quality assessment performance of the proposed MiDeCon-qualities as related works on both, minutia- and fingerprint-level. The implementation is publicly available.


Introduction
Fingerprints are one of the most used biometric modalities with a wide range of applications such as unlocking consumer smartphones or identity recognition in forensic investigations [19,10]. The most determinant factor to achieve high recognition accuracy in fingerprint recognition systems is the fingerprint sample quality [15]. Therefore, many large scale biometric deployments require measuring quality scores, such as the Unique Identification Authority of India (UIDAI) [24] and the European Union Visa Information System (VIS) [12].
In general, the quality of a fingerprint sample is defined as its utility for recognition [35]. The fingerprint quality is the key property for a high recognition performance by ensuring reliable enrolment and matching. Especially in forensic scenarios, quality assessment is of great significance since the assessment of the strength of evidence is a central activity in forensic casework [46]. However, forensic use-cases mainly have to deal with partial fingerprints and thus, the quality assessment of single minutiae are of great interest [13,30]. Current works on fingerprint quality assessment, such as [13,41], focused on supervised classifiers to predict quality given the fingerprint image. However, these approaches often neglect the quality of the extracted minutiae for the fact that current fingerprint recognition systems are mainly based on this minutiae information [43]. Previous works on the quality assessment of single minutiae are based on hand-crafted features and define the quality of a minutia through properties of the image quality [23,42,21]. Therefore, these solutions do not consider how well the utilized minutia extractor can extract true minutiae. Consequently, even if the image around a minutia is of high quality, the extraction of reliable minutia information might fail and thus, the quality of this minutia is wrongly estimated as high.
In this work, we propose a novel concept of measuring minutia and fingerprint quality based on minutia detection confidence (MiDeCon). The proposed concept can be applied to an arbitrary neural network based minutia extractor and does not rely on quality labels for training. To estimate the quality of a minutia, MiDeCon computes the extractor's confidence that the detected minutia is real. This confidence is determined over the agreement of stochastic variants of the extraction network created by random node eliminations. To estimate the quality of a full fingerprint, MiDeCon combines the highest minutia qualities resulting in a continuous quality value of the fingerprint.
The experiments were conducted on the four publicly available databases of the FVC 2006 [5] and make utilizes the widely-used NFIQ1 [36] and NFIQ2 [34] developed by the National Institute of Standards and Technology (NIST) as baselines. The results demonstrate significantly stronger fingerprint and minutia quality assessment performances of the proposed unsupervised approach in comparison to established approaches. This holds on all investigated sensor types (electric field, optical, and thermal sensors). MiDe-Con is publicly available under the following link * . The main novelty of the proposed MiDeCon approach is that it: • Does not require quality labels for training -Previous works often rely on error-prone labelling mechanisms without a clear definition of quality. Our approach avoids the use of inaccurate quality labels by using the minutia detection confidence as a quality estimate. Moreover, the training state can be completely avoided if pre-trained minutiae extraction neural network trained with dropout [33] is available.
• Considers difficulties in the minutiae extraction -Previous works estimates the quality of a fingerprint based on the properties of the image neglecting the minutiae extraction process. However, the extraction process might face difficulties that are not considered in the image properties and thus, produce unreliable minutia information. Our solution defines quality through the prediction confidence of the extractor and thus, considers this problem.
• Includes quality assessment of single minutiae -Unlike previous works, our solution assesses the quality of full fingerprints as well as the quality of single minutiae. This is specifically useful in forensic scenarios where forensic examiners aim to find reliable minutiae suitable for identification.

Minutia Quality Assessment
Despite that many of the best fingerprint recognition algorithms are based on minutiae information [43], only a few works proposed solutions for estimating the quality of single minutiae. In 2005, Kim et al. [20] proposed a minutia quality score by analysing the neighbouring ridge structure of the minutia in a thinned fingerprint image. In [42], Uz et al. defined the quality of a minutia based on its frequency it occurs in the different impressions. Maki et al. proposed the Minutia Confidence Index [23]. This index estimates the * https://github.com/pterhoer/ FingerprintImageQuality quality of a minutia based on the location, type, and orientation of other minutiae. NIST roll out Mindtct, a minutiae detection algorithm that additionally states the qualities of detected minutiae by combing low contrast, low flow, and high curve maps of the fingerprint.
To summarize, previous works define the quality of a minutia through the image quality around the minutia. However, this does not consider the extraction process which might result in error-prone outputs, e.g. due to model bias. Therefore, we propose defining the quality of a minutia as the detection confidence of the minutia extractor.

Fingerprint Quality Assessment
In contrast to minutia quality assessment, more research was done on the quality assessment on full fingerprints. Traditional approaches are based on the use of local [32,37,3,45,31] and global hand-crafted [22,7,27,11,9,28] features, or approaches that combine both kinds of features [1,44,29,8]. Besides the use of hand-crafted features, deep feature-learning solutions [41,26,13] were proposed recently. These approaches aim to predict quality by supervised training of a neural network classifier.
The engagement of NIST in this aspect is also worth mentioning. They investigated, along with their partners, 155 quality features from the literature and selected the best 14 to train classification models to estimate the quality of a fingerprint. This resulted in of NFIQ1 and NFIQ2 [2,21].
In concept, current solutions are training supervised classifiers to estimate the quality of a fingerprint image. Since current fingerprint recognition systems are still based on minutiae information [43] we argue that the quality of a fingerprint must reflect the qualities of its extracted minutiae. Therefore, we propose to estimate the quality of a fingerprint by combining minutia qualities defined over the detection confidences of the minutia extractor.

Methodology
We define the quality of a single minutia by its detection reliability of the minutia extractor that is based on the agreement between stochastic variants of the extractor network. Combing the detection reliabilities of the highestquality minutiae, MiDeCon assesses the quality of a given fingerprint. The conceptual idea of the proposed approach is shown in Figure 1. In the following, we provide a detailed description of how MiDeCon determines the quality of single minutiae and fingerprints given the minutiae extractor MinutiaeNet [25]. MinutiaeNet consists of two networks, CoarseNet and FineNet. While CoarseNet determines and extracts patches of potential minutiae in a given Fingerprint, FineNet estimates if the candidate patch consists of a minutia and extracts the minutia information from the candidate patches, such as location, orientation, and minutia type. Figure 1: Conceptual idea of MiDeCon: the system's minutiae extraction network is modified to additionally provide the minutia detection reliabilities based on the agreement of stochastic network variants. We propose using the detection reliability r (i) of a minutia m (i) as its quality indicator. To determine the quality of the whole fingerprint, the n highest detection reliabilities are combined. By integrating the minutia extractor in the quality assessment process, the proposed concept additionally takes into account difficulties that appear during extraction.

Assessing Minutia Quality
Our minutia quality assessment approach is based on the output of FineNet [25] that decides if the candidate patch of the fingerprint consists of a minutia or not. We calculate a novel minutia detection confidence based of this network and use this as the quality of the detected minutia.
More formally, given a fingerprint capture f and a neural network minutiae extractor ME, such as FineNet [25], we propose to use the detection reliability r (i) of the i th minutia as its quality indicator q (i) . The detection reliability follows the idea of Terhörst et al. [39] by using stochastic forward passes in a dropout-reduced neural network minutiae extractor ME trained with dropout [33]. Instead of propagating the input patch one time through the network ME, the input image is propagated m = 100 times through different variations of the network approximating a Gaussian process [14]. The network variants of ME are created by randomly eliminating nodes using similar dropout patterns as during the model's training. For each of the m stochastic forward passes through such a model variant, a stochastic softmax prediction x j is given describing the probability that the given patch contains a minutia assuming a Boltzmann distribution. This results in a set of stochastic prediction ..,m that is used to estimate the detection reliability of a given minutia m (i) . The detection reliability

MOD
, for a minutia m (i) consists of two parts, the measure of centrality (MOC) and the measure of dispersion (MOD). The MOC computes the mean of the stochastic outputs aiming at utilizing the probability interpretation of softmax assuming a Boltzmann distribution. The MOD quantifies the agreement of the different network variants by calculating the mean pairwise absolute distance of all scores in x (i) . A higher agreement and a higher probability indicate a higher detection reliability and vice versa. MOC and MOD are equally weighted since both measures result in values of similar magnitudes. We propose using the minutia detection confidence r (i) = rel(x (i) ) of the minutia m (i) as its quality indicator q (i) . The advantage of using the minutia detection confidence as the quality measure of a single minutia is that, unlike previous works, it takes into account the minutia extraction process, rather than solely depend on the properties of the image patch around the detection. Neglecting this process can result in over-estimations of the quality when the extractor is not able to reliably detect a minutia of a high-quality image or in under-estimations of the quality in case that the extractor is still able to capture reliable minutia information from a lower-quality image. Our proposed solution defines the quality of a single minutia through the prediction confidence of the extractor and thus, considers this issue.

Assessing Fingerprint Quality
Given a fingerprint f with extracted minutiae information {m (i) } and its corresponding detection reliabilities r (i) , we assume that the quality of a fingerprint is based on the highest detection reliabilities of its minutiae. Therefore, we define the quality Q(f ) of a fingerprint f as where E is mean operator over the first n entries of the sorted listr(f ). More precisely,r(f ) is the sorted list of the minutiae reliabilities {r (i) } of fingerprint f such that ∀ l ∈ [1, . . . N − 1] : r (l) ≥ r (l+1) is satisfied. Consequently, we define the quality Q of a fingerprint capture f as the mean of its n highest minutia (detection) reliabilities.
We choose n = 20 since this reflects the average number of minutiae in a partial fingerprint and allows the proposed methodology to be used in forensic scenarios [38]. However, this number can be arbitrarily adapted to the target application.
The advantage of this quality assessment approach is its direct use of the minutia qualities defined over their detection reliabilities. Most of the current fingerprint recognition systems are based on minutiae information [43] and thus, measuring the quality of the direct input of these systems is more likely to succeed in estimating suitable quality values than approaches that only rely on properties of the image quality.

Databases and Model Training
Th experiments are done on the publicly available benchmarks of the FVC 2006 [5]. This allows us to evaluate quality assessment over multiple sensor types. FVC2006 consists of 7200 fingerprint captures from 600 different fingers over four databases (DB). The images were collected by four different organizations in two different countries. Each database was captured with a different sensor type: DB1 (electric field sensor, 250 dpi), DB2 (optical sensor, 596 dpi), DB3 (thermal sweeping sensor, 500 dpi), and DB4 (synthetic, 500 dpi). DB4 consists of synthetic data that was generated with SFinFe [4]. Each database consists of subject-disjoint test and training sets, resulting in 6720 fingerprint images for testing and 480 for training.
For the experiments, we apply MiDeCon on the FineNet network * [25]. This network was originally not trained with dropout. Therefore, we retrained the last fully connected layer with dropout on the training set following the training procedure described in [25]. Since only one layer was trained, we adapted the initial learning rate to lr = 10 −4 .

Evaluation Metrics
To evaluate the fingerprint quality assessment performance, we follow the methodology by Grother et al. [17] using error versus reject curves. These curves show a verification error-rate over the fraction of unconsidered fingerprint images. Based on the predicted quality values, these unconsidered images are those with the lowest predicted quality and the error rate is calculated on the remaining images. Error versus reject curves indicates good quality estimation when the verification error decreases consistently for higher ratios of unconsidered images [40].
However, for quality assessment of single minutiae this assessment approach does not fit since the number of detected minutia strongly varies between different algorithms. Therefore, we determine the quality assessment performance of single minutiae by considering only a fixed number  of the highest quality minutiae for recognition. Assuming a lower verification error is achieved if the considered minutiae are of higher quality, this allows us to evaluate the quality assessment performance of single minutiae.
The verification error rates within the error versus reject curves are reported in terms of false non-match rate (FNMR) at a fixed false match rate (FMR) as specified in the international standard [18].

Workflow and Baselines
Since many of the recognition systems are still based on minutiae information [43], we utilize two frequently used fingerprint recognition systems, Minutia Cylinder-Codes * https://github.com/luannd/MinutiaeNet (MCC) from [6] and Bozorth3 [21] from NIST Biometric Image Software (NBIS). The minutiae-extraction for the matcher is based on MinutiaeNet [25] and NIST's Mindtct [21]. To evaluate the quality assessment performance, we compare the proposed solution against the widely-used fingerprint quality assessment solutions, NFIQ1 [36] and NFIQ2 [34] from NIST. Both approaches have become a standard in estimating fingerprint quality [16].

Minutia Quality Assessment
In Table 1, the minutia quality assessment performance is compared in terms of FNMR at FMRs of 10 −2 and 10 −3 using only a fixed number the highest quality minutiae. Moreover, the quality assessment approaches are compared on the four databases containing fingerprint images captured with different sensors. Generally, considering more minutiae for the recognition process decreases the recognition error. However, strong differences in the performance are observed for the different quality assessment methods. Both utilized fingerprint recognition methods (Bo-zorth3 and MCC) focus on local minutiae neighbourhoods. Since Mindtct assigns minutiae qualities over a quality map, neighbouring minutiae have similar quality estimates. This property favours Mindtct when using Bozorth3 or MCC, because these focus on local neighbourhoods, and explains the weaker performance of CoarseNet. Moreover, it shows that the natural choice of using the model's softmax output as the minutia quality does not work.
On synthetically generated fingerprint data, the quality estimates based on the Mindtct's hand-crafted features achieve the best results. On data captured from real sensors, our proposed method based on the detection confidence of single minutiae constantly achieves the lowest recognition errors. This holds for all investigated sensor types, i.e. electric field, optical, and thermal sensors (DB1, DB2, DB3) and demonstrates a strong minutiae quality assessment performance of the proposed approach. Figures 2 and 3 compare the quality assessment performance of our proposed approach with NFIQ and NFIQ2 on the Bozorth3 (Figure 2) and MCC (Figure 3) matchers. The quality assessment performance is shown as error-vs-reject curves as described in Section 4.2. The corresponding verification error is shown in terms of FNMR at an FMR of 10 −1 (a-d), 10 −2 (e-h), and 10 −3 (i-l). Moreover, the effectiveness of the different quality assessment solutions is analysed on three different sensor types (electric field, optical, and thermal) and synthetic data.

Fingerprint Quality Assessment
In all investigated scenarios, all quality assessment solutions show a solid and (relatively) stable performance. Our proposed method based on the combination of minu- Table 1: Evaluating minutia quality assessment -only a certain number of the highest quality minutiae are used for recognition. The recognition performance is reported at two FMRs on the Bozorth3 and the MCC matcher. Each DB was captured with a different sensor. Our proposed methodology based on minutia detection confidence shows lower recognition errors than related works [25,21] in all cases, except on the less-relevant synthetic data (DB4). This demonstrates the strong quality estimation performance for single minutiae of the proposed method. Just on the less-relevant synthetically generated data using the MCC matcher, NFIQ2 shows better quality estimates. However, on data captured from real fingerprints, the proposed methodology constantly shows a significantly stronger quality assessment performance than the baselines. Especially for electric field and optical sensors, the proposed approach outperforms the baselines by a large margin.

Conclusion
Previous works on the quality assessment of single minutiae or full fingerprint captures are either learning models based on error-prone quality labels or make use of naive hand-crafted features. Although many fingerprint recognition systems are still based on minutiae information, previous works on quality assessment neglected the minutia extraction process. In this work, we proposed a novel concept of measuring the quality of fingerprints and single minutiae based on an arbitrary minutia extraction network. The proposed method, MiDeCon, uses the minutia detection confidence of the utilized minutia extractor as the quality indicator for this minutia. This confidence is based on the agreement of stochastic variants of the extraction network. By combining the highest minutia qualities of a fingerprint, it successfully determines the fingerprint . Especially on the real-world sensor data, the proposed approach outperforms the widely-used NFIQ and NFIQ2 baselines. This holds true for all investigated sensor-types.
quality of this capture. Experiments on the publicly available FVC 2006 databases demonstrate that the proposed approach strongly outperforms the baselines, such as Mindtct and CoarseNet for quality assessment of single minutia and NIST's widely-used fingerprint image software NFIQ1 and NFIQ2 for fingerprint quality assessment. The strong performance of MiDeCon is especially observed when facing real-world sensor data, such as electric field, optical, and thermal sensors. In contrast to previous works, MiDeCon jointly (a) does not rely on quality labels for training, (b) produces continuous quality values, (c) considers difficulties in the minutiae extraction process, and (d) assesses the quality of both, full fingerprints and single minutia.