{'fraction_non_alphanumeric': 0.03248617735696827, 'fraction_numerical': 0.023706129422652523, 'mean_word_length': 4.6309460181721, 'pattern_counts': {'":': 9, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A new baseline for retinal vessel segmentation: Numerical identification and correction of methodological inconsistencies affecting 100+ papers

In the last 15 years, the segmentation of vessels in retinal images has become an intensively researched problem in medical imaging, with hundreds of algorithms published. One of the de facto benchmarking data sets of vessel segmentation techniques is the DRIVE data set. Since DRIVE contains a predefined split of training and test images, the published performance results of the various segmentation techniques should provide a reliable ranking of the algorithms. Including more than 100 papers in the study, we performed a detailed numerical analysis of the coherence of the published performance scores. We found inconsistencies in the reported scores related to the use of the field of view (FoV), which has a significant impact on the performance scores. We attempted to eliminate the biases using numerical techniques to provide a more realistic picture of the state of the art. Based on the results, we have formulated several findings, most notably: despite the well-defined test set of DRIVE, most rankings in published papers are based on non-comparable figures; in contrast to the near-perfect accuracy scores reported in the literature, the highest accuracy score achieved to date is 0.9582 in the FoV region, which is 1% higher than that of human annotators. The methods we have developed for identifying and eliminating the evaluation biases can be easily applied to other domains where similar problems may arise.


Introduction
Retinal images provide a noninvasive way to detect and monitor various conditions and diseases of the eye and body, such as diabetes-related retinopathies, macular degeneration, glaucoma and hypertension (Abramoff et al., 2010). Consequently, the computer-aided (semi-)automatic analysis and screening of retinal images has become a highly popular field of study in medical imaging, with thousands of papers and algorithms focusing on various anatomical parts and degenerations observable in retinal images (Abramoff et al., 2010;Trucco et al., 2019). The popularity of the field can be illustrated by the numerous challenges on the segmentation and grading of retinal images in recent years, organized by data science competition platforms (such as the Diabetic Retinopathy Challenge on Kaggle 1 ), and as special tracks of corresponding conferences (like "IDRiD: Diabetic Retinopathy -Segmentation and Grading Challenge" (Porwal et al., 2020)).
A particularly popular problem in retinal image analysis is the segmentation of retinal vessels (illustrated in Figure 1). The importance of the problem is threefold. On the one hand, the characteristics of the vasculature can aid the early detection of diseases like hypertension (lui Cheung et al., 2012). On the other hand, the removal of vessels usually precedes the detection of other anatomical parts. For example, the high curvature and junction points of thin vessels have similar visual features to microaneurysms (Long et al., 2020), consequently, the segmentation of the vasculature can indirectly aid the detection of mycroaneurysms. Finally, the segmented vasculature may serve as a guide for locating other anatomical parts, such as the macula (Yu et al., 2014).
In competitive research fields, where the primary goal is to find the most accurate solution to a problem, the proper comparison of algorithms is crucial. Comparability requires the coordination of evaluation data sets and methodologies that fit the purpose of the problem. As pointed out recently (Maier-Hein et al., 2018), even the grand challenges of recent years tend not to produce robust and reliable rankings. Therefore, the study of evaluations in active research areas such as retinal vessel segmentation is desirable to understand the state of the art.
In a self-organised fashion, three data sets have emerged as the de facto standard for the evaluation of vessel segmentation: DRIVE , STARE (Hoover et al., 2000) and HRF . Of these, HRF is relatively new, and many classical algorithms have not yet been evaluated on it. STARE is not split into training and test sets, and the ambiguities of cross-validation lead to uncertain and hardly comparable results. In contrast, DRIVE eliminates many ambiguities of the evaluation by providing a predefined split of training and test images. Because of their long history, DRIVE and STARE continue to be the primary data sets for evaluation in recent work (Jin et al., 2019;Xiuqin et al., 2019), and due to predefined test set in DRIVE, the results obtained on DRIVE are generally accepted as comparable, which is reflected in the numerous rankings in the papers showing the merits of the newly proposed algorithms. The most commonly reported performance measures are the accuracy, sensitivity and specificity of the segmentation. In a previous paper of ours (Kovács and Hajdu, 2016), we briefly pointed out that despite the predefined training and test sets in DRIVE, there is a potential methodological flaw in the evaluations, invalidating many rankings and comparisons in the literature. Namely, textual evidence suggests that some authors compute the performance scores in the field-of-view (FoV) region ( Figure 1(d)), while others compute the scores for all pixels of the images (the entire rectangular area including the black surroundings of the FoV in Figure 1(d)). This minor difference affects the accuracy and specificity scores enormously. Due to the continued interest in this topic, we found it worthwhile to investigate the evaluation methodologies and their impact on comparability in detail. In this paper, we carry out a detailed numerical analysis of the phenomenon including more than 100 papers from the literature. In order to work with a sufficiently large population and eliminate the ambiguity of crossvalidation, we investigate the accuracy, sensitivity and specificity scores reported for the test set of DRIVE. Our main contributions to the field can be summarized as follows: 1. We introduce numerical approaches to uncover some details of the evaluation methodologies used in 100 relevant papers. The results provide inevitable evidence for two main types of methodologies with non-comparable performance scores: evaluation under the FoV mask and evaluation using all pixels. 2. We attempt to eliminate the biases using numerical techniques to produce a more realistic ranking of algorithms. 3. Our results confirm that most rankings in the literature are based on non-comparable figures, the state of the art performance cannot be deduced without the numerical inference and categorization of evaluation methodologies and adjustments of published figures. 4. The techniques we introduce can be transferred to other areas where the identified methodological flaw is potentially present. 5. According to our best knowledge, this is the first study investigating the coherence of a subfield of medical image analysis by numerical techniques. 6. For complete reproducibility, the raw data, the implementation of the analysis and the results are shared in the GitHub repository https://github.com/gykovacs/ retina_vessel_segmentation. The paper is organized as follows. In Section 2, we describe the details of the data set DRIVE and formulate the problem we address by providing textual evidence for its existence. In Section 3, we introduce the numerical methods and present the raw results. Based on the findings of the numerical analysis, we discuss the state of the art in Section 4. Finally, we summarize the paper and draw conclusions in Section 5.

Problem formulation
In this section, we describe the main characteristics of the data set DRIVE, the performance scores, and we also discuss and illustrate the root cause of the problem we address.

The DRIVE data set
The Digital Retinal Images for Vessel Extraction (DRIVE) data set  contains 20 training and 20 test entries. Images were captured with a Canon CR5 non-mydriatic 3CCD camera with a 45 degree field of view and cropped to 584×565 pixels. Each training entry consists of a retinal image, a binary annotation of vessels (by annotator #1), and a binary mask of the Field of View (FoV) indicating the useful image content. The FoV masks cover 69% of the rectangular images. Each test entry includes an additional manual segmentation by annotator #2, enabling the evaluation of human annotators as a baseline. A complete entry from the test set is visualized in Figure 1.

Performance metrics
There are numerous metrics in the literature to characterize the performance of image segmentation algorithms. In retinal vessel segmentation, most authors report some metrics of binary classification performance at the pixel level. Particularly, a segmentation is compared to the ground truth image to determine the number of • true positives (tp) -correctly identified vessel pixels; • true negatives (tn) -correctly identified non-vessel pixels; • false positives ( f p) -non-vessel pixels identified as vessel; • false negatives ( f n) -vessel pixels identified as non-vessel; and introducing p = tp+ f n and n = tn+ f p for the total number of vessel and non-vessel pixels, respectively, the scores • accuracy (acc = (tp + tn)/(p + n)) -the proportion of all correctly identified pixels to all pixels; • sensitivity (sens = tp/p) -the proportion of correctly identified vessel pixels to all vessel pixels; • specificity (spec = tn/n) -the proportion of correctly identified non-vessel pixels to all non-vessel pixels are reported at the image level or averaged on all test images. Some authors report the complements of the sens and spec scores known as false negative (1sens) and false positive (1spec) rates, respectively. We highlight that vessel segmentation as a binary classification problem is highly imbalanced, i.e., the number of vessel pixels in the FoV region is 15% of the number of non-vessel pixels, leading to high absolute accuracy and specificity scores: a 0.1% improvement in accuracy implies the correct segmentation of about 250 additional pixels, which is 1% of the size of the vasculature.
Since the goal is usually to maximize both sensitivity and specificity (and thus maximize accuracy), some authors report other composite measures of sensitivity and specificity, such as the F 1 -score (the harmonic mean of sens and spec) or the AUC score (Hanley and Mcneil, 1982), however, these scores are reported too infrequently to rely a comprehensive analysis on them. Therefore, in the remainder of the paper, we focus on the acc, sens and spec scores.

The source of ambiguity
Although DRIVE specifies dedicated training and test sets, there is some ambiguity regarding the comparability of published performance scores. For example, one of the earliest papers  explicitly states: "Since the dark background outside the FOV is easily extracted, in this paper all experiments are done on the FOV only." On the other hand, (Villalobos-Castaldi et al., 2010) says: "Based on a total of 329,960 pixels contained in a fundus image of size 565 × 584 pixels, it is possible to obtain the TP, TN, FP, and FN quantities of the segmented image." Finally, the authors of (Yu et al., 2012) explicitly state that "It is mentionable that the accuracy and FPR [false positive rate] measures in Jiang (Jiang and Mojon, 2003), Martinez-Perez (Martinez-Perez et al., 2007), and Miri's (Miri and Mahloojifar, 2010) methods were calculated in the complete image instead of the FOV." Even this handful of quotations suggests that some authors evaluate performance only in the area covered by the FoV mask, while others use all pixels of the images. The pixels outside the FoV are usually non-vessel pixels accounting for about 31% of all pixels in the images. If they are treated as true negatives, the accuracy and specificity increases tremendously. To illustrate the magnitude of this effect, we evaluate the segmentation of annotator #2 against annotation #1 (as ground truth) under the FoV mask and including all pixels of the images. With the FoV mask, acc = 0.9473, sens = 0.7760 and spec = 0.9725, but using all pixels, acc = 0.9636, sens = 0.7756 and spec = 0.9818. Omitting the FoV mask increased the acc and spec values by more than 1.5% and 1%, respectively. The slight change in sensitivity is due to an average of 15 vessel pixels in the annotations #1 being outside the FoV. Since many authors report 0.1% improvements in accuracy, the variation in the region used for evaluation can cause an enormous bias when algorithms are compared and ranked.
In the rest of the paper, we develop numerical techniques to test the consistency of the published figures with the hypotheses that the authors either evaluated under the FoV mask or used all pixels of the images.

The need for the numerical analysis
We attempt to investigate the evaluation methodology of dozens of complex image processing pipelines. Collecting original implementations covering 15 years of technological development seems to be unmanagable. On the other hand, reimplementations of the algorithms are unlikely to reproduce the exact figures shared by the authors as all details of complex image processing pipelines are rarely shared due to length limitations. To avoid further bias and to treat all papers similarly and equally, we decided to base the study only on the shared performance scores. In order to draw conclusions without statistical uncertainties, we decided to rely on numerical analysis to reveal some details of the evaluation methodologies used by the authors.

Methods and raw results
In this section, the steps of the numerical analysis are described and the corresponding results are reported. The raw results are turned into insights and discussed in detail in Section 4. We introduce two special notations: The mean of a set of figures is denoted by an overline (such as acc standing for the mean of a set of accuracy scores), and asterisk indicates exact values with no uncertainty (such as tp * standing for the exact number of true positives) that can either be extracted from the publicly available data set or denote unknown but exact values extracted by the authors to calculate the performance scores they reported.

The papers and algorithms involved
To select a sufficiently large, influential and comprehensive population of algorithms and reported performance scores without subjective bias, we applied the following strategy. We searched for the term retinal vessel segmentation in three of the largest scientific publication databases (Elsevier ScienceDirect, IEEE Xplore and ACM Digital Library) and recursively added the most cited ones with all algorithms referenced in them until we reached 90 papers. Finally, 10 recent papers, mainly based on deep learning have been added to represent the latest trends. To maintain quality and achieve the goals of our study, papers that met any of the following exclusion criteria were omitted in the selection process: • not sharing acc, sens and spec scores for DRIVE; • early papers of authors with multiple papers in the field; • being published only on preprint servers; • not reporting performance scores to at least 3 decimal places (rounding of the scores to 2 decimal places introduces larger uncertainty than the size of the bias we investigate). A summary of the papers in Table 1 shows some interesting insights into the evaluation methodologies. Authors rarely phrase explicitly the region they use for evaluation, and when it is shared, a mixture of the two edge cases can be observed: the use of the FoV, and evaluation on all pixels. When the region of evaluation is not shared, one can still observe the two accuracy scores 0.9473 and 0.9637 for annotator #2 (the scores we calculated in subsection 2.3 under the FoV and using all pixels, repsectively). Finally, a slight variation can be observed in the scores being close to the one computed under the FoV (0.9473). This suggests that custom adjustments to the FoV may also have occured, but there is no textual evidence or indication of the nature of these adjustments.

Overview of the numerical analysis
The analysis is based on the interdependence of the scores acc, sens and spec. Particularly, these scores cannot take arbitrary values: they are linear functions of the four unknowns tp, tn, f p and f n, which need to sum up to the number of positive (vessel) p = tp + f n and negative (non-vessel) n = tn + f p pixels, while p + n needs to match the size of the FoV mask or the size of the image. Consequently, the reported values of a triplet of acc, sens and spec together with the exact number of positive and negative pixels in the images must satisfy certain conditions. In the rest of the section, this idea is developed further to deal with the numerical uncertainties of rounding and averaging, and to allow inferences to be made about the evaluation methodologies.

Image level analysis
Some authors have reported the acc, sens and spec scores at the image level. In this subsection, we develop a consistency test to decide whether the image level scores were computed in the FoV region or using all pixels of the images.

The image level consistency test
For each image, the true positives (tp * ) and true negatives (tn * ) calculated by the authors must satisfy the following inequalities by definition: where the total number of positive p * (vessel) and negative n * (non-vessel) pixels can be determined given the publicly available images, and denotes the numerical uncertainty of the reported figures. The setting = 10 −k /2 is the maximum numerical uncertainty when the figures reported to k decimal places are subject to rounding, and = 10 −k is the maximum uncertainty when truncation or ceiling to k decimal places is assumed.
Eliminating tp * and tn * from the inequalities gives 6 conditions that must hold for a given triplet of acc, sens and spec scores, regardless of the actual values of tp * and tn * : The conditions (5) allow us to test the consistency of a certain triplet of acc, sens and spec reported for an image with p * positives and n * negatives, with numerical uncertainty. Since we are concerned about two cases (evaluation using the FoV mask or all pixels), one can extract the total number of positives and negatives with and without the FoV and test whether the conditions for either hypothesis are satisfied. For example, treating annotation #1 as ground truth and annotation #2 as a segmentation for the test image in Figure 1, the rounded performance scores with the FoV mask are acc = 0.9492, sens = 0.7965, spec = 0.9722. In the FoV region, p * = 29412 and n * = 194965, and substituting these figures into the inequalities (5), all conditions are satisfied. On the other hand, including all pixels of the images, p * = 29440 and n * = 300520, and with this substitution the conditions fail. The conditions (5) are sharp in the sense that if the assumptions on p * and n * are correct, then the conditions must hold; if the assumptions on p * and n * are sufficiently far the figures used to compute the scores acc, sens and spec, then the test fails. However, due to the numerical uncertainty, scores computed in regions slightly different from the assumptions might also satisfy the conditions (5). This phenomenon is examined in the next subsection.

Sensitivity of the image level consistency test
To quantitatively characterize the sensitivity of the test to changes in the region of evaluation, we carried out a simulation treating annotation #1 as ground truth and annotation #2 as segmentation for all 20 images in the DRIVE test set. The FoV masks were gradually dilated to cover the images; for each mask size, the performance scores were calculated and rounded to 3 and 4 digits; and the tests derived in subsection 3.3.1 were applied to the scores to check whether they were computed under the original FoV masks or using all pixels of the images. The percentages of image level scores passing the tests at certain mask sizes are plotted in Figure 2 along with the average accuracy to illustrate the bias caused by the adjustment of evaluation region. The first thing to observe is that varying the size of the evaluation region has a remarkable effect on the accuracy scores. However, when the performance scores are rounded to 4 digits, a deviation as little as 1% from the assumed region  Meng et al. (2015) .9529 20 1 .9470 F 4 13 2 Hassanien (2018) .9793 20 1 4 24 52 Li et al. (2016) .9527 2 3 .9472 4 306 3 Wang et al. (2015) .9767 20 9 .9464 F 4 274 53 Imani et al. (2015) .9523 20 5 .9473 4 84 4 Villalobos-Castaldi et al. (2010) .9759 1 .9473 a 4 70 54 Singh and Srivastava (2016) .9522 20 1 a 4 75 5 Memari et al. (2017) .9722 3 .9464 4 23 55 Mo and Zhang (2017) .9521 2 3 4 68 6 Park et al. (2020) .  (2019) .9617 20 1 4 2 67 Roychowdhury et al. (2015) .9494 1 4 153 18 Waheed et al. (2015) .  Lupaşcu et al. (2010) .9597 7 1 .9473 4 329 78 Melinscak et al. (2015) .9466 1 4 118 29 Ricci and Perfetti (2007) .9595 2 1 .9473 F 4 769 79 Mendonca and Campilho (2006) .9463 2 .9473 F 4 983 30 Wu et al. (2020) .9582 3 4 9 80 Rezaee et al. (2017) .9463 1 .9473 4 34 31 Fathi and Naghsh-Nilchi (2013) .9581 20 1 4 95 81 Rahebi and Hardalaç (2014) .9461 20 1 4 38 32 Dash and Bhoi (2018) .957 20 1 3 14 82 Miri and Mahloojifar (2010) .9458 1 4 291 33 Budai et al. (2013) .  Frucci et al. (2016) .955 20 1 3 27 87 Azzopardi et al. (2015) .9442 3 4 445 38 Saroj et al. (2020) .9544 20 1 4 1 88 Strisciuglio et al. (2015) .9442 1 4 25 39 Yan et al. (2018) .  Table 1: A summary of the papers included in the study, with columns as follows. "Rank": the rank based on the published accuracy scores; "Key": reference to the paper; "acc": the published accuracy score; "Num. image level": the number of image level figures shared by the authors; "Num. aggregated": the number of aggregated figures shared by the authors; "Annotator #2 acc.": accuracy score of annotator #2; "Region of eval.": the region of evaluation explicitly phrased by authors, 'F' for FoV, 'A' for all pixels, empty when it is not phrased explicitly; "Decimal places": the number of decimal places to which the figures are reported; "Citations": the number of citations at the time of writing this paper.  makes the test fail on more than 50% of the 20 test images, as can be seen by the steep drop in the corresponding dashed lines around the size of the official FoVs (69% of the image size) and at 100% of the image size. The increased uncertainty of rounding to 3 digits widens the acceptance window (solid lines) and about 50% of the scores pass despite a 15% change in the size of the evaluation region. However, the passing rate drops to about 5% at the opposite assumptions, indicating that the two edge cases can still be distinguished by the test. The analysis suggests that the consistency test is able to distinguish the cases where the official (or slightly differing) FoV masks were used from those cases when all (or almost all) pixels of the images were included in the evaluation.
The results are summarized in Table 2 and plotted in Figure  3 in the accuracy-specificity plane. Surprisingly, 20 out of the 36 papers with image level scores turned out to be outliers. The ranking in Table 2 shows that almost all papers where the use of all pixels was accepted outperform those evaluating under the FoV mask, clearly showing the expected skew in the evaluations. A similar pattern can be seen in Figure 3: the scores where the use of the FoV mask was accepted tend to have lower accuracy and specificity scores than those where the use of all pixels was confirmed.

The analysis of aggregated figures
Most authors report only the averages of image level performance scores. In this subsection, we develop consistency tests to take into consideration the numerical uncertainty caused by the averaging.

Sensitivity of the consistency test for aggregated figures
Similarly to the image level analysis, the aggregated figures must pass the test if the assumptions are met. However, due to the increased numerical uncertainty, it is questionable whether the sensitivity of the test is high enough to distinguish the two corner cases we are concerned with. As before, a quantitative characterization of sensitivity can be obtained by varying the size of the evaluation region, but this time checking whether the aggregated figures pass the test with the p * i , n * i , i = 1, . . . , 20 numbers extracted according to the hypotheses. According to the results shown in Figure 4, a 15% deviation in size from the hypothesised region causes the test to fail when the scores are reported to 4 digits. With rounding to 3 digits, the two corner cases are still distinguished by the test. However, if the region of evaluation is between 77%-87% of the image size, the scores can pass the test with either assumption.

The results on aggregated figures
In some papers (such as (Wu et al., 2020)), multiple triplets of aggregated scores are reported to illustrate the effects of the various steps of the proposed methods. Similarly to the image level analysis, the majority rule is applied to make clear decisions about the region of evaluation used by the authors. The results are summarized in Table 3 and plotted in Figure 5. Again, we assigned 3 labels to the papers, in 30 cases the use of the FoV was accepted, in 18 cases the use of all pixels was accepted, and 16 papers were classified as outliers (the scores do not pass the test with either of the hypothesised regions). As before, the ranking in Table 3 and the illustration in Figure  5 confirm the expected bias in the evaluations: the algorithms evaluated on all pixels tend to have higher accuracy and specificity scores, in general.
To assess the sensitivity of the aggregated consistency test compared to the image level test, we compared the categorization based on image level and aggregated figures and found that the aggregated consistency test gave the same result in all cases where the use of the FoV mask or all pixels was confirmed at the image level. However, from the total of 20 outliers identified at the image level, in 15 cases the aggregated scores passed the consistency test with one of the assumptions. This suggests that  there can be further outliers among the papers in which only aggregated scores are reported and the consistency test confirmed one of the hypotheses.

An improved ranking
In this section, we attempt to derive a more realistic ranking of the algorithms by eliminating the bias caused by the differing regions of evaluation. The goal of the method we develop is to map the scores calculated on all pixels into the domain of evaluation with the FoV mask.
The idea is analogous to that of the consistency tests. Let the subscripts x F and x a refer to figures computed under the FoV mask and using all pixels, respectively, and let x d = x a − x F denote the difference. Given a particular triplet of image level scores acc a , sens a and spec a reported by the authors with = 10 −k numerical uncertainty, we would like to infer on acc F , sens F and spec F . Substituting the unknowns tp * a = tp * F + tp * d and tn * a = tn * F + tn * d into (1)-(4), one gets the conditions which must hold for some integer unknowns tp * F , tn * F , tp * d and tn * d . The constants p * F , p * d , n * F and n * d are known from the publicly available test set. Focusing on the accuracy score acc * F = tp * F + tn * F p * F + n * F first, we are interested in a lower and upper bound on the values it can take under the conditions (10). Recognizing that the conditions (10) with the objective function acc * F form a linear integer programming problem of the free integer variables tp * F , tn * F , tp * d and tn * d , one can determine the minimum (L) and maximum (U) values the objective function can take by exploiting any linear programming solver. These extrema provide a sharp interval estimate for the accuracy score evaluated under the FoV: acc * F ∈ [L(acc * F ), U(acc * F )]. The adjusted accuracy score is defined as the mid-point of the interval and the error of the adjustment is estimated as the half-width of the interval: (11) The adjusted image level sens F and spec F scores and corresponding error estimates are determined analogously.
In practice, we found that the formulation (10) leads to wide intervals due to mathematical solutions that are not aligned with the problem. In the formulation (10), the number of true negatives outside the FoV (tn d ) can be zero, that is, the linear programming solver considers variable configurations corresponding to segmentations where all pixels outside the FoV are segmented as vessels, which is unlikely. In order to tighten the intervals, the configuration space is reduced by making the natural assumption that no more negative pixels than 1% of the vasculature are segmented as vessels outside the FoV: With this constraint, the worst-case error of the adjustment when applied to all image level figures reduces to 0.002 with the average of 0.001, which we found sufficiently tight for further analysis.
Similarly to the image level adjustment, for the averaged scores, one can substitute tp * a,i = tp * F,i + tp * d,i and tn * a,i = tn * F,i + tn * d,i , i = 1, . . . , 20 into the conditions (6) -(9) leading to a set of linear integer constraints. Then, minimizing and maxi-mizing the objective function in the free integer variables tp * F,i , tn * F,i , tp * d,i , tn * d,i , i = 1, . . . , 20, one gets a sharp interval estimate for acc * F . The adjusted aggregated score acc F is determined as the mid-point of the interval and the corresponding error term acc F as the half-width of the interval. The adjusted sens * F and spec * F scores are determined analogously.
For illustration, the adjustment was applied to the scores calculated on all pixels and reported in our earlier paper (Kovács and Hajdu, 2016). The adjusted scores acc F = 0.9500, sens F = 0.7447, spec F = 0.9802 are well aligned with the true scores acc F = 0.9494, sens F = 0.7446, spec F = 0.9793 calculated in the FoV area, the error of the adjustment is not higher than 0.001 for any of the figures.
The adjustment was applied to the reported scores of all methods we identified to be evaluated using all pixels. The worst case of the adjustment is 0.003 with the average of 0.001. As the errors are an order of magnitude smaller than the bias, we believe that the ranking based on the adjusted scores in Table 4 is a more reliable baseline for the field than the original ranking in Table 1.

Insights
Based on the raw results and the adjusted scores, in this section, we provide some detailful insights into the field.

Insights from the consistency testing
Overall, the use of the FoV mask was accepted in 36 cases, and the scores reported in 28 papers were consistent with the use of all pixels for evaluation. 36 papers were found to be outliers with undoubtful evidence that the evaluation was carried out in an unknown third way. Binomial hypothesis testing at the usual significance levels fails to reject the null hypothesis that the authors have no preference in the choice of the evaluation methodology (p-value: 0.19). Consequently, we can state that neither of the two evaluation methodologies we investigated can be treated as a standard in the field.
Explicit phrasing of the region of evaluation was found only in 29 papers. Out of them, only in 15 cases were the reported scores consistent with the claims of the authors. These inconsistencies indicate that the most reliable source for the region of evaluation is the triplet of performance scores the authors shared.
The results of the consistency tests can be used to verify that the rankings in the papers are valid: if at least two algorithms evaluated in different regions are included in a ranking, it contains non-comparable figures. This phenomenon was confirmed in 91 of the 100 papers. Together with the 10 reviews (Almotiri et al., 2018;Moccia et al., 2018;Mookiah et al., 2021;Besenczi et al., 2016;Khan et al., 2018;Cetinkaya and Duran, 2020;Srinidhi et al., 2017;Singh et al., 2020;Fraz et al., 2012b;Fu et al., 2018)   in the title of this paper is confirmed: rankings are based on non-comparable figures in 100+ papers. As recent papers are also affected, we can conclude that ranking algorithms based on non-comparable figures is an existing and ongoing problem in the field.

Insights from the new ranking
The correlation coefficient between the original and the adjusted rankings is 0.18, indicating that the adjustment of scores leads to a substantial change in the recognition of efficient vessel segmentation techniques.
The reported scores show that the accuracy of annotator #2 was outperformed by 1% in 2007, and at the time of writing, near-perfect techniques are available with accuracy of 0.98-0.99. In contrast, the adjusted scores show that the accuracy of annotator #2 was only reached in 2014, and the highest accuracy to date is only 1% above it (evaluated under the FoV mask). This phenomenon shows similarities with the field of electrohysterogram classification: the near perfect accuracy scores reported in the literature were caused by a methodological flaw in the evaluations (Vandewiele et al., 2021).
In Figure 6(a) we have plotted the distribution of algorithms and evaluation methodologies in the adjusted accuracyspecificity plane. The plot shows that after eliminating the bias, the algorithms evaluated with all pixels have lower accuracy and specificity scores than those evaluated with the FoV mask. The independent two-sample t-test shows that the difference is statistically significant with a p-value of 10 −9 . The reason for this surprising phenomenon is that using all pixels to evaluate less effective techniques results in performance scores comparable to or even higher than those of more effective techniques evaluated under the FoV mask, and this brings publicity to less effective approaches.
Finally, we compare the performances of the methods by their operating principles. We have introduced 4 categories: deep learning, classical (thresholding, filtering, morphology, region growing), supervised (feature extraction and supervised machine learning, but not deep learning), and others (dominantly graph-cut based segmentation techniques). The results presented in Figure 6(b) show that deep learning techniques increase the accuracy by almost 1% compared to the best performing classical and supervised techniques.

A remark on the evaluation of deep learning techniques
From a development perspective, there is an important difference between heuristic image processing pipelines and highly flexible deep learning solutions that calls into question the suitability of current evaluation methodologies in the field. Unlike the heuristic improvement of conventional image processing pipelines, the tuning of hyperparameters in deep neural networks is relatively straightforward (automated in many cases), which increases the risk of overfitting when the performance scores measured on the test set drive the tuning of the network.
Although this could be prevented by using the training images of DRIVE for both training the weights in the network and tuning the network structure, we found evidence that some results are clearly overfitted to the test set: Both (Li et al., 2016)   and (Liskowski and Krawiec, 2016) optimize the network size and hyperparameters on the test images. Despite the obvious overfitting, we did not discard these results for two reasons. First, all other papers using deep learning declare the network structures and the hyperparameters. None of the authors describe how the network was derived from the training data, which risks the overfitting of the network structure to the test images. On the other hand, there is no doubt that deep learning techniques achieve better results than classical approaches. Although some papers may have reported too optimistic scores due to overfitting to the test images, they clearly dominate the field. Consequently, the development and evaluation of deep learning solutions for retinal vessel segmentation should follow the standards of the deep learning literature by using 3 sets of images (Larochelle et al., 2009): one to train the weights in the network, another to tune the network structure and a final one to test the performance of the optimised network. However, since DRIVE specifies only 2 subsets and further splitting of the training set (20 images) would lead to a nearly insufficient number of images for deep learning, a larger set of images should become the standard in this field.

Limitations
The analysis we carried out has numerous limitations. Despite the systematic search for influential papers, the selected group may provide a biassed representation of the field and the results of the selection process may change over time as the number of citations changes.
The consistency tests for aggregated figures tolerate greater deviation from the hypotheses than do the image level tests. This inherent imbalance in the rigour of the tests implies that some papers that passed the consistency test with aggregated scores might have failed if the authors had reported image level scores.
The adjusted scores have some numerical uncertainty. Although the estimated errors are small, the improved ranking is also uncertain to this degree.
We have focused primarily on the comparability of the reported performance scores. However, higher performance is not the only goal in science. If some methods are easier to understand, maintain, adapt, or are more efficient in terms of runtime, they are still valuable contributions to science. Therefore, the improved ranking does not devalue the contributions of any paper, as all papers provide insights into the capabilities of the implemented approaches.

Summary and Conclusions
In this study, we investigated the coherence of reported performance scores in the segmentation of vessels in retinal images. We collected 100 influential papers reporting accuracy, sensitivity, and specificity for the publicly available data set DRIVE. Based on textual references to two corner cases of evaluation methodologies (evaluation under the FoV mask or using all pixels of the images), we developed numerical methods to test the consistency of the reported scores with the two assumed corner cases (subsections 3.3.1, 3.4.1). Based on results of consistency tests, we attempted to remove bias and make the scores comparable to create a new baseline ranking (Section 3.5). Section 4 summarizes various findings and insights and also discusses the limitations of our analysis. The insights clearly show that there is no consensus in the field on the evaluation methodology. However, in most of the papers, including reviews of the field, the rankings and the objective evaluation of operating principles are biassed by the direct comparison of non-comparable figures. Based on the papers we selected and the adjustment we implemented, the current state of the art accuracy is 0.9582 (evaluated in the FoV region), which is about 1.1% higher than the performance of annotator #2. Despite the limitations of our method, we believe that the adjusted ranking in Table 4 provides a more realistic view of the state of the art than any ranking based on the direct comparison of reported figures.
The findings have numerous implications regarding the suitability and potential improvements of data sets, evaluation methodologies and reporting standards in the field: • A novel data set that takes into account the changing requirements of the field is desirable -potentially, this could emerge as a curated fusion of existing vessel segmentation data sets that provides predefined training, test and validation subsets for the reliable evaluation of deep learning techniques, and the sizes of the subsets should be based on well-established principles (Guyon et al., 1998). • The published average scores on DRIVE usually differ in the 3th-4th digits. Since these are the averages of only 20 image level scores, one can assume a relatively large variance, which makes the rankings ambigious. To investigate whether there were statistically significant improvements, the authors should provide image level figures at least as supplementary material and explicitly report more details about the evaluation methodology (e.g., the region they use to calculate the scores). • Authors should report the Dice score (Eelbode et al., 2020), dice = (2 · tp)/(2 · tp + f p + f n), which is a commonly accepted measure of segmentation performance.
Since dice is inherently free of the number of true negatives, the bias we study in this paper is naturally eliminated. Treating retinal vessel segmentation as a pixelwise binary classification problem, the AUC score (the standard performance measure for binary classification in machine learning) provides good insight into the trade-off between the sensitivity and specificity of segmentation algorithmsif applicable (the segmentation is based on pixelwise vesselness probabilities), authors should report AUC scores. • Maximizing some pixelwise performance measures of segmentation does not necessarily improve clinical applications. For example, an algorithm that provides an average accuracy of 0.95 with a small standard deviation may lead to a more reliable grading of disease severity than an algorithm with an average accuracy of 0.96 and a huge standard deviation, which means that it fails under certain conditions. Consequently, the evaluation of vessel segmentation methods should be more closely linked to clinical applications. The numerical techniques we have developed can be easily adapted to other problems where similar methodological flaws may arise. To allow full reproducibility and further analysis, all raw data, the implementation of the analysis and all results are shared in the GitHub repository www.github.com/gykovacs/ retinal_vessel_segmentation.