{'fraction_non_alphanumeric': 0.04176265385374844, 'fraction_numerical': 0.014249983545053644, 'mean_word_length': 4.211284513805523, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Holy Grail and the Bad Sampling - A test for the homogeneity of missing proportions for evaluating the agreement between peer review and bibliometrics in the Italian research assessment exercises

Two experiments for evaluating the agreement between bibliometrics and informed peer review - depending on two large samples of journal articles - were performed by the Italian governmental agency for research evaluation. They were presented as successful and as warranting the combined use of peer review and bibliometrics in research assessment exercises. However, the results of both experiments were supposed to be based on a stratified random sampling of articles with a proportional allocation, even if solely subsets of the original samples in the strata were selected owing to the presence of missing articles. Such a kind of selection has the potential to introduce biases in the results of the experiments, since different proportions of articles could be missed in different strata. In order to assess the 'representativeness' of the sampling, we develop a novel statistical test for assessing the homogeneity of missing proportions between strata and we consider its application to data of both experiments. Outcome of the testing procedure show that the null hypotesis of missing proportion homogeneity should be rejected for both experiments. As a consequence, the obtained samples cannot be considered as 'representative' of the population of articles submitted to the research assessments. It is therefore impossible to exclude that the combined use of peer review and bibliometrics might have introduced uncontrollable major biases in the final results of the Italian research assessment exercises. Moreover, the two experiments should not be considered as valid pieces of knowledge to be used in the ongoing search of the Holy Grail of a definite agreement between peer review and bibliometrics.


Introduction
The definite proof that peer review substantially agrees with some kind of bibliometric indicators is the Holy Grail for research assessment designers (RADs), since simpler, cheaper and more "objective" bibliometric indicators could wholly replace the peer review (Pride and Peter, 2018). Many RADs and scholars would welcome that definite proof since finally objective algorithms could substitute unreliable human peer reviewers, prone to nepotism and opportunism. In this perspective, the Holy Grail is an evaluation without human evaluators, ideally tending to a "view from nowhere" (Goukrager, 2012). It is usual to read about the contraposition of "objective bibliometric data" and "subjective peer reviews". It is not surprising at all that many descriptions of the aims of national research assessments emphasize that point. For example, the second Polish research assessment exercise is "based on a parametric assessment to make the evaluation more objective and independent from its peer" (Kulczycki et al., 2017). Again, in a final report of the research assessment in Italy, some members of the panel claim "the need of strongly reducing the peer evaluation, since it introduces a subjectivity representing a bias that cannot be normalized" (AREA7 Rapporto finale, p.113). Others RADs would welcome the definite proof for a less radical and more practical reason: if peer review and bibliometrics agree, it is possible to combine them in a universal research assessment where some disciplines -notably the humanities -cannot simply be evaluated by indicators.
Unfortunately, the search of the Holy Grail of research assessment has not been fruitful so far. The most extensive research campaign to date has produced negative results, even suggesting that the Holy Grail of evaluation does not exist: "This work -according to the authors -has shown that individual metrics give significantly different outcomes from the REF peer review process, showing that metrics cannot provide a like-for-like replacement for REF peer review" (Wilsdon et al., 2015).
In Italy, the governmental agency for research evaluation (ANVUR) conducted two research campaigns in search of the Holy Grail. During the two last national research assessments, VQR1 for the years 2004-2010 and VQR 2 for 2011-2014, ANVUR realized two extensive experiments (hereafter EXP1 and EXP2) by comparing evaluations reached by bibliometrics and by informed peer review (IPR) based on large samples of journal articles. Apparently, results of EXP1 and EXP2 were promising for the search of the Holy Grail. Results of EXP1 were published not only as official reports, but also disseminated as working papers or scientific papers in refereed journals by scholars working for the agency. Ancaiani et al. (2015) claimed that the results of EXP1 support "the choice of using both techniques in order to assess the quality of Italian research institutions". Bertocchi et al. published five identical working papers where they interpreted the results of EXP1 as claiming that peer review and bibliometrics "are close substitutes" (for all Bertocchi et al. 2013). In the peer reviewed version of the papers they concluded that "the agencies that run these evaluations could feel confident about using bibliometric evaluations and interpret the results as highly correlated with what they would obtain if they performed informed peer review" (Bertocchi et al., 2015). The results of EXP2 were also presented as a success in the official report: since a "not-zero correlation" was found (ANVUR, 2017), "we can hence conclude that the combined used of bibliometric indicators for citations and journal impact may provide a useful proxy to assess articles quality" (Alfò et al., 2017).
Two of the authors of the present paper analyzed EXP1. Despite they were unable to wholly replicate its results, since ANVUR did not disclose anonymized raw data, they documented many flaws (Baccini and De Nicolao, 2016a, 2016b, 2017a, 2017b. In particular, from the perspective of the sampling design, a stratified sampling of articles with proportional allocation was assumed by ANVUR experts. However, the results of EXP1 were not computed on the whole sample of articles, but solely on a subset of the sample owing to the presence of missing articles. This selection has the potential to introduce biases in the results of the experiment, since different proportions of articles could be missed in different strata. Since the design of EXP1 was repeated also for EXP2, the second experiment might also suffer from the same problem. The present paper is targeted to evaluate the "representativeness" of the samples adopted in EXP1 and EXP2. Hence, on the basis of the previous discussion, the focus is assessing the homogeneity of missing proportions in the strata. To this aim, after developing a novel statistical test, we apply our methodological findings to data of EXP1 and EXP2. Thus, the article is organized as follows: in Section 2 we present the structure of EXP1 and EXP2 by briefly reminding the essential issues of the Italian research assessment exercises. In Section 3 the theoretical development of the test for assessing missing proportion homogeneity is presented. Section 4 reports the results of the proposed testing procedure. Section 5 briefly concludes the paper by discussing if EXP1 and EXP2 can be considered as a valid piece of evidence in favor of the agreement between peer review and bibliometrics.

A brief description of the Italian experiments
A brief contextualization of EXP1 and EXP2 is preliminarily needed for understanding their relevance and role in the two Italian research assessments (this description is largely based on Baccini and De Nicolao, 2016a). The aim of both VQR1 and VQR2 were to evaluate research institutions such as universities or departments, and research areas and fields both at national or institutional levels. Each university, department and research field was classified by calculating the average score obtained by the research outputs submitted by researchers. To this end, all the researchers with a permanent position in an university had to submit a fixed number (with few exceptions) of research outputs (3 in VQR1 and 2 in VQR2). Each research work was then evaluated as Excellent, Good, Acceptable, Limited in VQR1 and Excellent, Elevated, Fair, Acceptable, Limited in VQR2, and received a score (scores slightly changed between the two exercises).
Both VQR1 and VQR2 were organized in 16 widely defined research areas. The 16 areas were: Mathematics and Informatics (Area 1), Physics (Area 2), Chemistry (Area 3), Earth Sciences (Area 4), Biology (Area 5), Medicine (Area 6), Agricultural and Veterinary Sciences (Area 7), Civil Engineering and Architecture (Areas 8a and 8b), Industrial and Information Engineering (Area 9), Antiquities, Philology, Literary studies, Art History (Area 10), History, Philosophy, Pedagogy and Psychology (Areas 11a and 11b), Law (Area 12), Economics and Statistics (Area 13), Political and Social Sciences (Area 14). These areas originates from the traditional classification of research areas adopted in Italy. For each area, an evaluation panel was established with a number of panelists proportional to the number of research outputs to be evaluated. Each panel was organized in sub-panels, specialized for specific research fields, so a total of 44 sub-panels were defined in both VQR1 and VQR2.
Panels directly managed and evaluated subsets of research products submitted for evaluation in their area of expertise. In both research assessments, research evaluation was analogously realized. Panels for the so-called "Bibliometric Areas", i.e. hard sciences, engineering and life sciences, evaluated papers mainly but not exclusively, through bibliometrics. The bibliometric algorithm changed between VQR1 and VQR2, but in both assessments it was based on the number of citations received by an article and on a journal indicator, e.g. the impact factor, of the journal in which it was published. In the case that the two indicators gave coherent indications, the algorithm generated a score for the article. Otherwise, if they disagreed (high number of citations and low impact factor or viceversa), the algorithm output was unable to attribute a defined score to the article and it was therefore classified as "IR" and evaluated by informed peer review.
Panels of the so called "non-bibliometric areas", i.e. Social Science and Humanities, excluding Economics and Statistics, evaluated submitted research products exclusively by IPR. Area 13 (Economics and Statistics) was an exception since the Area 13 panel developed a journal ranking where journals were classified as Excellent, Good, Acceptable, Limited (VQR1) or Excellent, Elevated, Fair, Acceptable, Limited (VQR2). All the articles published in one of the listed journals then received the score of the journal in which they were published. All other research outputs (books, chapters, articles published in journals not ranked by ANVUR) were evaluated by informed peer review.
A couple of anonymous reviewers chosen by one or two members of the sub-area panels, performed the informed peer review of the article, by using a predefined format (slightly different between the two research assessment and also between panels in the same assessment). One of the member of the panel who had chosen at least one of the referees, summarized then the two referee's reports and attributed one of the four scores to the journal article.
ANVUR coined the expression "evaluative mix" to denote this complex evaluative machinery that created many problems, documented for example by (Abramo and D'Angelo, 2016, 2017, Franceschini and Maisano, 2017. The main one is the possible biases induced by the adoption of different evaluation techniques. Indeed, if IPR produced scores systematically different from the ones produced by bibliometrics, this might have introduced systematic bias in the scoring system used for ranking institutions. Indeed, ANVUR realized EXP1 and EXP2 precisely for addressing that problem: a good agreement between bibliometric evaluation and evaluation performed by IPR might justify the adoption of the two different evaluation methods and preserve the comparability of results among areas, institutions, departments and research fields. Positive results of EXP1 and EXP2 were crucial for the soundness of Italian research assessment results: if peer review and bibliometrics do not agree and give significantly different results, the average scores of an institution might be distorted by the different percentage of scores attributed by IPR and by bibliometrics. EXP1 and EXP2 have an identical structure and rationale. The bulk of ANVUR experiments consisted in the analysis of the agreement between the evaluation obtained through IPR and bibliometric algorithms. The statistical technique adopted by ANVUR was Cohen's kappa (Cohen, 1960), the most popular index of interrater agreement for nominal categories (Sheskin, 2003). High level of agreement should be interpreted as justifying the use of bibliometric and IPR in a same research assessment.
Both in EXP1 and EXP2, according to "Appendice B" of the Final Reports of both VQR1 and VQR2, ANVUR adopted a stratified random sampling with proportional allocation of the population constituted by the journal articles submitted to the research assessments (sample size was about 10% of the population size). Indeed, the Final Reports remark that: "The sample was stratified according to the distribution of the products among the sub-  areas of the various areas" (ANVUR 2017, Appendice B, p.1 our translation). For EXP1 we know the data of the population and of the sample at a sub-area level, while for EXP2 only data for areas are instead available. Each article of the samples received a score by the bibliometric algorithms and was also evaluated by IPR. Up to this point, the design of both experiments is apparently correct, even if a major problem arose during the procedure of bibliometric evaluation. As we have previously remarked, the bibliometric algorithms might result in an inconclusive classification IR for some articles for which the disagreement between citations and impact factor did not permit to automatically assign a score. Both in EXP1 and EXP2, all the articles classified as IR were dropped from the experiments. The consequent distortion in the sample was not accounted for by ANVUR, that just computed the agreement indexes for the articles in the sub-sample. Tables 1 and 2 show the sizes of the article population, of the sample and of the reduced final sub-sample according to the stratification based on the areas for EXP1 and EXP2. Table 3 reports the same sizes for the stratification based on the sub-areas, which was solely available for EXP1.
For EXP1, the reduction of the sample was not disclosed neither in ANVUR's official reports nor in Ancaiani et al. (2015). Two of the authors of this paper documented (Baccini and De Nicolao, 2017b), with reference to Ancaiani et al. (2015), that the results of EXP1 were not computed on the whole random sample of articles, but on a subset of the sample. Serious concerns about the whole experiment were raised, by highlighting that unknown biases might have been introduced due to the missing items. The reply  concentrated not on biases but on "representativeness" of the selected subset (Baccini and De Nicolao, 2017a). Benedetto et al. (2017) wrote that: "the distribution of the subsample across the scientific areas is fairly uniform and proportional to the one resulting from the full sample, i.e. the subsample can be considered as representative of the population of reference in terms of its distribution among scientific areas. [. . . ] Furthermore, [. . . ] the ex-post distribution of bibliometric evaluations is pretty similar in the reference population and in the subsample, confirming that the subsample is a correct representation of the population of reference, also, and perhaps more importantly, in terms of bibliometric results. We hence conclude that [. . . ] the evaluation of concordance has been performed on a sample that is fully representative of the original population of articles to be evaluated". Despite the problem was known, ANVUR proceeded for EXP2 by adopting the same strategy: stratified sampling of papers with proportional allocation, dropping of papers with inconclusive bibliometric score (IR), calculation of the agreement without disclosing information about the biased selection of papers (ANVUR, 2017).
In order to gain a basic qualitative intuition of the problems induced by the such a selection of papers, it suffices to observe that ANVUR removed from both EXP1 and EXP2 the more problematic articles for which the bibliometric algorithm was unable to reach a score. We cannot exclude that these articles were also the more problematic to be evaluated by peer reviewers. If this is true, ANVUR conducted both experiments on sub-samples "more favorable" to agreement than the complete samples.
From a statistical point of view, drawbacks would arise if the removal of articles from the sample occurred in a non-proportional way between the strata. Thus, in the next section, we derive a procedure for testing the homogeneity of missing proportions between the strata.

Testing the homogeneity of missing proportions
Let us suppose a population of N units partitioned into L strata. Moreover, let us assume that N l is the size of the l-th stratum, i.e. N = L l=1 N l . A stratified sampling is carried out by drawing n l units in each stratum according to simple random sampling without replacement and n = L l=1 n l . In the following, with a slight abuse, we also adopt the notation n = (n 1 , . . . , n L ).
In this setting, each unit of the l-th stratum may be missed with probability π l ∈ [0, 1] -independently with respect to the other units. Thus, the size of missing units in the l-th stratum, say M l , is a random variable (r.v.) distributed according to the Binomial law with parameters N l and π l , i.e. the probability function (p.f.) of M l turns out to be where 1 A represents the indicator function of the set A. Let us assume that the r.v. X l represents the size of missing units of the l-th stratum in the sample. By supposing that the units are missing independently with respect to the sampling, the distribution of the r.v. X l given the event {M l = m} is the Hypergeometric law with parameters n l , m and N l , i.e. the corresponding conditioned p.f. is given by On the basis of these findings and by using the result by Johnson et al. (2005, p.377), the r.v. X l is distributed according to the Binomial law with parameters n l and π l , i.e. the p.f. of X l turns out to be for each l = 1, . . . , L. Obviously, the X l 's are independent r.v.'s. For subsequent use, we also consider the random vector Y = (X 1 , . . . , X L ). Let us consider the null hypothesis of missing proportion homogeneity between strata H 0 : π l = π, ∀l = 1, . . . , L versus the alternative hypothesis H 1 : π l = π, ∃l = 1, . . . , L. Thus, for a given realization of the random vector Y , say (x 1 , . . . , x L ) ∈ N L such that L l=1 x l = s, the likelihood function under the alternative hypothesis is while the likelihood under the null hypothesis is Hence, the likelihood estimator of (π 1 , . . . , π L ) under the alternative hypothesis is given by ( π 1 , . . . , π L ) where π l = X l /n l , while the likelihood estimator of π under the null hypothesis is given by π = S/n where S = L l=1 X l . The likelihood-ratio test statistic could be adopted in order to assess the null hypothesis. However, in the present setting the large-sample results are precluded since the sample size n is necessarily bounded by N and the data sparsity could reduce the effectiveness of the large-sample approximations. A more productive approach may be based on conditional testing (for more details on this issue, see Lehmann and Romano, 2005, chapter 10). First, we consider the χ 2 -test statistic -asymptotically equivalent in distribution to the likelihoodratio test statistic under the null hypothesis -which in the present setting reduces to It should be remarked that the r.v. S is sufficient for π under the null hypothesis. Hence, in such a case, the distribution of the random vector Y given the event {S = s} does not depend on π. Moreover, under the null hypothesis, this conditional distribution is the multivariate Hypergeometric law with parameters s and n, i.e. the p.f. of Y given {S = s} turns out to be x l = s} . Therefore, by assuming the conditional approach, an exact test may be carried out. Indeed, if t represents the observed realization of the test statistic T , the corresponding P -value turns out to be where C t = {x : x ∈ A, T (x) ≥ t}. It should be remarked that the previous P -value may be approximated by means of a Monte Carlo method by generating realizations of a Hypergeometric random vector with parameters s and n. The generation of each realization requires (L − 1) Hypergeometric random variates -for which suitable algorithms exist, see e.g. Hörmann et al. (2004) -and hence the method is practically feasible.
The previous testing procedure may be generalized to the case of strata partitioned in sub-groups and testing the homogeneity of missing proportions in the sub-groups is also required. Hence, let us now suppose that the l-th stratum is partitioned into G l subgroups and let us assume that N lk is the size of the k-th sub-group in the l-th stratum, i.e. N l = G l k=1 N lk and N = L l=1 G l k=1 N lk . In addition, a stratified sampling is carried out by drawing n lk units in the k-th sub-group of the l-th stratum according to simple random sampling without replacement, in such a way that n l = G l k=1 n lk and n = L l=1 G l k=1 n lk . Moreover, we also assume that n l = (n l1 , . . . , n lG l ) for l = 1, . . . , L. Finally, as to a frequently-adopted notation in this section, if a = (a 1 , . . . , a k ) and b = (b 1 , . . . , b m ) represent two vectors, their concatenation is defined as (a, b) = (a 1 , . . . , a k , b 1 , . . . , b m ).
Similarly to the setting considered in the simpler case, each unit in the k-th sub-group of the l-th stratum may be missed with probability π lk -independently with respect to the other units. Hence, if X lk represents the size of missing sampled units in the k-th subgroup of the l-th stratum, the r.v. X lk is distributed according to the Binomial law with parameters n lk and π lk . More precisely, the p.f. of X lk is given by n lk x π x lk (1 − π lk ) n lk −x 1 {0,1,...,n lk } (x) for each l = 1, . . . , L. In turn, the X lk 's are independent r.v.'s. In this framework, let us consider the global null hypothesis of missing proportion homogeneity between all the sub-groups H g 0 : π lk = π, ∀k = 1, . . . , G l , l = 1, . . . , L versus the alternative hypothesis H g 1 : π lk = π, ∃k = 1, . . . , G l , l = 1, . . . , L. In complete analogy with the simpler case, by assuming that Y l = (X l1 , . . . , X lG l ) for l = 1, . . . , L, a global test statistic for assessing H g 0 is given by where π g,lk = X lk /n lk and π g = S/n, where -consistently with respect to the adopted notation -we assume that S = L l=1 X l and X l = G l k=1 X lk . Under the null hypothesis H g 0 , the distribution of the random vector (Y 1 , . . . , Y L ) given the event {S = s} is the multivariate Hypergeometric law with parameters s and (n 1 , . . . , n L ), i.e. the corresponding conditioned p.f. is given by n lk x lk n s where x l = (x l1 , . . . , x lG l ) for l = 1, . . . , L, while B = {(x 1 , . . . , x L ) : x lk ∈ {max(0, n lk − n + s), . . . , min(n lk , s)}, Hence, if t g represents the observed realization of the test statistic T g , the corresponding P -value is given by In turn, this P -value may be approximated by means of a Monte Carlo method by generating realizations of a Hypergeometric random vector with parameters s and (n 1 , . . . , n L ). The generation of each realization requires ( L l=1 G l − 1) Hypergeometric random variates. If H g 0 is rejected, the null hypothesis of missing proportion homogeneity between the sub-groups within each stratum H s 0 : π lk = π l , ∀k = 1, . . . , G l , ∀l = 1, . . . , L could be considered, jointly with the collection of the single L null sub-hypotheses H s,1 0 , . . . , H s,L 0 , where H s,l 0 : π lk = π l , ∀k = 1, . . . , G l . The null hypotheses H s 0 and H s,1 0 , . . . , H s,L 0 may be simultaneously assessed by considering the test statistics T s and T s,1 , . . . , T s,L , where n lk ( π s,lk − π s,l ) 2 π s,l (1 − π s,l ) and π s,lk = X lk /n lk and π s,l = X l /n l . It should be remarked that the random vector Y = (X 1 , . . . , X L ) is sufficient for (π 1 , . . . , π L ) under the null hypothesis H s 0 . In such a case, the distribution of the random vector (Y 1 , . . . , Y L ) given the event {Y = x} does not depend on (π 1 , . . . , π L ). Moreover, under the null hypothesis H s 0 and conditioning to the event {Y = x}, the L random vectors Y 1 , . . . , Y L are independently distributed according to multivariate Hypergeometric laws. In addition, the distribution of the random vector Y l given the event {Y = x} is the multivariate Hypergeometric law with parameters x l and n l . Hence, it turns out that the conditioned p.f. is Hence, T s,1 , . . . , T s,L are conditionally independent r.v.'s -even if they do depend on T s .
Assuming the conditional approach, (L+1) exact tests may be jointly carried out. Indeed, if t s and t s,l represent the observed realizations of the test statistics T s and T s,l respectively, the corresponding P -values are and where we assume that C ts = {(x 1 , . . . , x L ) : x l ∈ B l , T s (x 1 , . . . , x L ) ≥ t s } and C t s,l = {x l : x l ∈ B l , T s,l (x l ) ≥ t s,l }. Obviously, these P -values simultaneously hold. Finally, it should be remarked that the previous P -values may be approximated by means of a Monte Carlo method by generating realizations of L independent Hypergeometric random vectors with parameters x l and n l for l = 1, . . . , L. Thus, in such a case the generation of each realization requires ( L l=1 G l − L) Hypergeometric random variates.

Data and results
We have applied the testing procedures developed in the previous section to the data of EXP1 and EXP2 by considering the areas as the strata (see Tables 1 and 2) and to the data of EXP1 by considering the sub-areas as the sub-groups (see Table 3). In order to avoid the obvious criticism that the tests will reject the null hypotheses since Area 13 displays no missing articles, we have performed the testing procedures after having eliminated data for this area. At first, we have considered the null hypothesis H 0 of missing proportion homogeneity between strata both for EXP1 and EXP2. As to EXP1, the null hypothesis H 0 can be rejected since the P -value corresponding to the test statistic T was less than 10 −6 . As to EXP2, the rejection of the null hypothesis H 0 is in turn justified since the P -value corresponding to the same test statistic was less than 10 −6 . As previously remarked, in the case of EXP1 the data were also available for the sub-groups -in addition to strata. Hence, the simultaneous procedure for testing the homogeneity of missing proportions between the sub-areas of the scientific areas may be applied. In such a case, the P -value corresponding to the test statistic T s was given by 0.00001, while the P -values corresponding to the test statistics T s,l were 0.00002 for Area 1, 0.02401 for Area 2, 0.01740 for Area 3, 0.24973 for Area 4, 0.16275 for Area 5, 0.07389 for Area 6, 0.03962 for Area 7, 0.68388 for Area 8 and 0.96818 for Area 9. Hence, the simultaneous null hypothesis H s 0 of missing proportion homogeneity between the sub-areas within each scientific area should be also rejected. The rejection of H s 0 is mainly induced by the Areas 1, 2, 3 and 7, i.e. the areas for which the null hypotheses H s,l 0 could be reasonably rejected.

Conclusion
This paper investigated the "representativeness" of the sampling procedure adopted in EXP1 and EXP2 by ANVUR. We have rigorously formulated the problem as one of testing missing proportion homogeneity between the strata of a population. After having developed the appropriate statistical tests, we have applied the theoretical findings to the data of the two experiments conducted by ANVUR. Results of the testing procedure show that the null hypotesis of missing proportion homogeneity between the scientific areas should be rejected for both experiments. The null hyphotesis of homogeneity should be rejected also when the sub-areas for each scientific area are considered. As a consequence, the sampling selection adopted by ANVUR for EXP1 and EXP2 cannot be considered as "representative" of the population of articles submitted to the research assessment. Indeed, "representativeness" could be solely guaranteed if the missing proportions in the strata induced by article elimination were homogenous. Results of this paper are relevant from two points of view. From the point of view of the Italian research assessments exercises, they demonstrate that the results of the experiments cannot be considered at all as validating the use of the dual metod of evaluation adopted. At the state of current knowledge, it cannot be excluded that the use of a dual method of evaluation introduced uncontrollable major biases in the final results of the assessment. Since all evidence drawn from data in the official research reports shows that peer reviewers' scores were, on average, lower than bibliometric ones, aggregate results for research fields, departments and universities might be affected by the proportion of research outputs evaluated by the two different tecniques: the higher the proportion of research outputs evaluated by peer review, the lower the aggregate score. From the point of view of the search of the Holy Grail of research assessment designers, this paper documents that the experiments conducted by ANVUR do not bring a valid contribution to the discussion about agreement between peer review and bibliometrics. Therefore, the papers describing Italian experiments conducted by ANVUR and authored by ANVUR collaborators should not be cited as valid pieces of knowledge.