{'fraction_non_alphanumeric': 0.028755958788251577, 'fraction_numerical': 0.011071813009380286, 'mean_word_length': 4.978625603309584, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Metrics and peer review agreement at the institutional level

In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative. A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review at the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science. This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for national research assessment exercises.


I. INTRODUCTION
Since the 1980s, performance-based research funding systems (PBRFS) were introduced in many countries in order to strengthen accountability of research institutions and steer their strategic behaviour. PBRFS may vary considerably in how they function (Hicks, 2012;Zacharewicz et al., 2019), but they have one element in common: the need to evaluate research. Peer review is often considered the principal method for evaluating scientific products. Indeed, some countries, such as the UK, have opted for research assessment that is primarily based on peer review. In large research assessment exercises, peer review may become costly. To facilitate the assessment, bibliometric indicators can be used to inform the judgement of peers. In Italy, the research assessment exercise, known as the Valutazione della Qualità della Ricerca (VQR), uses an informed peer review approach, where review by selected panellists and external peers is supported by bibliometrics in fields for which bibliometric indicators seem informative (see Ancaiani et al. (2015) for more details).
A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes, or whether they differ substantially. This question has been repeatedly addressed in the context of the UK Research Excellence Framework (REF), culminating in a systematic large-scale comparison between peer review and metrics in the Metric Tide report (Wilsdon et al., 2015, Supplementary Report II). We believe that this report has two crucial shortcomings (Traag and Waltman, 2019): (1) the agreement between peer review and metrics was studied at the publication level, in contrast to the aggregate institutional level at which the REF outcomes are relevant; and (2) the internal agreement of peer review a) Electronic mail: v.a.traag@cwts.leidenuniv.nl itself (i.e. the extent to which different reviewers or different peer review panels come to the same conclusion) was not considered. In the Italian context, ANVUR, the agency tasked with the implementation of the VQR, collected data on peer review and quantified the internal agreement of peer review at the publication level. The results of the analysis showed that the agreement between metrics and peer review is similar to, or higher than the agreement between two independent reviewers (Bertocchi et al., 2015), although this result has been subject to debate De Nicolao, 2016, 2017). In this paper, we use the ANVUR dataset to study the internal peer review agreement at the institutional level. Similar to Traag and Waltman (2019) we find that the agreement between peer review and metrics tends to be higher at the institutional level than at the individual publication level. In addition to Traag and Waltman (2019), we quantify the internal peer review agreement at the institutional level, which is also higher than at the publication level. Most importantly, we find that the agreement between metric and peer review is generally on par with the internal agreement among two reviewers for the fields included in our analysis.
In the next section, we provide a short background of PBRFS and a brief description of the Italian VQR exercise. We then present the collected data and outline the bibliometric methodology, followed by a summary of the main results obtained in our analysis. Finally, in the conclusion, we discuss the connection with broader questions around evaluation.

II. PERFORMANCE-BASED RESEARCH FUNDING SYSTEMS
Since the early 1980s, public management changed around the world. Reforms led to the redesign of the main public administration mechanisms at all levels and in all sectors, including higher education systems and arXiv:2006.14830v1 [cs.DL] 26 Jun 2020 public research organizations. Over the last decades, a considerable number of countries, particularly EU member states, implemented performance-based research funding systems (PBRFS). According to the definition provided by Hicks (2012), and used in Zacharewicz et al. (2019), the main characteristics of PBRFS are the following: • Research output and/or impact is evaluated expost.
• The allocation of research funding depends (partly) on the outcome of the evaluation.
• The assessment and funding allocation takes place at the organisational level.
• PBRFS are a national or regional system.
PBRFS exclude any kind of degree programmes and teaching assessment. Grant based funding, which is based on ex-ante evaluation of grant proposals, is excluded, and so are funding systems that assign funds only on the basis of the number of researchers or PhD students. Furthermore, PBRFS provide, directly or indirectly, tools and mechanisms to allocate research funds. They are performed at the national level, not on the local or institutional level, and they typically do not result in mere suggestions or recommendations to evaluated organisations, but affect the allocation of resources (Hicks, 2012;Zacharewicz et al., 2019).
Many countries have no research performance-based elements in their funding allocation at all (for a detailed examination see especially the work of Zacharewicz et al. (2019)). Countries that do base their funding allocation partly on research performance do so in a variety of ways. This includes countries, such as the United Kingdom and Italy, which both implemented a PBRFS, but using a different approach. The rationales of both exercises may be summarised as follows: (1) steering public funds allocation on the basis of quality, excellence or meritocratic criteria; (2) providing comparative information on institutions for benchmarking purposes; and (3) providing accountability regarding the effectiveness of research management and its impact in terms of public benefits; (Abramo and D'Angelo, 2015;Franceschini and Maisano, 2017). The United Kingdom developed the oldest and best-known assessment exercise, which is nowadays called the Research Excellence Framework (REF), which is a point of reference for many PBRFS. Italy introduced a PBRFS more recently, nowadays called the Valutazione della Qualità della Ricerca (VQR), which was partly inspired by the UK REF, although there are also marked differences.
Whereas the UK REF claims to use metrics moderately and is mainly based on peer review assessments, the Italian VQR makes more extensive use of bibliometric indicators, especially in the STEM and Life Science sectors (Zacharewicz et al., 2019). These differences are not absolute, but gradual: both rely partly on metrics and partly on peer review, but they do so in different ways. Peer review is often seen as a kind of gold standard for research evaluation (Wilsdon et al., 2015). The research community has long debated whether to use metrics or peer review for research evaluation: both methods have strengths and weaknesses. It is in this context that we study the agreement between peer review and metrics. VQR evaluates research outputs of all permanent scientific staff in 96 universities and 39 public research organisations. With reference to the period 2011-2014, a total of 52 677 researchers submitted what they considered to be their best outputs for evaluation (2 for each university researcher, and 3 for each scientist employed in a public research organisation). All in all, 118 036 outputs were submitted, out of which 70% were journal articles. The remaining types of research outputs included a wide range of materials such as book chapters, conference proceedings and even work of arts. Outputs were classified in 16 research areas and ANVUR appointed a Gruppo di Esperti della Valutazione (GEV), a panel of experts, for each research area. The criteria used for evaluation referred to originality, methodological rigour and impact.
In humanities and social sciences (with the exception of Economics and Statistics), a pure peer review system was employed, assisted by external (national and international) reviewers: overall, almost 13 000 reviewers were used in VQR 2011-2014. In Science, Technology, Engineering and Medicine areas (STEM) and in Economics and Statistics, the same procedure is used, but in addition bibliometric indicators are also produced by AN-VUR to inform the panels. More specifically, peer evaluation was integrated with the use of bibliometric indicators concerning citations and journal impact, drawn from the major international databases (see Anfossi et al. (2016), for a description of the bibliometric algorithm used in the exercise). The indicators used were the 5-Year Im-pact Factor and the Article Influence Score (AIS) for the WoS database and the Scimago Journal Rank (SJR) and the Impact Per Publication (IPP) indicators in Scopus. On the basis of the bibliometric algorithm, when citations and journal impact indicators provided contrasting results the paper was not assigned an evaluation class but was rather sent to external review for informed review (IR).

III. DATA AND INDICATORS
The analysis carried out in this paper is based on a sample extracted from the full dataset of research items that were submitted for the VQR 2011-2014. Ths sample was limited to the research areas that made use of metrics, i.e. all STEM areas and Economics and Statistics. A random sample of 10% all publications submitted to VQR was drawn, stratified on the basis of the scientific area. All publications in the sample have been sent to two independent peer reviewers, irrespective of their bibliometric indicators. The response rate of reviewers has been high and the final sample (i.e., the number of articles that were effectively peer reviewed) covered 9.3% of all the articles submitted to the VQR.
We limited our analysis to university institutions, hence excluding public research organisations and other research bodies. We matched the publications with the CWTS in-house version of Web of Science (WoS): the final sample used in this paper includes 4 560 publications, almost 8% of the reference population, submitted by 78 Italian universities. A few universities that only had one or two publications in the sample had no match with WoS at all. It is worth noting that universities were not always included in all areas. Overall, the empirical sample could be considered a reasonable approximation of the population of reference (see also Alfò et al., 2017). Post-stratification analysis confirmed that the sample is also sufficiently representative at the institutional level. The sample comprises in most cases between 6 and 10% of the reference population ( Fig. 1).
As stated, each publication in the sample was assessed by two independent reviewers. We randomly determined which reviewer is considered as reviewer number 1 and which one is considered as reviewer number 2. Each reviewer rated the publication on each of the three criteria of originality, methodological rigour and impact. Each criterion was assigned a score from 1 to 10 and we summed the three scores to obtain an overall score that ranged from 3 to 30.
For each paper included in the sample, we calculated two indicators on the basis of WoS, one at the article level and one at the journal level: (1) the normalised citation score (NCS) for each paper, given by the number of citations divided by the average number of citations of all publications in the same field and the same year; and (2) the normalised journal score (NJS), which is the average NCS of all publications in a certain journal and a certain year. We took into account citations up to (and including) 2015, to be consistent with the timing of the VQR. We used the WoS journal subject categories for calculating normalised indicators. In the case of journals that were assigned to multiple subject categories, we applied a fractionalisation approach to normalise citations (Waltman et al., 2011). Publications within journals in the multidisciplinary category (e.g. Science, Nature, PLOS ONE) were fractionally reassigned to other subject categories based on their references. Besides the bibliometric information calculated from WoS, we also considered the indicators gathered by ANVUR during the VQR itself. Those indicators may come from various sources (e.g. Scopus, WoS, MathSciNet), and for different publications different journal indicators may be used 1 (e.g. 5-year Impact Factor, Article Influence Score, SJR, IPP). All scores were normalised as percentiles with respect to the field definitions as provided by the data source. This procedure allowed the VQR to gain a greater degree of flexibility in practice (Anfossi et al., 2016), but also made the data more heterogeneous, thereby complicating the interpretation of the results. Nonetheless, we include such indicators in this study in order to compare them to the bibliometric information obtained exclusively from WoS. Besides the two reviewers' scores for each paper, we hence obtained the two VQR percentile metrics (a citation metric and a journal metric), and the two WoS metrics (a citation metric and a journal metric). We want to compare the agreement between metrics and peer review in a fair way to the internal agreement of peer review. In order to do so, we consistently com-pare all scores and metrics to the overall score of reviewer 1. Internal peer review agreement is then quantified by the agreement of the score of reviewer 2 with the score of reviewer 1. Likewise, for each metric, we calculate the agreement with the overall score of reviewer 1. By performing the analysis in this way, the agreement between metrics and peer review can be compared in a fair way to the internal agreement of peer review. If we had chosen to compare each metric to the average score of reviewers 1 and 2, this would have already cancelled out some 'errors' in the scores of the reviewers, and as a result, the agreement of the metrics with the reviewer scores would not have been directly comparable to the internal peer review agreement.
At the level of institutions, there are two views of the aggregate scores: a size-dependent view, considering the total over a certain score (e.g. NCS), and a sizeindependent view, considering the average over a certain score (e.g. NCS). For the size-dependent view, we simply take the sum of each score, while for the size-independent view we take the average of each score. Note that at the aggregate level we still speak of the score of reviewer 1 and reviewer 2, even though this refers to two sets of reviewers, not to two individual reviewers. (2019) we do not calculate correlations, but deviations from predicted values. Whereas Traag and Waltman (2019) focused on predicting whether a publication would be classified as 4 * ("World Leading"), based on whether a publication belonged to the top 10% of its field, we here focus on predicting the numerical reviewer score based on the number of (normalised) citations or journal impact. We therefore use a simple linear regression to estimate how metrics can predict reviewer scores. We estimate the regression for each metric separately and each research area (i.e. GEV) separately. We only estimate the regression coefficients for the size-independent case, and then simply scale the predicted value with the number of publications, similar to all other scores. In this way, the estimate of the regression coefficient is not biased by the skewness in the number of publications.

Similar to Traag and Waltman
We calculate two statistics at the institutional level: the median absolute deviation (MAD) for the sizeindependent view and the median absolute percentile deviation (MAPD) for the size-dependent view. These statistics are calculated for each research area (i.e. GEV) separately. We first focus on the size-independent view. Let y i be the average reviewer score for institution i (within a specific research area), and x i the average metric score for institution i. Through simple OLS linear regression, the predicted average reviewer scoreŷ i is then y i (x i ) = α + βx i . The MAD for metric x is then defined as (1) We now shift to the size-dependent view and the calculation of the MAPD. Let p i denote the number of publication of institution i. We then obtain the size-dependent reviewer score p i y i and the size-dependent metric score p i x i . We also simply scale the predicted reviewer scorê y i to obtain p iŷi . The MAPD is then defined as We also calculate the MAD at the individual publication level, the MAPD does not make sense at the individual level, as there is no size-dependent view at that level.
Since these statistics are based on a sample, we try to estimate what would be reasonable outcomes at the population level of all publications submitted to the VQR. We do this by bootstrapping the sample, and repeating the entire procedure to calculate MADs and MAPDs for each bootstrapped sample. We perform 1000 bootstrapped samples and report the interval in which 95% of the bootstrapped statistics fall.

V. RESULTS
We first analyse the outcomes in the size-independent view. We show the citation score (NCS) versus the score of reviewer 1 in a scatter plot (Fig. 2) and the score of reviewer 2 versus reviewer 1 in a scatter plot (Fig. 3). It is readily apparent that there are quite some differences, not only between the NCS and peer review but also between the two reviewers themselves. By calculating the MAD (and MAPD), we quantify the level of differences for each of the scores.
In Fig. 4 we report the size-independent results of the MAD and in Fig. 5 we report the size-dependent results of the MAPD. Both graphs yield relatively similar results. Overall, the agreement between metrics and review is comparable to the internal agreement between two reviewers. There are clearly some differences between research areas. For example, the agreement is generally relatively high in Medicine (low MAD and MAPD), while the agreement is lower in Civil Engineering (high MAD and high MAPD). In one research area, Physics, the agreement between two reviewers is higher than the agreement between metrics and peer review: the MAD for the normalized citation score (NCS) is 1.9, while the MAD between two reviewers is 1. In the size-dependent view, this amounts to an MAPD of 7.9% for NCS and an MAPD of 3.7% between reviewers. For Agricultural and Veterinary Sciences the agreement between two reviewers is also somewhat higher, with an MAD of 0.9 and an MAPD of 5.4% compared to the agreement between NCS and peer review, with an MAD of 1.3 and an MAPD of 4.0%.
Our results show that indicators based on citations (NCS and the citation percentile) show a similar agreement with peer review as indicators based on journal metrics (NJS and journal percentile). Citations and journal indicators could therefore both provide information about evaluation outcomes. The indicators based on WoS (NCS and NJS) also show a similar agreement with review as the more heterogeneous indicators that could be freely chosen from different data sources in the VQR (citation and journal percentiles). This suggests that the heterogeneity of using different data sources does not deteriorate, nor ameliorate, the agreement with review. Finally, the error bars based on bootstrapping are clearly skewed upwards. This is because there are a number of institutions with only a few publications and a relatively small absolute difference. In a bootstrap sample there is a probability that a small institution may be entirely excluded, thereby affecting the calculation of the median. To study the effect of institutions with a small number of publications, we replicated all calculations, including the bootstrapping, on institutions that have at least 5 publications. We obtain qualitatively similar results.
When comparing our institutional results to the results at the individual publication level (Fig. 6), we see that the differences between research areas become less pronounced. Overall, the MAD at the individual publication level is about 3 for all indicators, including peer review itself. Physics and Chemistry show slightly higher levels of agreement (lower MAD) and Medicine and Civil Engineering somewhat lower levels of agreement (higher MAD), but the differences are quite small. The MAD is generally higher at the level of individual publications, compared to the institutional level, showing that "errors" indeed tend to "cancel out" at the aggregate level. It also suggests that the differences between research areas that we observed at the institutional level mostly result from differences in institutional sizes. When institutions are larger more "errors" tend to "cancel out".

VI. DISCUSSION AND CONCLUSIONS
We analysed the agreement between several bibliometric indicators and peer review. The contribution of our analysis is twofold: (1) we analysed the agreement at the institutional level, instead of the individual publication level; and (2) we also quantified internal reviewer agreement at the institutional level. We found that the agreement between bibliometric indicators is on par with the internal agreement between two reviewers. The agreement between peer review and citation-based indicators was comparable to the agreement between peer review and journal based indicators. Finally, as expected, the agreement at the institutional level was higher than at the individual publication level.
Our results are relevant in the context of performancebased research funding systems (PBRFS). In this context, evaluations typically take place at the level of entire institutions. Our results suggest that using peer review or bibliometric indicators, whether citation-based or journal-based, would yield similar outcomes of the evaluation. The agreement between peer review and bibliometric indicators is not perfect, but the differences that arise are comparable to the differences between reviewers themselves. This suggests that similarly different results could have been obtained if other reviewers would have evaluated the publications.
There are several reservations that we should make regarding our results. First of all, our results are based on a single reviewer. We may expect that using multiple reviewers will increase the internal agreement. In reality, most evaluations consider multiple reviewers. It is possible therefore that the agreement between bibliometric indicators and peer review is lower compared to the internal agreement when using multiple reviewers. Based on the current data, we cannot conclude how this would affect the results. It might warrant to study this experimentally by involving more reviewers.
Secondly, bibliometric indicators and peer review may show certain biases. That is, even if the overall level of agreement is comparable, indicators and peer review may show particular differences. For example, reviewers may be biased towards institutions of a higher reputation, and bibliometric indicators may favour certain types of methodologies or topics over others.
Additionally, there may be other effects of using indicators instead of peer review. De Rijcke et al. (2016) show that researchers may try to improve their evalu-ation, and they may seek to augment their bibliometric indicators, leading them to "think with indicators" (Müller and De Rijcke, 2017), instead of pursuing scientific research as they see fit. For example, it has been suggested that Italian authors try to improve their citation statistics by citing each other (Baccini, De Nicolao, and Petrovich, 2019). In a seminal study, Butler (2003) found that the Australian PBRFS stimulated the production of low-impact articles because only productivity was rewarded. This result has been questioned recently by van den Besselaar, Heyman, and Sandström (2017), who found that impact was actually improving after the introduction of the Australian PBRFS. Regardless of the exact findings, it remains challenging to attribute causality of the possible effects of policies in PBRFS (Aagaard and Schneider, 2017 causally influenced by the journal (Traag, 2019). Possibly, citations, journals and peer review are all influenced by common underlying characteristics of the publication. However, it is also possible that the journal affects the peer review outcome so that a publication would have been evaluated differently if it were published elsewhere (Wilsdon et al., 2015). Similarly, it is also possible that citations might affect the peer review outcome. If bibliometric indicators affect the outcomes of peer review, there are a few possibilities. On the one hand, we might want to minimize the influence of bibliometric indicators so that "peer review" really reflects the expert's view.
On the other hand, such an influence may be desirable and may simply reflect the fact that "informed peer review" has been practised. However, if reviewers do not do much more than translating bibliometric indicators into peer review outcomes, there may also be a little added benefit to peer review in this context.