{'fraction_non_alphanumeric': 0.03031693487594638, 'fraction_numerical': 0.02730162892071712, 'mean_word_length': 4.975714845280063, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Metrics to evaluate research performance in academic institutions: A critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative

Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.


Introduction
The Australian Government's ERA (Excellence in Research for Australia) is an attempt by the Australian Research Council (ARC) to rate the research impact of each university on a 5-point scale within 180 Fields of Research (ARC 2010a) using a combination of metrics and peer evaluation by an evaluation committee (2011c). The ERA has been controversial for several reasons, including its reliance on journal ranking (Cooper and Poletti 2011;Haddow and Genoni 2010;Moosa 2011;Northcott and Linacre 2010;Pontille and Torny 2010;Serenko and Dohan 2011), its use of grant income as an indicator (Svantesson and White 2009), its reliance on peer review in some disciplines (Abramo et al. 2009;Abramo and D'Angelo 2011;Taylor 2011), and the high cost (involving a budget allocation of AU$35.8 million; Carr 2009). In addition, there remain questions surrounding the utility of esteem indicators (Donovan and Butler 2007). The use of the indirect index H 2 offers a stronger indication of citation impact: a publication count of n signifies output but not impact; a h-index of n (n publications, each cited n times) is a weak indicator of impact; a H 2 of n is a strong indicator impact because it assures that the citing articles are themselves cite-worthy. A high H 2 index of a publication typically indicates that papers associated with the research front in a field (the highly-cited papers) are based on this publication. Thus, the publication in question contributes to the scientific progress in the field (see Bornmann et al. 2010). The strength of H 2 is that it cannot be manipulated as easily as a citation count on which the conventional h-index relies (Labbé 2010). This is because H 2 relies on citations by substantive papers that are themselves heavily-cited, and overlooks less substantive citations irrespective of frequency. The use of H 2 rewards 'good' science: simple publication counts encourage publication without regard for impact; citation counts and the h-index reward provocative comment and literature reviews that may be often cited; while H 2 rewards foundation science that will be noticed in frequently-cited reviews. This is consistent with the stated aims of ERA to "[drive] research excellence rather than just research quantity" (Jones 2011). The H 2 -index is easily computed using Scopus and Web of Science, and a web application is available to calculate the index with Google Scholar (Thor and Bornmann 2011). One limitation of H 2 is that it develops more slowly than more direct indicators such as citation counts. And like all such indicators, the quality of the H 2 estimates remains dependent on the quality of the underlying bibliometric database (Meho and Yang 2007).
The Australian ERA (Excellence in Research for Australia), like most research evaluations, seeks to appraise research achievements during a defined reference period -in contrast to the conventional application of the h-index which summarises life-time achievement. However, the h-index, the indirect H 2 and other related indices may be applied to defined periods such as the 6-year reference period used in the ERA (2003for ERA 2010, and 2005for ERA 2012.
The ERA seeks to evaluate research achievement in each of 180 defined Fields of Research (FoRs, ANZSRC 2008), defined largely by a prescribed list of journals (ARC 2010b). Derivatives of the hindex, including the indirect H 2 index, apply equally to such selected subsets as to the uncensored literature. This paper seeks to evaluate the utility of the H 2 index for automating an assessment of literature within FoRs, both generally and in the specific case of forestry (FoR 0705 Forestry Sciences). The Australian Research Council's list of journals (ARC 2010b) that define this field has some limitations (Vanclay 2008(Vanclay , 2011, but it nonetheless provides a serviceable yardstick with which to compare institutions. Several aspects of this yardstick are examined in this paper. The ERA rankings are allocated subjectively by Research Evaluation Committees guided by some bibliometric data, including tables showing the standing of institutions within each FoR based on relative citation impact (per article, 7 classes delimited by 0.01, 0.8, 1.2, 2, 4, 8 cites/paper;ARC 2011c) and on percentile analysis (5 classes, top 1%, 5%, 10%, 25%, 50% based on cites/paper; ARC 2011c). ERA 2010 also reported the mean relative citation impact (RCI) for each institution, so that an institution with 50 'good' publications is likely to be ranked higher than an institution with 100 'good' and 100 'average' publications. The disadvantage of RCI is that citation distributions are-as a rule-highly skewed and should not be arithmetically averaged. With percentile ranks, the citation of each paper is rated in terms of its percentile in the citation distribution (Leydesdorff et al. 2011). ERA's use of percentiles is commendable and has many advantages over other current standard techniques (based on mean citation rates). Specific RCIs and percentiles have not been disclosed by ERA, but comparable data for 0705 Forestry Sciences during 2005-10 are reported in this paper.

Data
The present study involves diverse sources of data: 1. the ERA rankings from 2010 (ARC 2011e), 2. the ARC list of journals classified as 0705 Forestry Sciences (Lamp 2011; Annex 1), 3. a compilation of forestry institutions that are active in research and training, and 4. bibliographic records retrieved from Elsevier's Sciverse Scopus, the official data provider to ERA in 2010.
Scopus provided many tools to facilitate the analysis, and most information was retrieved through a search defined by an affiliation-identifier, a list of ISSNs of journals, and the census period (2003-08 or 2005-2010). However, periodic updates of Scopus alter some data, and it is not possible for a casual user to specify a reference date, so the same search conducted on two successive days may, in some instances, return a different number of citations. To minimize the effects of such updates, efforts were made to retrieve relevant Scopus data for each comparison within a short time period, so that entries in any table remain comparable. A further complication is that the casual user of Scopus cannot derive retrospective values of H 2 , because these may be inflated by recent references. Scopus provides some capacity to derive retrospective h-indices, but offers no ability for the casual user to limit citation counts at the second tier required to estimate H 2 ( Figure 1).
A list of 380 forestry institutions (Annex 2) was compiled, initially from Wikipedia (Anon 2011) and Laband and Zhang (2006), but with additions and deletions to focus on tertiary educational institutions active in forest research and training. This list focuses on universities and other tertiary institutions engaged in both education and research, and omits government research institutions whose primary focus is research rather than training. Institutions with no published output during 2005-2010 in 0705 Forestry journals visible to Scopus are omitted from this list. The list should be viewed as representative rather than authoritative, because deficiencies in the 0705 list of journals may cause the omission of some worthy institutions, especially amongst those that publish in non-English journals. The 0705 list (Annex 1) of 85 journals recognised by the 2010 ERA does not include all relevant journals (Vanclay 2011) and has poor coverage of journals in languages other than English. Despite these limitations, it remains the yardstick defined by the Australian Research Council for use in the ERA.

Method
The overall objective of the present analysis is to examine the utility of the indirect H 2 index to calculate automatically the research performance of institutions without the use of journal rankings. In doing so, it explored 4 underlying questions: 1. How effective is the ERA journal list in defining a body of research? 2. How does H 2 compare with ERA ratings? (in forestry and generally) 3. How do global forestry institutions rate on this scale, and what is 'world standard'? 4. How to adjust H 2 for the size of institution? Most of these questions can be addressed with data derived from Scopus, using the advanced search facility. Many searches were restricted to match a FoR-specific journal list, and this can be specified easily and unambiguously by specifying journal ISSNs. Similarly, Scopus has a useful ability to specify an affiliation identifier to uniquely identify institutions. Scopus offers the ability to limit searches, for instance to exclude articles published in 2011, but not the ability to exclude citations received during 2011, so the casual user of Scopus is limited in their ability to backdate analyses. Thus many of the analyses reported focus on the 2012 ERA reference period (2005-10) because the period of citation accrual (8-80 months) more closely corresponds to that experienced during the ERA evaluation. At the time of writing, publications from 2003-08 have accrued citations for 32-104 months, substantially longer than the interval prevailing at the time of the ERA analysis.

How well do journals defined as 0705 Forestry cover the forestry literature?
The analysis reported in this paper follows ERA in relying on three assumptions that define the data under analysis: the use of Scopus as a source of citation data, the use of the ERA journal list within each discipline, and the use of the ANZ classification of science disciplines. Thus the utility of the present analysis depends in part on the extent to which the ERA 0705 journals (Annex 1) adequately represent the forestry literature. Scopus was the official data provider to ERA 2010, and in some instances provides more comprehensive coverage than Web of Science, and more reliable coverage than Google Scholar, so represents a good data source (Falagas et al. 2008, Li et al. 2010. Table 1 shows a summary of the 1412 articles found with a Scopus search for documents published with an Australian author during 2005-10 with the 'Forestry' in the title, abstract or keywords. These articles in Table 1 appeared in 389 journals, including 35 of the 85 journals in Annex 1. These documents may be included in ERA explicitly if the journal is classified 0705 Forestry (e.g., Forest Ecology and Management, and 84 other journals in Annex 1), implicitly if the journal is classified 07 Agriculture (e.g., Agricultural and Forest Entomology) or MD Multidisciplinary (e.g., Science and Nature), or excluded if the journal is classified otherwise. In addition, some journals receive more than one classification (e.g., Agricultural and Forest Meteorology classified as 0401 Atmospheric Sciences and 0705 Forestry Sciences), and ERA allows submitting institutions to nominate which of the two FoR codes should be assigned to the article. Not all journals indexed by Scopus are recognised by ERA, and of the 1412 articles reported in Table 1, only 1304 are recognised by ERA. Of these, 605 would be detected by a search based on 0705 journals, but since some journals have more than one classification, the number of articles included in a 'hand-crafted' 0705 submission could vary from 550 (the minimal case with articles in journals classified only as 0705) to 802 (all articles both explicitly and implicitly classified as 0705 Forestry). Table 1 is indicative, because not all forestry articles include the keyword 'forestry' (e.g., articles about wood science may not mention forestry), and not all articles containing the word 'forestry' are primarily about forestry science (e.g., an article about sediments affecting coral might mention casually forestry and agriculture as possible sources of sediment). Nonetheless, Table 1 offers some reassurance that the ERA classification recognises a substantial proportion of the forestry material indexed by Scopus. Note that this view based on articles is more favourable than a comparable view based on journals, because the leading journal (Forest Ecology and Management) carries 16% of the articles, and 16% of the journals carry only one article. Whilst Table 1 is indicative for forestry, different patterns may arise for other disciplines and FoRs. Further flexibility was introduced in the 2012 ERA by the 'reassignment exception', which allows "articles which have significant content (66% or more) that could best be described by a particular four-digit FoR code" to be re-assigned to that Field of Research (ARC 2011a). Forestry fares rather well in such a test (Table 1), because much of forestry research in Australia appears in journals classified unambiguously as 0705 Forestry Sciences. This is not the situation for all disciplines, and microbiology offers a striking contrast to forestry. A search of Scopus for scientific articles containing the word 'microbiology', with an Australian affiliation published during 2005-10 yields a total of 3552 articles in 582 journals. Of these 582 journals, 48 are amongst the 162 journals on the ERA list of 0605 Microbiology journals. As with the forestry example, the distribution of articles is skewed, with one journal (Journal of Virology) publishing 7% of the articles and 8% of the journals bearing only one article. Table 2 illustrates that the FoR 0605 Microbiology is defined more ambiguously within ERA, with many of the relevant journals having multiple FoR classifications. The major ambiguity is with 1108 Medical Microbiology, with over one-third of the articles in the 'microbiology' search receiving multiple classifications including both these classifications (FoR 0605 Microbiology and 1108 Medical Microbiology). Table 2 reveals that an automated search of Scopus can provide a good approximation of an institution's ERA submission in 0705 Forestry Sciences, because it may retrieve 46% of the relevant material, while the ERA 2010 rules allowed the institution to vary this content between 42% and 62% of this material. In contrast, multiple classifications of journals (especially between 0605 Microbiology and 1108 Medical Microbiology) mean that while an automated Scopus search for 0605 Microbiology journals will still retrieve 47% of the material, an institution may arrange their submission to include as little as 3% or as much as 64% of the material. The flexibility will increase with the 'reassignment exception' in ERA 2012. The ambiguity reported in Table 2 applies equally to the ability of the ERA to compute relative citation impact (RCI) and to compile percentile analyses. This suggests that some reappraisal of the units of evaluation within ERA may be warranted. What is the basis for the ERA ratings for 0705 Forestry?
ERA has published the 2010 methodology (ARC 2011c), but not the data used to derive the rankings of Australian forestry institutions. It is not possible for the casual user of Scopus to reconstruct the data used in ERA 2010, but comparable data for the ERA 2012 reference period (2005-10) can be compiled. ERA standardises citation counts to calculate a relative citation impact (RCI) by calibrating to the global average citations per publication (cpp) within each FoR. Table 3 illustrates the national and global benchmarks used to standardise and classify 0705 Forestry submissions to ERA, compiled for the period 2005-10. One question that arises from Table 3 and the way ERA 2010 used percentile analysis is whether the citation pattern created by a global aggregation reflects that of individual institutions, or whether institutions vary in their citation patterns. For instance, one might suspect that dedicated research institutions might exhibit a different pattern of citations than an extension agency: the former might have a more diverse pattern (some highly-cited successes, a few speculative proposals infrequently cited) than the latter which might be expected receive consistent but modest citations. However, this speculation is not well supported by the evidence. Two major universities with elite forestry programs (Swedish University of Agricultural Sciences SLU, University of British Columbia; Annex 2) exhibit citation patterns similar to the pattern accruing to global pooled publications, suggesting that the pooled global data is adequate to calibrate Australian university output. Few extension agencies publish prolifically enough to establish a reliable trend, but the pooled data from several extension agencies exhibit a pattern similar to that of the Center for International Forestry Research (CIFOR) and the Australian CSIRO ( Figure 2). Although the evidence suggests that the pooled global data is a reasonable basis to standardize citation counts, it may not be desirable to do so. At the time of writing (August 2011), the mean citations per 2010 publication in 0705 Forestry was 0.9 (cpp, Table 3, global average), so a modest number of citations can easily inflate to high RCI (e.g., 8 citations is sufficient to attain the highest RCI class VI, Table 4). The threshold of the top percentile for 2010 publications is only 7 citations, so a modest cluster of citations (e.g., arising from a special issue of a journal) can very easily leverage an article into the top percentile. It seems imprudent to introduce such heavy weighting of the most recent publications, at a time when their long-term acceptance remains uncertain. Radicchi and Castellano (2011) show that for recent publication "citation patterns are generally far from stationary". Such weighting creates an unreasonable incentive for academics to indulge in citation manipulation, either by writing "potboilers" or by forming citation clubs.
A third problem with the data reported in Table 3 is that the mean citation counts (cpp) need to regarded cautiously, because the distribution of citations is highly skewed. The problem is evident internally in Table 3, since the mean values (left columns) differ greatly from the medians (right column). This difference is further illustrated in Figure 3 which reveals the skewed distributions, with few articles receiving over a hundred citations, and hundreds of articles receiving no citations. These distributions highlight the necessity for a percentile rank approach (rather than e.g. the mean RCI) in the analysis of research impact. The data presented in Table 3, plus the list of 0705 journals (Annex 1), allow the compilation of a table similar to that presented to Research Evaluation Committees (RECs) to assist them in their appraisal of institutions (ARC 2011c). Table 4 offers such a compilation for 0705 Forestry during 2005-10. The h-index and H 2 index are not included amongst the data presented to the RECs, but are included here as a possible synthesis or alternative. Our view is that Table 4 offers too much data and too little information, and that a more concise synthesis would be more insightful. It may be sufficient simply to present the percentiles in conjunction with the H 2 . Tables 1 and 2 have indicated some of the complications that arise as a result of the multiple classification of some journals. This problem has been exacerbated by the introduction of the 'reassignment exception', which allows great flexibility in the material that may be offered in an ERA submission. Table 5 indicates how, even in an unambiguously-specified field as forestry, this flexibility can lead to large changes in the data tabulated for the RECs, and reiterates the robustness and necessity of the h-index and H 2 . Table 5 illustrates 3 cases: one that simply includes all material included in journals classified 0705 Forestry; another all-inclusive case that uses the reassignment exception to include all material with relevant keywords (forestry, silviculture, timber, eucalyptus), and a third selective case that uses the opportunity presented by the 'reassignment exception' to build the strongest possible case by omitting infrequently-cited material. It is noteworthy that the tabular REC data (RCI classes, percentiles) are easily manipulated in this way, but that the indirect H 2 index remains robust. The apparent three-fold increase in 'high impact outputs' (i.e., the proportion of work in RCI classes IV and V, and in top 10%) in the selective case is enabled through the flexibility of the 'reassignment exception', the ERA threshold of 50 articles, and use of means and percentiles. How does H 2 compare with ERA ratings?
ERA attempts to calibrate the performance of universities in each of several Fields of Research (FoRs, ANZSRC 2008) defined via a prescribed set of journals (ARC 2010b). Most of the journal output for each institution can be retrieved from Scopus, because journal articles usually indicate an author's institutional affiliation. However, the journal classification is not unambiguous, with some journals assigned to more than one FoR (Tables 1 & 2), and this limits the ability to automate the computation of H 2 and other indicators of impact. In addition, the ERA ranking accounts for the publications of all staff within an institution at the census date (31 March of the preceding year), including newly-appointed staff whose earlier publications may not bear their new institution's name. Thus an automatically-compiled assessment from Scopus could differ from an institution's submission in both staff composition and in the selection of articles in multi-classified journals. Notwithstanding these differences, a Scopus compilation should offer a good approximation of ERA submissions by established institutions.
The basis for ERA rankings in 2010 varied between disciplines. In most areas of science considered in this paper, rankings were based on research outputs (ranked journals, citation analysis), grant income, esteem (membership of learned societies, research fellowships) and applied measures (patents, plant breeder's rights). Research outputs were assessed over a six year reference period (2003-08 in ERA 2010), whilst other indicators were assessed over a three year period (2006-08). It is likely that these indicators were correlated, since for instance, publications are a pre-requisite for grant success, but no data on these correlations and the implied redundancy have been made public. Thus a citation analysis drawn from Scopus (or other data providers) is unlikely to explain fully the ERA ratings, but it may offer some useful insights.
Indirect H 2 indices were computed for 0705 Forestry (including all journals classified 0705, but none classified 07 or MD) for the four institutions that received ERA ratings in 2010 (Table 4). Whilst the results show promise of a strong correlation between H 2 and ERA ranking, the sample size is too small for definitive conclusions.
Because the eligible forestry literature is unambiguously defined in the ERA rules (Table 1), it is feasible to automate a Scopus search to compute H 2 indices to assess performance of the institutions involved. However, forestry appears unique in this regard, and the ambiguous definition of the eligible literature in many other disciplines (Table 2), coupled with the other indicators involved (grant income, esteem, applied measures) means that automated estimates of H 2 do not correlate well with the ERA ranking of institutions (Table 6). However, Table 6 may offer an unduely pessimistic view, because it samples ill-defined FoRs (cf. Table 2), and compares ERA rankings from the reference period 2003-08 with H 2 indices derived for 2005-10 (due to limitations of Scopus). It appears desirable to re-examine both the scope of the Fields of Research and the FoRs assigned to journals appears needed, to limit 'game-playing' (e.g., Bornmann 2010) by institutions preparing submissions, and to allow the ERA evaluation process to be streamlined and partially automated. It is noteworthy in Table 6 that different disciplines have different citation patterns, so that a H 2 value of 5 in 1105 Dentistry and 0608 Zoology corresponds to an ERA rating of 3 ("At world standard"), whereas a H 2 of 7 is required to achieve the same rating in 0912 Materials Science and 1115 Pharmacology. Such differences are commonly observed: for instance, Slyder et al. (2011) reported that forestry articles received more citations than geography articles from the same institution. Table 6 also reveals that H 2 performs better than other indicators (such as h-index) in well-defined FoRs (such as 1105 Dentistry with 83% correct) than in ill-defined FoRs (such as 0608 Zoology).

How do forestry institutions compare on this scale?
Forestry institutions worldwide attained H 2 scores between 0 and 11 during the 6-year period 2005-10 ( Figure 4; Annex 2). The distribution of these scores is skewed, with many low scores, and few high scores. This distribution suffers censorship of three kinds: it includes only articles in 0705 Forestry journals (sensu stricto, Annex 1) and overlooks work in other journals (e.g., Nature and Science), some of which may be highly-cited, and so under-represents work of some strong institutions. It is biased towards Anglophone institutions, because of the English-language bias in Scopus (Moya-Anegón et al. 2007) and in the ERA 0705 list of journals (Annex 1). It focuses on institutions with greater research activity, and does not attempt a complete survey of all institutions, so underrepresents institutions with low research output. Nonetheless, it offers an indicative distribution of institutional performance. Such distributions pose some challenges, in that that the mean (e.g., relative citation index) is meaningless, but value of percentiles (i.e., the 1%, 5%, 10%, 25% and 50% reference points used by ERA 2010) remain relatively robust, because of the integer nature and limited range of H 2 .  There is a high correlation between the H 2 index and the conventional h-index (0.95, P<0.001, Figure  5), so the question arises whether the established h-index (which is easier and more convenient to derive) has the same utility as the H 2 index. Similar results are reported by Bornmann et al (in press). However, Figure 5 illustrates that the same H 2 may have a wide range of h-index, and that the same hindex may give rise to a several different H 2 indices. For instance the following eight institutions have the same H 2 (6) (7), Eidgenössische Technische Hochschule Zürich (6), Santiago de Compostella (Mexico,5) -and these are arguably ranked in order of impact in forest research output. If it is accepted that the former eight are comparable, and the latter four are ranked, then the implication is that the indirect H 2 discriminates more effectively than the conventional h-index, at least in terms of the impact achieved by these institutions.

How to adjust for size?
Despite its stated aim to "[drive] research excellence rather than just research quantity" (Jones 2011), ERA appears to favour large institutions. In many evaluation systems including ERA and other university ranking systems, the top ranks are occupied by institutions that are both good and large, whereas the mid-ranks may be populated by a diverse mix of entities that are either good or large. Thus one of the challenges of an evaluation system is to discriminate between the often confounding effects of quality (citation impact) and quantity (publication output). The h-index and H 2 are both influenced to some extent by the number of contributors, so it is interesting to use the present data to explore the effect of size ( Figure 6). Outputs for four forestry institutions have been reported above, based on numbers of research staff at these institutions and nationally reported by Sinclair (2010). The single-author datum in Figure 6 reflects the greatest H 2 of single-authored papers. These data suggest a logarithmic relationship between staff numbers and H 2 ( Figure 6), with the line of best fit H 2 = 3.0 + 1.05 Ln(N), suggesting that one way to adjust for research team size may be H 2 -Ln(N).

Conclusion
Some of the bibliometric data (especially mean RCI) collated by the ERA and tabulated for presentation to Research Evaluation Committees (RECs) are potentially problematic. One issue is the use of mean RCIs despite the statistical issues associated with skewed distributions. Another limitation is that the computation of percentiles on a year-by-year basis places undue weight on most recent publications, creating an opportunity for manipulation. The tabular data comprising RCIs and percentiles offer the RECs great detail but little synthesis, and it may be more effective to offer additional syntheses such as the indirect H 2 which may be helpful to RECs. The indirect H 2 index is objective, can be computed automatically and efficiently from databases such as Scopus, is resistant to manipulation, and a good predictor of impact (in the sense pursued through ERA). However, efficient calculation of indices such as H 2 requires revision of both the number and diversity of the Fields of Research (FoRs), and of the multiple FoRs assigned to journals within the ERA system.
Since H 2 is a newly introduced indicator it is important that further studies on its usefulness and validity should be conducted. Independent of the results of these studies, it is always necessary that this indicator is used in combination with standard bibliometric measures (e.g., RCI classes). Research performance is a multi-dimensional issue which should be reflected by more than one indicator.