{'fraction_non_alphanumeric': 0.030619865571321882, 'fraction_numerical': 0.013336178384722074, 'mean_word_length': 5.195307336417713, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Scores of a specific field-normalized indicator calculated with different approaches of field-categorization: Are the scores different or similar?

Usage of field-normalized citation scores is a bibliometric standard. Different methods for field-normalization are in use, but also the choice of field-classification system determines the resulting field-normalized citation scores. Using Web of Science data, we calculated field-normalized citation scores using the same formula but different field-classification systems to answer the question if the resulting scores are different or similar. Six field-classification systems were used: three based on citation relations, one on semantic similarity scores (i.e., a topical relatedness measure), one on journal sets, and one on intellectual classifications. Systems based on journal sets and intellectual classifications agree on at least the moderate level. Two out of the three sets based on citation relations also agree on at least the moderate level. Larger differences were observed for the third data set based on citation relations and semantic similarity scores. The main policy implication is that normalized citation impact scores or rankings based on them should not be compared without deeper knowledge of the classification systems that were used to derive these values or rankings.


Introduction 1
It is one principle in the Leiden manifesto for the professional application of bibliometrics to use field-normalized scores instead of simple citation counts (Hicks, Wouters, Waltman, de Rijcke, & Rafols, 2015). These scores reflect the impact of papers against the backdrop of their reference setspapers published at the same time and in the same field. An important topic in the calculation of these scores is the definition of fields, which are used as reference sets (Wilsdon et al., 2015;Wouters et al., 2015). Four different approaches of fieldcategorization are currently (mainly) used for normalizing impact without a clear preference for one alternative: (1) journal sets, (2) intellectual assignments, (3) citation relations, and (4) semantic similarity scores. Waltman and van Eck (2019) reported Pearson correlations between field-normalized citation scores using Web of Science (WoS) subject categories and classifications based on citation relations for faculties, departments, and research groups at selected universities. They found correlations of 0.89 and higher as well as mean absolute score differences of 0.17 and lower. Scheidsteger, Haunschild, Hug, and Bornmann (2018) studied the concordance of field normalized scores for publications of a research institution focused on computer science using WoS and Microsoft Academic. Their normalization procedure used different classification schemes (WoS subject categories and fields of research from Microsoft Academic). They found a good agreement of the statistical concordance of the scores.
In this study, we compare normalized citation scores, which have been calculated based on the three approaches of field-categorization to build reference sets. We are interested whether they lead to the same, similar, or different scores for the same papersif the formula for calculating the scores is held constant. Since all approaches are in current use for fieldnormalization in similar research evaluation contexts, we expect similar scores. Large differences between the scores could question the use of field-normalized scores in research evaluation, as long as no standard approach has been established. Such a study of possible differences is needed because the effect of the choice of a classification scheme on the values of the normalized indicators is largely unknown. When comparing normalized citation impact values of individual publications or aggregates thereof (e.g., scientists or universities) such effects should be known. If the effects are large, normalized values that were calculated using different classification schemes should not be compared with each other.
This study focuses on chemistry and related sciences utilizing a comprehensive dataset from CAS, a division of the American Chemical Society. CAS offers the largest database of the literature in these fields including intellectual assignments of fields to papers.

Approaches of field-classification
This study compares the agreement of normalized citation scores for the same papers, which have been calculated based on the following three field-categorization approaches: (1) The most frequent approach in bibliometrics is to use subject categories that are defined by Clarivate Analytics for WoS or by Elsevier for Scopus to assign papers to fields.
The subject categories pool journals to sets, which publish papers in similar research areas (e.g., biochemistry or economics). It is an advantage of journal sets that they define a multidisciplinary classification system covering all research areas (Wang & Waltman, 2016).
It is a disadvantage of the sets that they stretch to their limits with multi-disciplinary journals (e.g., Nature or Science) and journals covering many subfields (e.g. Physical Review Letters, or The Lancet). These journals cannot be reliably and validly assigned to one field (Haddow & Noyons, 2013) on the journal basis. However, the papers of multidisciplinary journals can be reassigned on a paper-level (Evidence, 2010).
(2) To overcome the limitations of journal sets, Bornmann, Mutz, Neuhaus, and Daniel (2008) propose to use mono-disciplinary classification systems (Waltman, 2016), e.g., CAS sections in chemistry and related areas Bornmann, Schier, Marx, & Daniel, 2011), MeSH (Medical Subject Headings) terms in biomedicine Leydesdorff & Opthof, 2013;Strotmann & Zhao, 2010), or PACS (Physics and Astronomy Classification Scheme) codes in physics and related areas (Radicchi & Castellano, 2011). In these systems, experts in the field or the authors themselves assign each specific paper to the corresponding subfield, highlighting the most important aspects of the papers. It is an advantage of these systems that they have been introduced to reflect the subfield patterns in specific fields. Their disadvantage is that they can only be used for the normalization of papers from one discipline (and related areas).
(3) Waltman and van Eck (2012) introduced a multi-disciplinary classification system, which is based on direct citation relations between papers. The algorithm for computing the classification system needs three basic parameters as input in addition to the direct citation network: (i) the number of levels of the system, (ii) the resolution parameter, and (iii) the minimum number of papers per class (field). The approach is already in use in the Leiden ranking (see http://www.leidenranking.com/) for the calculation of normalized impact scores.
The empirical results of Klavans and Boyack (2017) indicate that algorithmically constructed classifications are more accurate than classifications based on journal sets. Similar positive results have been published by Perianes-Rodriguez and Ruiz-Castillo (2016). Leydesdorff and Milojević (2015) criticize the classification system as follows: "Because these 'fields' are algorithmic artifacts, they cannot easily be named (as against numbered), and therefore cannot be validated. Furthermore, a paper has to be cited or contain references in order to be classified, since the approach is based on direct citation relations" (p. 201). It seems yet that Sjögårde, Ahlgren, and Waltman (2020) might have found a solution for this problem. Very recently, Clarivate Analytics introduced an algorithmic classification with three levels based on WoS data (Szomszor, Adams, Pendlebury, & Rogers, 2021).
(4) Boyack and Klavans (2018) provided a publication classification system that is based on semantic similarity scores (i.e., a topical relatedness measure). The relatedness measures were based on words in titles and abstracts of PubMed publications along with their MeSH terms. They used PubMed records from 1975 onwards for the clustering procedure according to the topical relatedness measures because only very few records before 1975 contained abstracts.

Statistics
The overview of Waltman (2016) demonstrates that several different approaches of calculating field-normalized scores have been developed. In this study, we use the so-called normalized citation score (NCS) to compare field-normalized scores, since it is still the most frequently used approach. For the calculation of the NCS, each paper's citation count is divided by the average citation count in a corresponding reference set. The reference sets are defined by the papers, which belong to the same field (as defined by the field categorization approach) and publication year as the focal paper. If, for example, the paper has 3 citations and the average in the field is 10.67, the NCS of the paper is 3/10.67=0.28. The NCS is formally defined as = where ci is the citation count of a focal paper and ei is the corresponding citation rate in the field (Lundberg, 2007;Rehn, Kronman, & Wadskog, 2007;Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011). Since the number of citations received by a paper depends on the time since publication, the NCS is calculated for publications from the same year. Using the different approaches of field-categorization, we calculated six NCS for every paper: NCSWoS (based on WoS journal sets), NCSCAS (based on CAS sections), NCSL15, NCSL12_2lvl, NCSL12_3lvl (all three based on citation relations), and NCSST (based on topical relatedness measures). Statistical analysis was done using R (R Core Team, 2018).
In this study, we are interested in the relationship between NCSWoS, NCSCAS, NCSL15, NCSL12_2lvl, NCSL12_3lvl, and NCSST to investigate the extent of agreement and disagreement between the different NCS values. We group the papers in our dataset according to the Characteristics Scores and Scales (CSS) method. This method was proposed by Glänzel, Debackere, and Thijs (2016) for normalizing citation counts. However, we do not use the CSS method here for normalizing the citation counts, only for grouping the papers according to their citedness as represented by their NCS value. For each NCS separately (NCSWoS, NCSCAS, NCSL15, NCSL12_2lvl, NCSL12_3lvl, and NCSST), the CSS classifications are obtained by (1) truncating the publications at their mean NCS and (2) recalculating the mean of the truncated part. Performing this procedure three times leads to four impact classes. Following Glänzel, et al. (2016), we labeled the four classes with "poorly cited", "fairly cited", "remarkably cited", and "outstandingly cited". The poorly cited papers are below the average impact of all papers; the other three classes are above this average and further differentiate the papers in the high impact sectors.
We undertook 15 pairwise comparisons to investigate the differences between the three NCS variants. Each pair is compared in a 4 x 4 contingency table. The cells in the diagonal of the table reveal the papers, which have been assigned to a CSS class in agreement of both NCS, and the share of papers assigned in agreement are calculated. This procedure leads to a similarity measure referred to as "level of agreement". We further calculated the weighted Kappa coefficient in this study, which is a robust alternative to the share of agreement, since the possibility of agreement occurring by chance is taken into consideration (Gwet, 2014). Table 1 shows the weights that we used in the calculation of the Kappa coefficient. The weighted Kappa considers that disagreements between the various categories are different. For example, the difference between "Poorly cited" and "Outstandingly cited" is larger (and should have less weight) than the difference between "Poorly cited" and "Fairly cited" (which should have more weight). The weights in Table 1 are frequently used in studies measuring agreement with weights. The weighted Kappa coefficients were calculated using the R package 'irr' (Gamer, Lemon, & Singh, 2019).
We abstained from calculating correlation coefficients in this study, because we are interested in the agreement between two NCS (Lowenstein, Koziol-McLain, & Badgett, 1993).
Correlation is a poor substitute for agreement. For example, systematic bias might be ignored.
Suppose NCSCAS, and NCSWoS have a perfect correlation (see the example of x and y above), but the NCSCAS consistently measures citation impact 0.5 levels lower than the NCSWoS. In this case, the NCS would measure citation impact very differently despite the high correlation between both NCS variants. are also assigned to a secondary section. To avoid multiple classifications of publications in this study, only the primary section assignment is used following previous studies Bornmann, et al., 2011). The section assignments are human-curated by scientists at CAS with specialized knowledge that lets them accurately extract and verify data and insights from each publication. The classification does not seem to be affected by the "indexer effect": according to Braam and Bruil (1992), the indexer classification accords with author preferences for 80% of the publications. We calculated the NCSCAS values using the CAS sections and the citation counts from the CAplus database. The WoS subject categories and the corresponding number of papers that are in our matched data set with at least 5000 papers are shown in Table 2. Besides chemical WoS subject categories, many biological and a few physical WoS subject categories can be observed in this list of most frequently occurring classifications in our data set.  density, although L12_2lvl has a slightly smaller paper density than WoS and CAS. WoS has a larger variation of paper densities across the fields than CAS which can be expected when the overlap of a multi-disciplinary classification system with a mono-disciplinary classification system is analyzed. Also, L12_3lvl, L15, and ST form a group of rather similar paper densities.  Figure 2 visualizes the change of CSS assignment of the analyzed papers as an alluvial diagram. The alluvial diagram was produced using the R package 'alluvial' (Bojanowski & Edwards, 2016). The most changes can be observed between CSS classes 1 and 2. The color is assigned to the CSS classes: 1 in gray, 2 in red, 3 in green, and 4 in blue. Papers that stay in the same CSS class are represented by a band of the color of the class. If papers change the CSS class from one classification system to another, they are represented by a multi-color band (e.g., gray and red if the papers change from CSS class 1 to CSS class 2). The broader the band, the more papers are represented. In the following analysis, we observed that there was a reduction in similarity measures due to many changes of CSS classes.    Our results do not seem to be affected by the data reduction due to the multiple matches with the partly overlapping publication sets of the different classification systems. We calculated Lin's concordance coefficients for selected combinations with single matches only, publications. Such comparisons indicate that the agreement of WoS and L15 is slightly higher in the fields covered by the mono-disciplinary databases CAplus and PubMed than in the single match of the multidisciplinary classification systems.

Discussion
According to Ioannidis, Boyack, and Wouters (2016) "the basic premise of normalization is that not all citations are equal. Therefore, normalization can be seen as a process of benchmarking". Although it is standard in bibliometrics to use field-normalized citation scores for cross-field comparisons (of universities, for example), different approaches exist for calculating these scores. The differences refer either to the method of calculating the scores (percentiles have been proposed as an alternative to scores based on average citations, Bornmann & Marx, 2015) or to the approach of field categorization which are used to build the reference set for each paper. In this study, we addressed the second aspect by comparing the normalized scores, which have been calculated based on three different approaches.
The analysis of the scores basically reveals an agreement which is at least at the moderate level except for the comparison of SciTech with CAS. Since we used the same method for calculating the scores based on the different approaches, the moderate level is lower than the level that we expected. The parallel use of the different approaches in the current research evaluation practice should have led to a generally higher level of agreement.
The main policy implication is that normalized citation impact scores or rankings based on them should not be compared without deeper knowledge of the classification systems that were used to derive these values or rankings. However, our results also show that normalized scores based on intellectual field assignments are more in agreement with scores based on journal sets than with scores based on citation relations or semantic similarity scores. Thus, one can expect more similar scores based on intellectual assignments and journal sets than on algorithmically constructed classification systems. The reason for the similarity might be that intellectual assignments and journals are better rooted in the disciplines than virtual constructs based on algorithmically constructed classification systems. Another possible explanation is that classification systems of similar granularity are more likely to produce similar results.
The WoS journal sets and the intellectual assignments by CAS have overall a similar granularity and show a substantial agreement. However, the classification systems L12_3lvl, and ST show only a moderate agreement although they have a similar granularity. CAS Sections were developed by CAS scientists with specialized knowledge in scientific disciplines and used to accurately index data and insights from each publication. According to Sugimoto and Weingart (2015), the establishment of new journals is a sign of emerging new disciplines. This might help to explain the similar results for WoS journal sets and CAS sections.
The results of this study should be interpreted against the backdrop that the main results of this study focus on one discipline only: chemistry and related areas. However, selected comparisons without the focus on chemistry and related areas showed very similar results. Furthermore, other statistical analyses could be performed. It is not clear whether our results can be generalized. Thus, we encourage similar studies with data from other disciplines using different statistical methods and as many classification schemes as possible.