{'fraction_non_alphanumeric': 0.03357403409513884, 'fraction_numerical': 0.01635276309585846, 'mean_word_length': 5.023015991630548, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

Lee, Walsh, and Wang (2015) - based on Uzzi, Mukherjee, Stringer, and Jones (2013) - and Wang, Veugelers, and Stephan (2017) proposed scores based on cited references (cited journals) data which can be used to measure the novelty of papers (named as novelty scores U and W in this study). Although previous research has used novelty scores in various empirical analyses, no study has been published up to now - to the best of our knowledge - which quantitatively tested the convergent validity of novelty scores: do these scores measure what they propose to measure? Using novelty assessments by faculty members (FMs) at F1000Prime for comparison, we tested the convergent validity of the two novelty scores (U and W). FMs' assessments not only refer to the quality of biomedical papers, but also to their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel (tags"new findings"and"hypothesis")? We used these and other tags to investigate the convergent validity of both novelty scores. Our study reveals different results for the novelty scores: the results for novelty score U are mostly in agreement with previously formulated expectations. We found, for instance, that for a standard deviation (one unit) increase in novelty score U, the expected number of assignments of the"new finding"tag increase by 7.47%. The results for novelty score W, however, do not reflect convergent validity with the FMs' assessments: only the results for some tags are in agreement with the expectations. Thus, we propose - based on our results - the use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.


Introduction
For many years, peer review was the only method used for research assessments (Bornmann, 2011). Since the 1980s, however, peer review has been increasingly complemented by bibliometric indicators. According to Belter (2018), "bibliometrics are now regularly used in national research evaluation exercises in countries around the world … and are incorporated in many high profile university rankings such as the Times Higher Education ranking [see https://www.timeshighereducation.com/world-university-rankings] and the Shanghai Ranking [see http://www.shanghairanking.com]. Bibliometrics are also routinely used at the individual level to evaluate researchers for promotion and tenure and, in some cases, to make funding decisions." Today, bibliometrics is often employed in so-called informed peer review processes, in which peers assess scientific units (e.g., institutions) based on bibliometric indicators (and further metrics) (Gunashekar, Wooding, & Guthrie, 2017). Berger and Baker (2014) list some factors which drive the increasing use of bibliometrics in research evaluation: "shrinking budgets, heightened competition, increasing scientific research output, and longevity of the major metrics. Also important are what may seem to be contradictory trends; technology and globalization. Technology has spurred the enhancement of tools used to generate bibliometric data, but also has provided more opportunities to measure different data, causing some in the research community to question the importance of the standard metrics" (p. 83). Today, research assessments based on bibliometrics are integral parts of many scientific activities (Moed, 2017).
Indicators are measurable quantities which are "substitutes for something less readily measurable and is presumed to associate with it without directly measuring it" (Wilsdon et al., 2015, p. 5). Bornmann and Marewski (2019) coined the term bibliometrics-based heuristics for scientific performance assessments if the decisions are based on only two pieces of information from the available indicators space: citations and publications. Most of the indicators which have been developed in bibliometrics hitherto focus on four major areas: "productivity, collaboration, research topics, and citation impact" (Belter, 2018). For example, citation counts are frequently used proxies for scientific impact, although past research might influence current research which is not reflected in citations (e.g., because the impact is induced by inspiring conference presentations).
In recent years, another area of indicators has been added to the established areas by introducing indicators which might capture the novelty of research described in publications (Uzzi et al., 2013;Wang et al., 2017). Novel combinations of existing knowledge are seen as prerequisites for important research able to inspire later research. The novelty indicators are usually based on counting unusual combinations of cited references in papers. Since novelty is seen as related to unusual combinations of existing knowledge, this concept has been transferred to bibliometrics based on cited references (as proxies of existing knowledge in the archives). The new area of novelty indicators is of specific interest for the field of research evaluation, since this is interested in research leading to new breakthroughs (Winnink, Tijssen, & van Raan, 2016): "Scientific progress is driven by important, infrequent discoveries that cannot be readily identified and quantified, which makes research assessment very difficult" (Rodríguez-Navarro, 2016).
When new indicators are introduced in scientometrics, the first task is to measure whether these indicators really measure what they propose to measure (Panel for Review of Best Practices in Assessment of Research et al., 2012). In a recent empirical study, Tahamtan and Bornmann (2018) interviewed authors of landmark papers in bibliometrics to investigate whether the great potential of their papers is reflected in the cited references. Tahamtan and Bornmann (2018) conclude that the ideas for the papers might not necessarily have been inspired by past publications. The cited literature "seems to be important for the contextualization of the idea in the field of scientometrics. Instead, we found that … [the] ideas are the result of finding solutions to practical problems, result from discussions with colleagues, and profit from interdisciplinary exchange" (p. 906). Thus, the results of Tahamtan and Bornmann (2018) question the common use of cited references to measure novelty. In other words, the results of this qualitative study might be a first hint that cited references are not useful data for measuring the novelty of papers.
Thus, we started to conduct this additional quantitative study in which we explore the concepts of measuring novelty further. We compared novelty scores with novelty assessments by peers (experts in the corresponding fields) to test the convergent validity of the proposed metrics. The convergent validity of the scores would be given, if they were in fact positively related to the assessment by peers. The study of convergent validity based on quantitative empirical data seems necessary, since Lee et al. (2015) already "encourage use of such metrics as a new tool for science policy indicators" (p. 694).

2
Measuring creativity based on novelty (scores) In the last two decades, scholars have used various types of bibliographic information and methods to identify breakthrough-class papers, such as Redner (2005), Schneider and Costas (2017), and Ponomarev, Williams, Hackett, Schnell, and Haak (2014). Winnink et al. (2016) investigated the specific characteristics of breakthrough-class papers which they identified using various bibliographic information (e.g., cited references or number of authors) and methods proposed hitherto. Identifying and measuring novelty in science based on quantitative indicators has a long tradition (in scientometrics and beyond), since novelty is regarded as one of the most important conditions for excellent research.
Creativitythe medium in which novel directions in research emergehas frequently been interpreted as an evolutionary search process within structured spaces (i.e., disciplinary archives) including certain elements (i.e., single knowledge proposals). The process of imaging novel combinations of elements from this spacecrossing established (disciplinary) boundariesis interpreted as creativity (Burt, 2004;Lee et al., 2015;Mednick, 1962;Simonton, 2003). In other words, recombination processes are seen as the main mechanisms for being creative which manifest in publishing breakthrough-class papers (Kaplan & Vakili, 2015). Koestler (1964) introduced the term "bisocitation" to describe these processes which connect well-known, but previously unconnected entities. There is an emerging line of research which has operationalized this approach by measuring the novelty of journals, patents, and research proposals (e.g., Boudreau, Guinan, Lakhani, & Riedl, 2016;Fleming, 2001;Packalen & Bhattacharya, 2017;Verhoeven, Bakker, & Veugelers, 2016). Basically, all these studies follow "the combinatorial novelty literature, which views research as a problem solving process involving various combinatorial aspects so that novelty comes from making unusual combinations of preexisting components" (Wang, Lee, & Walsh, 2018, p. 1074. These combinatorial processes are seen as creative acts. Uddin and Khan (2016) used the combination of usual and unusual keywords by authors to identify knowledge innovation and the introduction of new concepts. A similar measurement of "thematic novelty" of papers has been used by Carayol, Llopis, and Lahatte (2016) which is based on the rareness of keywords combinations. These authors found, for instance, that high thematic novelty of papers is associated with high citation impact. Schubert and Schubert (1997) and Schubert (2013)  (2) Wang et al. (2017). Both approaches are based on the premise that creative ideas in science are based on novelty which is reflected in unusual combinations of cited literature (Uzzi et al., 2013). In other words, many unusual combinations point to creative research for which the chance of becoming highly-cited or breakthrough-class status is high.

Post-publication peer review at F1000Prime
F1000Prime is a post-publication peer review system in the biomedical area.
According to Tennant et al. (2018), it is a recommendation service which provides "postpublication evaluation and recommendation of significant articles, often through a peernominated consortium". So-called Faculty Members (FMs) recommend, score, and classify papers for inclusion in the F1000Prime database. To ensure that the literature in the relevant topic areas has been adequately covered in the database, the tables of contents of major general and specialist journals in biomedicine are scanned. F1000Prime recommends on average more than 1,000 papers each month, corresponding to a fraction of all published papers in biology and the medical sciences. Thus, the majority of papers published in these areas are not covered in the F1000Prime database (see below). When FMs recommend a paper for F1000Prime, they also write a brief review explaining what makes the paper so important and rate it as "good", "very good" or "exceptional" (equivalent to scores of 1, 2 or 3 stars, respectively). F1000Prime uses the individual scores to calculate the total scores for each paper (FMs recommendations), which are used to rank the papers in each discipline.
FMs also assess the papers with the following tags, if appropriate: 1 The classifications in bold reflect aspects of novelty in research; our expectations are that they are related positively to the novelty scores. For example, we expect that a paper with many assignments of the "new finding" tag from FMs will also have high novelty scores of the indicators considered in this study. The expectations for the tags which are not printed in bold are zero or negative correlations with the novelty scores (e.g., for the confirmation of previously published hypotheses). The "controversial" tag is in italics since it is not entirely clear whether it reflects novelty or not. The tag is basically applied to studies that in the opinion of a FM 'goes against the grain' with regard to current understanding. The finding reported in the paper should be interpreted as 'circumspect', namely one need to carefully think about it before being widely accepted. F1000Prime further assigns the following publication types to the papers: "clinical trial", "systematic review/meta-analysis" and "review/commentary". However, these further assignments are not relevant to the research question of the current study and have not been considered.
F1000Prime recommendations have been used already in several studies for the correlation with metrics. One such study was interested in the relationship of quantitative (metrics-based) and qualitative (human-based) assessments of research. Waltman and Costas (2014) found "a clear correlation between F1000 recommendations and citations. However, the correlation is relatively weak" (p. 433). A similar result was published by Mohammadi and Thelwall (2013). Bornmann and Leydesdorff (2013) tested the correlation between several bibliometric indicators and F1000Prime recommendations and found that the "percentile in subject area achieves the highest correlation with F1000 ratings" (p. 286).
Bornmann (2015) used logistic regression models to examine the convergent validity of the F1000Prime peer review system. His results show that "the proportion of highly cited papers among those selected by the faculty members is significantly higher than expected. In addition, better recommendation scores are also associated with higher performing papers" (p.

2415).
Anon (2005) reports the following additional results which are based on the journal rather than the paper level (see also Jennings, 2006): "papers from high-profile journals tended to be rated more highly by the faculty; there was a tight correlation (R 2 = 0.93) between average score and the 2003 impact factor of the journal". Du, Tang, and Wu (2016) differentiated between non-primary research or evidence-based research publications on one side as well as translational research or transformative research publications on the other side.
According to their results, the former are more highly cited rather than highly recommended by FMs, and the latter are more highly recommended than highly cited.

Measuring convergent validity
Validity is a key concept in psychometrics; it is the extent to which metrics, indicators, assessments etc. measure what they claim to measure. Different types of validity exist (e.g., face validity, content validity, construct validity or predictive validity). Convergent validity belongs to the construct validity type and is the degree to which two metrics, indicators, assessments etc. of constructs (e.g., productivity, quality, novelty or creativity) which theoretically should be related are in fact (i.e., empirically) related (Panel for Review  convergent validity would not be satisfied in a bibliometric experiment that found little or no correlation between, say, peer review grades and citation measures". Harnad (2008) argues similarly: metrics "need to be jointly tested and validated against what it is that they purport to measure and predict, with each metric weighted according to its contribution to their joint predictive power. The natural criterion against which to validate metrics is expert evaluation by peers" (p. 103).
It is the purpose of the current study to analyze the convergent validity of the indicators introduced to measure the novelty of papers. We generated the scores for two indicators based on the explanations by Uzzi et al. (2013) revised by Lee et al. (2015) and Wang et al. (2017) for papers published in the biomedical area. We matched the papers with expert judgements (by FMs) from the F1000Prime database. We were especially interested in their tag assignments to papers which reflect the novelty of reported research (e.g., "article suggests new targets for drug discovery"). We tested whether papers receiving these tags have higher novelty scores than papers without these tags.

Datasets and computations
In order to test the convergent validity of the novelty scores, we calculated the scores for all papers in the F1000Prime dataset that were published between 2013 and 2015. We used three recent publication years to have a sufficiently large dataset for studying the convergent validity. Two novelty scores included in our analyses are based on the combinations of journals in the cited references of the focal papers, as well as the combinations of journals in the cited references of papers that are not among the focal papers.
We retrieved these additional data from an in-house database based on the Web of Science (WoS) database from Clarivate Analytics. This database covers papers since 1980, i.e. only cited references published since 1980 are considered in our analyses. Besides the two aforementioned novelty scores based on the focals' cited references, we calculated a simple score measuring novelty based on keyword combinations. The keywords used for this score were also retrieved from our in-house database.
Novelty score U The first novelty indicator denoted as "novelty score U" has been proposed by Uzzi et al. (2013) and revised by Lee et al. (2015). First, at the journal level, they define the commonness of each journal pair (journal i and journal j) in year t as where Nijt is the number of pairs of cited references where the two references have been published in the journals i and j and co-cited in year t, Nit is the number of pairs of references where one of the references has been published in journal i and the pair has been co-cited in year t, and Nt is the number of all pairs of references that have been co-cited in year t. Second, Lee et al. (2015) choose the 10th percentile of the series of commonness values for the pairs of journals corresponding to the pairs of cited references of a paper as an indicator of commonness at the paper level. The novelty score is then defined as the negative natural logarithm of this commonness score at the paper level. Lee et al. (2015) provide a detailed explanation of the indicator.

Novelty score W
The second indicator denoted as "novelty score W" has been introduced by Wang et al. (2017). For the calculation of this score, all (distinct) pairs of referenced journals of the focal paper are considered, if they fulfil the following conditions: (1) the journal pair has not been co-cited prior to the focal paper; (2) the journal pair is co-cited at least once in the three years following the focal paper's publication year. For each of these new journal pairs, it is measured how easy it is to make this new combination of the two journals by comparing their co-cited journals. If no or only a few other journals have been co-cited with both journals of the new journal pair, this is considered as an indicator that it is not easy to make this new combination (which would be an indicator for a rather novel focal paper).
In contrast, if both journals of the new pair of referenced journals have been co-cited with a third journal many times, this is considered as an indicator that it is easy to make this new combination (which would be an indicator for a less novel focal paper). The ease of combining the two journals of a new journal pair is defined as the cosine similarity between the vectors containing the numbers of co-citations of the two journals with other journals (cocitation profiles). The co-citation profiles are determined based on journal pairs co-cited in the three years prior to the focal paper's publication year. In all calculations, the 50% least cited journals (based on citations over the three years prior to the focal paper's publication year) are excluded. Finally, the novelty score for a focal paper is defined as where denotes the cosine similarity between the co-citation profiles of journals Ji and Jj. The sum is taken over all new pairs of referenced journal pairs as described above.

Keyword score K
The two aforementioned metrics to measure novelty are based on novel combinations of cited references. Since Tahamtan and Bornmann (2018) have questioned the use of cited references for measuring novelty and for tracing knowledge flow in general, we additionally included a relatively simple metric for measuring novelty in this study, which is not based on novel cited references, but on keyword combinations (see above). We compared this metric with the novelty scores based on cited references: do we receive other results if we use a score which is not based on cited references, but on keywords?
The keyword score for a given paper is defined as where is the number of new keywords in a paper j and is the total number of keywords in paper j. Therefore, keyword score K is the proportion of new keywords whereby newness is judged against the available keywords in one subject category from the same publication year. If a paper belongs to more than one subject category, K has been calculated multiple times, and we selected the maximum score for the further statistical analysis.
For our final dataset, the novelty scores were matched with the recommendations by FMs of the F1000Prime dataset and the number of citations from our in-house database. The final dataset comprises the novelty scores W and U, the recommendations by FMs and the number of citations received for 23,206 papers. Since the keyword score K is available for a slightly reduced set of 23,041 papers (the missing values are due to lacking keywords), we could not perform all statistical analyses with the final dataset.

Statistics
In this study, we tested the convergent validity of metrics measuring novelty by using tag assignments by FMs. The variable "tag assignments" is a count variable recording how many times a specific tag (e.g., "new finding") has been assigned to a paper by FMs. We used regression models to investigate whether (and to what extent) the assignments depend on the novelty of the papersas measured by the various novelty scores (see section 2). Regression models for count outcomes are based on the Poisson distribution (Long & Freese, 2014).
Given our dependent variables (e.g., "new finding" or "good for teaching" assignments), we cannot assume that each paper was at risk of receiving tags for the same amount of time. Since the papers in our dataset are from different years, the papers have different exposure times. The different amounts of time between publication and date of data compilation clearly affect the number of tags. To adjust for exposure time, we additionally included the publication years of the papers in the model. Since "publication year" is a factor variable (i.e., a variable with three categories), we included two variables as further independent variables in the regression models (besides the novelty and keyword scores). The third category of "publication year" is the reference category. The formula of the models changes as follows PRM: = exp ( 0 + 1 1 + 2 2 + 3 3 ) In the interpretations of the results of the regression models, we do not focus on the estimates of the regression coefficients. We further calculatedbased on these estimatespercentage change coefficients for a standard deviation increase in the independent variable (e.g., novelty score U). The percentage change coefficients simplify the interpretation and comparison of the regression models' results.
Since the novelty and keyword scores as well as citation counts and peers assessments do not follow normal distributions, we decided to calculate regression models with robust Citation counts and assessments by peers are usually used as proxies for the quality of papers (Martin & Irvine, 1983). In this study, we compare the results based on the novelty scores with the results based on citation counts and peers assessments. The variable "FMs recommendations" is the sum of the scores (1, 2 or 3 stars) which the papers receive by FMs.
We were interested in how the results of the regression models for the novelty scores differ from those for the quality assessments (using citation counts and peers assessments, respectively, as independent variables instead of novelty and keyword scores). Whereas FM recommendations are quality assessments of papers, citation counts reflect one part of quality: the impact of research within the scholarly community (Martin & Irvine, 1983). Since all

Results
In this study, we empirically investigated the convergent validity of both novelty scores (U and W) by using tag assignments by FMs for comparison. We also examined keyword score K for comparison. For investigating convergent validity, we performed regression models with novelty or keyword scores as independent, and the number of tag assignments by the FMs as dependent variables. Our expectations for the regression analyses were that the number of assignments for the tags "hypothesis" (article presents an interesting hypothesis), "new finding" (article presents original data, models or hypotheses), "novel drug target" (article suggests new targets for drug discovery), and "technical advance" (article introduces a new practical/ theoretical technique) are positively related to novelty and keyword scores. More assignments of these tags which are associated with newness in different forms should appear alongside higher novelty scores. For "confirmation", "good for teaching", "negative/ null results", and "refutation", we expect small or negative correlations with the novelty and keyword scores.
We do not have clear expectations for "controversial". We calculated several regression models for the different dependent (F1000 FMs' tag assignments) and independent (novelty and keyword scores, FMs recommendations, and citations) variables. Since the number of models is too large to present all of them in detail, we show the results of one model in detail and below focus on the most important (and comparable) results from all models. The model for the detailed analysis refers to "new finding" as dependent and "novelty score U" as independent variables. Table 2 shows the key figures for the dependent and independent variables. The mean value for the "new finding" tag is 0.85 (i.e., the papers received about one "new finding" tag on average); 40% of the papers were published in 2013 and 28% in 2015 (see the variable "publication year"). The results of the PRM are presented in Table 3. The estimated coefficient from the PRM for novelty score U is statistically significant and positive. The percentage change data in the table reveals the size of the effect: for a standard deviation increase in novelty score U (i.e., an increase with the value 1), the expected number of assignments of the "new finding" tag increase by 7.5%, holding publication years constant. The standard deviation for novelty score U is approximately one. Thus, the effect of the independent variable is as expected and substantial. It is an advantage of the percentage change data that it is comparable not only across different F1000Prime tags (the result is standardized by using the standard deviation), but also across the models, which are based on keyword scores, FMs recommendations and citations. The standard deviation has the same meaning in the independent variables.  Notes. *** p<0.001, ** p<0.01, * p<0.05. Pseudo R 2 is reported in each cell in square brackets.
Let us start the interpretation of the results in Table 4 with the findings for citations and FMs recommendations. Both variables are expected to reflect the "quality" dimension of published research and we predicted a positive relationship with all tag assignments. As the results in the table reveal, the expectations are met for all tags, but with one exception: papers with negative or null results are expected to have fewer citations than papers with positive results. However, this is something which has been reported on citations previously. For example, Jannot, Agoritsas, Gayet-Ageron, and Perneger (2013) found a citation bias favoring significant results in medical research.
The percentage change data in Table 4 is larger for FMs recommendations than for citations. Since the tags are from the same experts as the recommendations, higher percentage change values can be expected for the experts than for the citations.
The results in Table 4 for novelty score U completely match the expectations: all signs of the percentage change data are in the expected directions. Thus, the indicator "novelty score U" seems to have convergent validity with the assessments by peers. As the results further reveal, the indicator seems to be especially suitable for identifying novelty in targets for drug discovery. The percentage change value is very high (58.78%). However, larger percentage change values are also visible for other tags reflecting newness: "hypothesis" (17.71%)although statistically not significant -and "technical advance" (28.78%). The percentage change value for "new finding" is comparatively low (7.47%). This might be the case because the tag has been very frequently assigned to papers by the experts: 70.21% of the papers received the tag "new finding".
The findings for novelty score W are not as favorable as those for novelty score U.
Only the results for three tags are as expected: "confirmation", "negative/null results", and "hypothesis". Furthermore, the percentage change data for novelty score W turned out to be generally lower than those for novelty score U. It seems that novelty score W is not as convergently valid as novelty score U. We calculated additional regression models for novelty score W for comparison, since Wang et al. (2017) generated and used a categorical novelty variable in their study as follows: (1) non-novel papers (if a paper has no new journal combinations), (2) moderately novel papers (if a paper makes at least one new combination but has a novelty score lower than the top 1% in our dataset), and (3) highly novel paper (if a paper has a novelty score among the top 1% in our dataset). Wang et al. (2017) calculated this variable in a slightly other way, since they used all papers of the corresponding subject categories to identify the top 1%. We did not deem it as necessary to use all papers from corresponding subject categories for determining the top 1%, since all papers in our dataset are from the biomedical area.
As the findings in Table 4 for the categorical novelty variable W reveal, they also do not agree with the expectations for many tags. The results only change slightly in comparison to the novelty score W variable.
For comparison with the novelty scores based on cited references, we calculated keyword scores K. As the results in Table 4 reveal, keyword scores K perform similar unsatisfactorily as novelty score W: the results for only two tags are in the expected direction. Note. *** p<0.001, ** p<0.01, * p<0.05. Pseudo R 2 is reported in each cell in square brackets.
In a further step of the empirical analyses, we investigated the robustness of our results. We additionally performed probit regression models which can be used to model binary outcome variables. In these models, the inverse standard normal distribution of probability is modeled as a linear combination of the independent variables (Long & Freese, 2014). In the previous PRMs, we used the sum of the FMs scores as dependent variables. For the robustness checks, the data have been dichotomized showing whether a paper received a certain tag or not. These binary variables can be analyzed with probit regression models. In the regression models based on novelty score W, we included W as categorical variable, since some models based on W did not converge.
The results of the models for novelty scores U and W as well as keyword score K are reported in Table 5. The fully standardized coefficients can be interpreted as follows (here for the novelty score U and "new finding" variables): for every standard deviation increase in novelty score U, a paper's propensity to receive the "new finding" tag is expected to increase by 0.12% standard deviations, holding the publication year constant. As the results of all regression models in the table reveal, the conclusions which we formulated previously based on the PRMs do not change substantially with the new results. The further results mainly confirm the robustness of the PRMs.
In the final step of the empirical analyses, we compared novelty and keyword scores for two further groups of papers to test the robustness of our main findings reported previously. The results in Table 4 reveal that 70.21% of the F1000Prime papers have the "new finding" tag. We can infer from this result that "newness" is a typical sign of papers included in the F1000Prime database. We compared therefore the novelty and keywords scores of two groups: (1) included in F1000Prime and (2)  could not consider all papers published in these years, since the FMs only select papers from biomedicine and related areas. Waltman and Costas (2014) solved this problem by using the following procedure in their study on F1000Prime: they identified all subject categories of the papers included in F1000Prime and selected all papers which are in at least one subject category previously identified. We followed this procedure in our study. We performed logistic regression models (with robust standard errors) to determine the relationship between novelty and keyword scores and the inclusion of papers in F1000Prime. We performed logistic instead of probit regression models (see above), since the probit regression models did not converge in all cases. In the logistic regression models, we used the binary variably "included in F1000Prime" as dependent (see above) and the novelty and keyword scores as independent variables. We included all novelty and keyword scores in  Table 6. The results of the three regression models (for 2013, 2014, and 2015) are shown in

Discussion
Creativity is the essence in which breakthrough-class research emerges. Research without creative inspiration might become "normal science"according to the notation introduced by Kuhn (1962)but will not be exceptionally enough to be "revolutionary". The novel and unusual combination of existing knowledge from the archives has been suggested as act of creativity. Citation data are especially suited to measuring the use of novel (or typical) combinations for certain pieces of research, since it is usual in science to cite the shoulders on which research stands (Merton, 1965): according to the ethos of science, all publications which have influenced the reported research should be cited in a manuscript (Merton, 1973). Thus, the novelty of research might be reflected in the typicalas many others door novelas only a few others douse of cited references. Lee  focus on novel combinations. According to Lee et al. (2015), "it is important to find indicators that allow us to unpack the concepts of novelty and impact, in order to better understand the drivers of creativity" (p. 685).
In previous studies, the success of the scores to measure novelty has been investigated by citation analyses (see the overview of Zeng et al., 2017). The results of Uzzi et al. (2013) including "17.9 million papers across all scientific fields suggest […] that the highest-impact science draws on primarily highly conventional combinations of prior work, with an intrusion of combinations unlikely to have been joined together before. These patterns suggest that novelty and conventionality are not factors in opposition; rather, papers that mix high tail novelty with high median conventionality have nearly twice the propensity to be unusually highly cited" (p. 471). The results by Wang et al. (2017) reveal that papers with high novelty scores tend to be less cited after their appearance, but are more highly cited after several years than papers with low scores. Furthermore, novel research is "not only more likely to become a big hit itself but also more likely to stimulate follow-on research which generates major impact" (Wang et al., , p. 1420. Mairesse and Pezzoni (2018) used a novelty score based on cited references data to study the impact of "standard" and "novel" papers published by French scientists. They also foundin agreement with Wang et al. (2017) that novelty is reflected in citations only in the long run.
Although previous research has used novelty scores in various empirical analyses, only one study has been published up to nowto the best of our knowledgewhich investigated whether novelty (or creativity) is really rooted in cited references (see section 1).
However, the authors -Tahamtan and Bornmann (2018)undertook a qualitative study based on interview data. The current study tested empirically (quantitatively) the convergent validity of novelty scores: do these scores measure what they propose to measure? The investigation of validity is an important step in the development of new indicators (Thelwall, 2017); only validated indicators should be used in the practical application. Using novelty assessments by FMs of F1000Prime for comparison, we tested in this study the convergent validity of three novelty scores (U, W, and K). Peers' assessments at F1000Prime do not only refer to the quality of biomedical papers, but also their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel?
Does the paper present a good overview of research on a certain topic explaining why it can be used for teaching activities? We used these tags to investigate the convergent validity of the novelty scores.
Our study reveals different results for the novelty scores: the results for novelty score U are (mostly) in agreement with our expectations concerning the results for the different tags. We found, for instance, that for a standard deviation increase in novelty score U, the expected number of assignments of the "new finding" tag increases by 7.47% (the result is statistically significant). The results further show that this indicator seems to be especially suited to identifying papers suggesting new targets for drug discovery. The corresponding percentage change information from the regression model turned out to be very high (58.78%). The results for novelty score W, however, do not reflect convergent validity with the assessments by FMs: only the results for some tags are in agreement with the expectations. Thus, we proposebased on our resultsthe use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.
Measuring the convergent validity of newly proposed novelty scores is very much necessary, but the approach used in this study is not without its limitations.
(1) Since F1000Prime data focuses on the biomedical field, our results are only valid for this area.
Although it would be interesting to also have results for other areas, similar datasets with peer assessments for many papers are not available (to the best of our knowledge).
(2) Since our results refer to only two novelty scores based on cited references (U and W), it would be interesting to have similar investigations of the other scores which have been proposed previously. Although most of the other scores are not based on cited references data (but on keywords, MeSh terms etc.), their convergent validity can be tested also with F1000Prime data. With keyword score K, we considered one of these metrics in this study for comparison; however, it is a very basic metric, and more advanced metrics based on keywords, MeSh terms etc. might be considered in future studies. (3) The results of Bornmann, Tekles, and Leydesdorff (2019) suggest that FMs might be influenced by the citations to the paper when they make their judgements. Hence, to the extent that citations influence the assignment of F1000 tags reflecting aspects of novelty in research, there would be measurement errors in these assignments as measures purely of novelty (rather it would be measuring some combination of novelty and impact).