{'fraction_non_alphanumeric': 0.023483754313623886, 'fraction_numerical': 0.010385773034475916, 'mean_word_length': 5.146371213661313, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Retractions

Abstract To what extent does “false science” affect the rate and direction of scientific change? We examine the impact of over 1,100 scientific retractions on the citation trajectories of articles that are related to retracted papers in intellectual space but were published prior to the retraction event. Following retraction and relative to carefully selected controls, related articles experience a lasting 5% to 10% decline in the rate of citations received. This penalty is more severe when the retracted article involves fraud or misconduct rather than honest mistakes. In addition, we find that the arrival rate of new articles and funding flows into these fields decrease after a retraction.


I. Introduction
I N 2005, South Korean scientist Woo-Suk Hwang and his colleagues published an article in Science claiming they had isolated embryonic stem cells from a cloned human embryo via nuclear transfer (Hwang et al., 2005). Immediately following publication, scientists around the world took time and resources to replicate and continue this line of inquiry, thus building on the exciting (albeit controversial) field of human embryonic stem cell production using cloning. Less than a year later, the paper was formally retracted from the literature amid claims of error and later findings of fraud and embezzlement. In the aftermath, the government of South Korea curtailed investment in stem cell research for five years and, globally, scientists no longer built on the fraudulent Hwang paper. Some researchers abandoned the field altogether, while others pursued adjacent research lines that built on firmer foundations (Furman, Murray, & Stern, 2012). It took several years before researchers started to explore some of the novel hESC production methods proposed in the controversial paper, particularly parthenogenesis. In late 2007 Harvard researchers published definitive results showing that some of the (lesser) claims made by the Korean team were actually useful insights into other methods of hESC production (Kim et al., 2007). Until this new research, the field had been stifled because the retracted paper "sent a lot of scientists on a wild goose chase and down false paths," in the words of stem cell researcher Robert Lanza, quoted by the Associated Press (2005).
This dramatic incident illustrates the central questions of our paper: To what extent does false science affect the rate and direction of scientific research? To address this question, we examine the impact of retractions-publications in the academic literature that are withdrawn by authors or editors-on cumulative knowledge production along retracted research lines. We do so using a novel approach to characterize the intellectual scope of research fields and their proximity to specific (retracted) papers. Our analysis is timely because false science-a term we use to cover a broad range of phenomena, from mistakes to plagiarism to difficulties in replication to systematic fraud-has received considerable recent attention (Fang, Steen, & Casadevall, 2012;Lacetera & Zirulia, 2009;Pozzi & David, 2007). For scholars of scientific and technological change, retractions provide an unusual lens to deepen our understanding of the dynamics of cumulative knowledge production, particularly as we seek to move beyond analyses of the determinants of the rate of inventive activity toward an understanding of the factors shaping the choice of research direction (Aghion, Dewatripont, & Stein, 2008;Dasgupta & David, 1994;Furman & Stern, 2011).
The spillover effects of retractions on the evolution of research fields are particularly important given the broader welfare implications that arise from scientists shifting their position in intellectual space (Aghion et al., 2008;Acemoglu, 2012;Borjas & Doran, 2012). However, evidence is currently limited. As a starting point, systematic data on journal article retractions show a strong upward trend in frequency, but as in the case of criminal activity, the underlying magnitude of scientific mistakes and misdeeds remains poorly established (Martinson, Anderson, & de Vries, 2005). In addition, a recent analysis shows that the majority of retractions are caused by misconduct (Fang et al., 2012). More salient for the evolution of fields, Furman, Jensen, and Murray (2012) provide evidence that retraction notices are effective in alerting follow-on researchers to the shaky foundations of a particular paper. Citations to retracted papers decline by over 60% in the postretraction period relative to carefully matched controls. Their analysis, however, focuses on the fate of the retracted papers themselves, not whether and to what extent retractions influence the evolution of adjacent research areas. It also does not distinguish between different types of false science associated with retracted events, although this heterogeneity is of primary importance since the information that retraction provides regarding the veracity of associated knowledge can vary widely. Thus, the challenge for our paper is to elucidate the impact of different types of retractions on related research lines and the magnitude of spillovers to research in proximate intellectual space.
Our conceptual approach follows Acemoglu (2012), Aghion and coauthors (2009), and others in understanding research as arising through a cumulative process along and across research lines that can be traced out empirically through citations from one publication to another (Murray & Stern, 2007). This approach is grounded in the assumption that knowledge accumulates as researchers take the knowledge in a particular publication and use it as a stepping stone for their follow-on investigations (Mokyr, 2002). Although it is a commonplace insight that the process of knowledge accumulation unfolds within an intellectual space (Hull, 1988), it has proven surprisingly difficult for social scientists to gain empirical traction on this concept (see Wang, 2010, andBorjas &Doran, 2012, for rare exceptions). We conceptualize retraction events as "shocks" to the structure of the intellectual neighborhoods around the retracted papers and implement a procedure to delineate the boundaries of this space in terms of related publications in a way that is scalable and transparent, and with scant reliance on human judgment. We are then interested in studying whether researchers increase or decrease their reliance on related papers following the retraction event. We differentiate this cumulative response across three types of retractions: papers with results that have been clearly shown to be invalid and should not be used as the basis of future research (which, borrowing from Newton's aphorism regarding the process of knowledge accumulation as "standing on the shoulders of giants," we label "absent shoulders" papers); papers where retraction creates doubt about, but does not clearly nullify, the value of the content for follow-on research ("shaky shoulders"); and papers where retraction does not cast aspersions on the validity of the findings ("strong shoulders"). 1 A priori, retraction events could be thought to have two countervailing effects on the intensity of follow-on research direction. On the one hand, researchers may simply substitute away from the specific retracted paper and increase their reliance on other research in the same intellectual field, effectively increasing the prominence of the unretracted papers in that same field. On the other hand, researchers (or their funders, or both) may substitute away from the related research line and not simply from the retracted paper. Our results clearly show that the latter effect dominates.
Using the PubMed Related Citations Algorithm (PMRA) to delineate the fields surrounding over 1,100 retracted articles in the biomedical research literature, we show that 60,000 related articles experience on average a 6% decline in their citation rate following retraction, relative to the background citation rates for 110,000 control articles that appeared in the same journals and time periods (an empirical approach to controlling for citation trajectories that has been used effectively in prior work on the effect of scientific institutions and governance; Furman & Stern, 2011). Moreover, this effect is entirely driven by the fate of articles related to retractions with shaky or absent shoulders: There is no broad impact on the field when the retraction occurred because of plagiarism or publisher error. In contrast, mistakes, fake data, and difficulties arising in replication attempts have 1 Isaac Newton acknowledged the importance of cumulative research in a famous 1676 letter to rival Robert Hooke: "What Des-Cartes did was a good step. You have added much several ways, & especially in taking ye colours of thin plates unto philosophical consideration. If I have seen further it is by standing on ye sholders of Giants" (quoted in Inwood, 2003, p. 216). negative spillover effects on intellectual neighbors. Although the collateral damage (measured in lost citations) is about ten times smaller than the direct penalty suffered by the associated retracted article, we find the effect to be persistent and increasing in magnitude over time.
We exploit finer-grained levels of our data in order to paint a deeper picture of the impact of retraction on related fields and on heterogeneity in the size of the effect. The negative effect is concentrated among articles most related to the retracted paper. It is also stronger among relatively "hot fields" of research, in which a high fraction of related articles appear contemporaneously with the ultimately retracted articles, and "crowded" fields, in which the most-related articles achieve particularly high PubMed relatedness rankings. These results suggest that the degree of scientific competition within a field affects the way in which negative shocks affect knowledge accumulation.
We conclude our analysis by examining the proximate causes and potential underlying mechanisms behind the observed citation decline. We find evidence that publication rates in the fields affected by a retraction markedly decrease following retraction, relative to control fields. Similarly, we find that funding by the National Institutes of Health (NIH) in these fields declines in an even sharper fashion. We consider two mechanisms that may lie behind these effects. First, we examine evidence regarding the strength of a learning interpretation relative to one based on status concerns. On the one hand, we might simply be observing that retraction events enable scientists to discover that a particular field offers fewer prospects of important findings than was previously believed, leaving them to substitute away from that field onto lines of research that are not directly adjacent to the retracted knowledge. Alternatively, scientists in the affected fields might believe that their reputation will be besmirched if they tie their scientific agenda too tightly to a field that has been "contaminated" by a retraction. Status concerns of this kind would just as surely drive away previous (or potential) participants in the field, but such shifts would this time be construed as constituting underinvestment in the affected areas from a welfare standpoint.
We find suggestive evidence that the status interpretation accounts for at least part of the damage suffered by retraction-afflicted fields. First, we document that even in the set of articles related to retractions offering entirely absent shoulders to follow-on researchers, intent matters in modulating the observed citation responses: the penalty suffered by related articles is much more severe when the associated source article was retracted because of fraud or misconduct, relative to cases where the retraction occurred because of "honest mistakes." Second, starting from the premise that status considerations are less likely to drive the citing behavior of scientists employed in industry relative to that of academic citers, we show that the former are much less responsive to the retraction event than the latter. While a learning story suggests strengthening the retraction system in its current incarnation, the evidence for a status explanation suggests that researchers overreact to retraction notices under the current system.
In the remainder of the paper, we examine the institutional context for retractions as the central approach to governing scientific mistakes and misconduct and lay out our broad empirical strategy. We then turn to data, methods, and a detailed presentation of our results. We conclude by outlining the implications of our findings for the design of governance mechanisms that could help the "Republic of Science" better cope with the specific challenges posed by the existence of false scientific knowledge.

II. Institutional Context and Empirical Design
Knowledge accumulation-the process by which followon researchers build on ideas developed by prior generations of researchers-has been long understood to be of central importance to scientific progress and economic growth (Mokyr, 2002;Romer, 1994). In deference to Sir Isaac Newton, this cumulative process is often referred to as "standing on the shoulders of giants," but is conceptualized more prosaically as the way in which researchers in one generation learn from and build on prior research. A variety of institutions and incentives have arisen to support this cumulative process. While substantial scholarship has focused on understanding the role of openness in facilitating knowledge accumulation, 2 there is scant evidence regarding the role of institutions that support the fidelity of scientific knowledge (NIH Office of Research Integrity 2007; Pozzi & David, 2007;Lacetera & Zirulia, 2009) and even less exploration of their effectiveness (Furman, Jensen, et al., 2012;Lu et al., 2013). This is particularly unfortunate in light of recent instances of large-scale scientific fraud and mistakes that have brought to the fore concerns regarding the scientific and economic effectiveness of the institutions that govern the cumulative production of knowledge. Episodic popular and political interest is usually inspired by the discovery of highprofile cases of fraud (Babbage, 1830;Weiner, 1955;Broad & Wade, 1983;LaFollette, 1992) and the recent rise in misconduct has attracted the attention of the scientific community, including specialized blogs such as RetractionWatch. 3 In contrast to popular accounts, which often focus on the shock value and the scandalous aspects of scientific misconduct, an economic analysis of false science hinges on its impact on cumulative scientific progress. If researchers are unwittingly building on false or shaky research results, their effort is wasted and scientific progress stifled. To our knowledge, this study is the first to document systematically how false science shapes the direction of scientific research.

A. Institutional Context
Very few practices or systems exist to identify and signal research misconduct or error. In the United States, key public funders have created the Office of Research Integrity (ORI) to investigate allegations of fraud or misconduct (Pozzi & David, 2007). More broadly applicable is the system of retractions that journals themselves use to alert readers when a research publication is stricken from the scientific literature. Retractions can be made by all or some of the authors of a publication or by the journal's editor, directly or at the request of the authors' employer. These events can occur for a variety of reasons, as we describe below. Retraction events remain very rare, with the unconditional odds of retraction standing at about 1 per 10,000, regardless of the data source used to calculate these odds (see Lu et al., 2013, for tabulations stemming from Thomson-Reuters Web of Science database). Figure A of section I in the online appendix documents secular increases in the incidence of retractions in PubMed, where this incidence is measured as both a raw frequency and a proportion relative to the total size of the PubMed universe. 4 As a matter of institutional design, the system of retractions treads a treacherous middle ground in managing the integrity of scientific knowledge. At one end of the spectrum, scientific societies and journals could make significant investments in replicating and verifying all studies prior to publication, while at the other end, a knowledge registration system with no filtering mechanism could require researchers to expend considerable time and energy on replication and validation. The actual system in existence today relies heavily on peer review but provides only a limited guarantee that published knowledge is of high fidelity. As a result, reputational incentives play an essential role to ensure the integrity of the scientific enterprise (Merton, 1973).
In practice, retraction notices are idiosyncratic and vary widely in the amount of information they provide, ranging from a one-line sentence to a more elaborated statement of the rationale behind the retraction event. Understanding their impact on the scientific community is of central importance to the process of cumulative knowledge production and in deriving implications for the allocation of resources, human and financial, within and across scientific fields.

B. Empirical Design
Our core research questions require that we overcome two separate data collection challenges. First, we must develop a coding scheme to parse the underlying reasons behind each 4 While this paper is not focused on the determinants of false science but rather its impact, it is worth noting that the rise in instances of false science (or at least the increase in its documentation via retraction notices) may be linked to a range of factors, including the increasingly complex and collaborative organization of the scientific enterprise (Wuchty, Jones, & Uzzi, 2007) and the growing competition for resources in science. Lacetera and Zirulia (2009) note that competition has ambiguous effects on the incidence of scientific misconduct since scientists can also gain prominence by detecting instances of false science. of the (over 1,100) retractions that serve as "shocks" to the range of intellectual fields we examine. Our coding must also account for the degree to which the retraction leaves intact versus knocks down the foundations on which follow-on researchers may build. Second, we need a credible approach to systematically identify other journal articles that lie in close proximity in intellectual space to the retracted articles as well as a metric to measure their degree of proximity.
Categorizing retraction events. To meet the first challenge, we have developed a detailed taxonomy of retracted articles to capture the differences in the meaning of the retraction events for follow-on researchers, as described in section II of the online appendix. In a second step and taking inspiration from Newton's aphorism, we then systematically assigned the 1,104 retractions in our sample to three mutually exclusive buckets, denoted "strong shoulders," "shaky shoulders," and "absent shoulders," respectively: • Strong shoulders means that the retraction does not in any way degrade the validity of the paper's analysis or claims. This may happen in when a publisher mistakenly printed an article twice, authors published an ostensibly valid study without securing approval to publish the (unchallenged) data, or an institutional dispute over the ownership of research materials arose.
• Shaky shoulders means that the validity of claims is uncertain or that a fraction of the results is invalid. The typical use of this category concerns instances where results could not be replicated, among other reasons.
• Absent shoulders is the appropriate code for retractions associated with fraudulent results, as well as in cases where a mistake in experimental procedure irretrievably invalidates the paper's results.
In addition, we differentiate between retractions for which the authors intentionally attempted to subvert the scientific truth and those for which the article needed to be retracted because of an honest mistake with no indication of foul play. We therefore examined retractions to develop a code for different levels of intentional deception. 5 We use "no sign of intentional deception" to code cases where the authors did not intend to deceive, such as instances of miscommunication, contamination of research materials, or coding error. "Uncertain intent" applies where fraud is not firmly established, but negligence or unsubstantiated claims raise questions about the authors' motives. The "intentional deception" code is reserved for cases where falsification, misconduct, or willful acts of plagiarism and self-plagiarism appear to have occurred and were verified by author admissions or independent reviews of misconduct.
Delineating research fields. To delineate the boundaries of the research fields affected by retracted articles, we develop an approach based on topic similarity as inferred by the overlap in keywords between each retracted articles and the rest of the (unretracted) scientific literature. Specifically, we use the PubMed Related Citations Algorithm (PMRA), which relies heavily on Medical Subject Headings (MeSH). MeSH terms constitute a controlled vocabulary maintained by the National Library of Medicine that provides a finely grained partition of the intellectual space spanned by the biomedical research literature. Important for our purposes, MeSH keywords are assigned to each scientific publication by professional indexers, not by the authors themselves; the assignment is made without reference to the literature cited in the article. We then use the Related Articles function in PubMed to harvest journal articles that are proximate to the retracted articles, implicitly defining a scientific field as the set of articles whose MeSH keywords overlap with those tagging the ultimately retracted article. As a by-product, PMRA provides us with both an ordinal and a cardinal dyadic measure of intellectual proximity between each related article and its associated retraction. For the purposes of our main analysis, we consider only related articles published prior to the retraction date. We distinguish those published prior to the retracted article and those published in the window between the retracted article's publication date and the retraction event itself. We also exclude related articles with any coauthors in common with the retracted article in order to strip bare our measure of intellectual proximity from any "associational baggage" stemming from collaboration linkages. Finally, we build a set of control articles by selecting the "nearest neighbors" of the related articles-the articles appearing immediately before or immediately after in the same journal and issue, as in Stern (2011) andFurman, Jensen et al. (2012). 6 Empirical strategy. Together, these advances allow us to estimate the causal impact of retraction events on the vitality of scientific fields. We start by examining the impact of a retraction on the citations to the retracted papers themselves, in a reprise of the earlier study by Furman, Jensen et al. (2012), but using a more complete sample and carefully differentiating the effect across different types of retractions. Indeed, to the extent that retractions are highly differentiated in the information they impart to follow-on researchers regarding the strength of the shoulders on which they stand, we would anticipate that this type of variation would powerfully moderate the impact on follow-on citations. We then perform the main exercise of the paper by examining the impact of retraction events on citations to related articles and their controls in a simple difference-in-differences Retraction reasons for a set of 1,104 original articles indexed by PubMed, published between 1973 and 2008 and retracted before the end of 2009. This sample is further broken down into three subsamples. The strong shoulders subsample comprises 202 articles retracted for typically innocuous reasons, or at least reasons that do not cast doubt on the veracity of the results contained in them. The shaky shoulders subsample comprises 289 retracted articles for which either the retraction notice or information retrievable on the Web cast some doubt on the extent the results should be built on by follow-on researchers. Finally, the absent shoulders subsample contains 589 retracted articles that are the source sample for the bulk of the analysis. For these cases, we could ascertain with substantial certainty that the results are not to be relied on for future research. This can occur because of intentional misconduct on the part of the researchers involved or because of mistakes on their part. The comprehensive spreadsheet listing of these retracted articles, complete with the references used to code retraction reasons, can be downloaded at http://jkrieger.scripts.mit.edu/retractions/.
framework. Again, we separately estimate the impact of different types of retractions.
Finally, we explore the mechanisms that may be at play, focusing on the set of absent shoulder retractions. We do so by exploring citations to related articles made by authors in academia versus industry, on the assumption that status effects (in comparison to learning effects) are more likely to influence the citing behavior of academic researchers than their private sector counterparts. We also develop an analysis of the rate of production of related articles (rather than citation to these related articles) in the pre-and postretraction period. Similarly, mapping related articles to NIH funding, we explore how resources devoted to scientific fields are influenced by retractions, comparing again to control fields. Overall, this empirical design advances our ability to examine issues related to the direction of research across scientific fields and provides a nuanced understanding of the role of retractions in the process of cumulative knowledge production.

III. Data and Sample Construction
This section details the construction of our multilevel, panel data set.

A. Retracted Articles
We begin by extracting from PubMed, the public access database that indexes the life sciences literature, all original journal articles that were subsequently retracted, provided that these articles were published in 2007 or earlier and retracted in 2009 at the latest. After purging from the list a few odd observations, 7 we are left with a sample of 1,104 7 These include an article retracted and subsequently unretracted, an erratum that was retracted because of disagreement within the authorship team about whether the original article indeed contained an error, along with a few others.
articles. 8 As detailed in section II of the online appendix, we develop an exhaustive category scheme to code the reasons that explain the retraction event. These reasons are tabulated in table 1. 9 In our next step, we classify each retraction into one of three categories that denote whether the results contained in the source article can be relied on for followon research. The strong shoulders subsample comprises 202 articles retracted for reasons that do not cast any aspersion on the validity of the results contained there. In contrast, we classify 589 retractions (53.4%) as providing absent shoulders for follow-on scientists to stand on, often because of fraudulent data or other types of misconduct. Finally, the shaky shoulders category (289 events or 26.2% of the cases) groups retraction events for which the validity of the results remains shrouded in uncertainty.
Most of our analyses focus on the 589 observations belonging to the absent shoulders subsample (table 2). The papers in this subsample were published between 1973 and 2007 and took an average of three years to be retracted, though many of the more recent articles were retracted within one year-perhaps because of a higher probability of detection since the dawn of the electronic publishing era. Although this subsample is dominated by instances of fraud or other types of misconduct, 31% of the events appear to be the result of honest mistakes on the part of the investigators involved, with a further 8% for which it is unclear whether the scientists actively subverted the scientific process in the 8 In comparison, Lu et al. (2013) extract 1,465 retraction events from Thomson Reuters Web of Science over the same period. The Web of Science covers a wider cross-section of scientific fields (including the social sciences and engineering) but has shallower coverage than PubMed in the life sciences. By combining the events corresponding to life sciences journals as well as multidisciplinary journals, such as Science, PNAS, or Nature, it appears that the life sciences account for between 60% and 70% of the total number of retractions in the Lu et al. sample. 9 Despite extensive efforts, we were unable to locate a retraction notice in 24 (2.17%) cases. These 589 retractions can be grouped into 334 distinct cases. A case arises because a researcher, or set of researchers, retracts several papers for related reasons (e.g., because of repeated fraud).
course of performing the research and reporting its results. 10 Regardless of intent, however, it would be a mistake to consider each observation as completely independent from all the others in the sample. Close to 60% of the observations can be grouped into cases involving more than one retraction event, for example, because the same rogue investigator committed fraud in multiple papers or because the same contaminated research materials were used in multiple published articles. Figure B of section I in the online appendix displays the histogram of the distribution of retraction events by retraction case (N = 334). The case identifier will play an important role in the econometric analysis since all of our results will report standard errors clustered at the case level of analysis.

B. Related Articles
Traditionally it has been difficult to assign to individual scientists, or articles, a fixed address in "idea space," and this data constraint explains in large part why bibliometric analyses typically focus on the determinants of the rate of scientific progress rather than its direction. The empirical exercise in this paper hinges crucially on the ability to relax this constraint in a way that is consistent across retraction events and also requires little, if any, human judgment.
This challenge is met here by the use of the PubMed Related Citations Algorithm (PMRA), a probabilistic, topicbased model for content similarity that underlies the Related Articles search feature in PubMed. This database feature is designed to aid a typical user search through the literature by presenting a set of records topically related to any article returned by a PubMed search query. 11 To assess the degree of intellectual similarity between any two PubMed records, PMRA relies crucially on MeSH keywords. MeSH, the National Library of Medicine's (NLM) controlled vocabulary thesaurus, consists of sets of terms naming descriptors in a hierarchical structure that permits searching at various levels of specificity. There are 26,581 descriptors in the 2012 MeSH edition (new terms are added to the dictionary as scientific advances are made). Almost every publication in PubMed is tagged with a set of MeSH terms (between 1 and 103 in the current edition of PubMed, with both the mean and median approximately equal to 11). NLM's professional indexers are trained to select indexing terms from MeSH according to a specific protocol, and consider each article in the context of the entire collection (Bachrach & Charen, 1978;Névéol, Dogan, & Lu, 2010). What is key for our purposes is that the subjectivity inherent in any indexing task is confined to the MeSH term assignment process, which occurs upstream of the retraction event and does not involve the articles' authors.
Using the MeSH keywords as input, PMRA essentially defines a distance concept in idea space such that the proximity between a source article and any other PubMed-indexed publication can be assessed. The algorithm focuses on the smallest neighborhood in this space that includes 100 related records. 12 Given our set of source articles, we delineate the scientific fields to which they belong by focusing on the set of articles returned by PMRA that satisfy five additional constraints: (a) they are original articles (as opposed to, e.g., editorials, comments, or reviews); (b) they were published up to the year that precedes the calendar year of the underlying retraction event; (c) they appear in journals indexed by the Web of Science (so that follow-on citation information can be collected); (d) they do not share any author with the source; and (e) they are cited at least once by another article indexed by the Web of Science in the period between their publication year and 2011. Figure C of section I in the online appendix runs through a specific example in the sample to illustrate the use of PMRA. 13 Section III of the online appendix illustrates 12 However, the algorithm embodies a transitivity rule as well as a minimum distance cutoff rule, such that the effective number of related articles returned by PMRA varies between 4 and 2,642 in the larger sample of 1,104 retractions, with a mean of 172 records and a median of 121. 13 To facilitate the harvesting of PubMed-related records on a large scale, we have developed an open source software tool that queries PubMed and PMRA and stores the retrieved data in a MySQL database. The software is available for download at http://www.stellman-greene.com/FindRelated/. through an example how PMRA processes MeSH keyword information to delineate the boundaries of research fields.
For the set of 589 retractions with absent shoulders, the final data set comprises 32,699 related articles that can be ordered by relatedness using both an ordinal measure (the rank returned by PMRA) and a cardinal measure, which we normalize such that a score of 100% corresponds to the first "nontrivial" related record. 14 As a result of these computational and design choices, the boundaries of the fields we delineate are derived from semantic linkages to the exclusion of other considerations, such as backward and forward citation relationships, or coauthorships. Judgment and subjectivity are confined to the initial indexing task, which assigns keywords to individual articles. The individuals performing these tasks are trained in a consistent way, draw the keywords from a controlled vocabulary that evolves only slowly over time, and do not have any incentives to window-dress the articles they index with terms currently in vogue in order to curry attention from referees, editors, or members of funding panels. Of course, the cost of this approach is that it may result in boundaries between fields that might only imperfectly dovetail with the contours of the scientific communities with which the authors in our sample would self-identify. The main benefit, however, is that it makes it sensible to use citation information to evaluate whether the narrow fields around each retracted article atrophy or expand following each retraction event.

C. Identification Strategy and Nearest-Neighbor Controls
A natural starting point to identify the spillovers of retraction events on their associated fields is to examine changes in citations received by the set of related articles after the retraction, relative to before, using a simple related article fixed-effects specification. Since the retraction effect is mechanically correlated with the passage of time as well as with a paper's vintage, our specifications must include age and calendar-year effects, as is the norm in empirical studies of scientific productivity (Levin & Stephan 1991). In this framework, the control group that pins down the counterfactual age and calendar time effects for articles related to a current retraction comprises other related articles whose associated retraction occurred in earlier periods or will occur in future periods. This approach may be problematic in our setting. First, related articles observed after their associated retraction event are not appropriate controls if the event affected the trend in the citation rate. Second, the fields from which retractions are drawn might not represent a random cross-section of all scientific fields, but might be subject to idiosyncratic life cycle patterns, with their productive potential first increasing over time, eventually peaking, and thereafter slowly declining. If this is the case, fixed effects will overestimate the true effect of the retraction effect, at least if we rely on articles treated in earlier or later periods as an implicit control group.
To mitigate these threats to identification, our preferred empirical strategy relies on the selection of matched controls for each related (i.e., "treated") article. In concrete terms, we select as controls for each related article their nearest neighbors in the same journal, volume, and issue-the two articles that immediately precede and follow the treated article. When the related article is first or last in the particular issue of the journal considered, we select a single control. The final data set corresponding to the absent shoulders subsample comprises 65,461 such controls.
One potential concern with this control group is that its members may also be affected by the retraction treatment, since they are drawn from the same set of journals as the related articles. In what follows, we ignore this threat to identification for three reasons. First, the fields identified by PMRA are extremely thin slices of intellectual space, and their boundaries do not depend on journal or date of publication information (see section III of the online appendix). Second, in the extremely rare cases in which one of these nearest-neighbor controls also happens to be related to a retraction through the algorithm, we select instead the article that is twice removed in the table of contents from the focal related article. Finally, as can be observed in table 3, the rate at which the controls cite the retraction with which they are indirectly associated is almost two orders of magnitude smaller than the rate of citation that links the retractions with the treated (i.e., related) articles.
Citation data. PubMed does not contain citation data but we were able to retrieve this information from the Web of Science (up to the end of 2011) using a Perl script. We further process these data to make them amenable to statistical analysis. First, we eliminate all self-citations, where self-citation is inferred by overlap between any of the cited authors with any of the citing authors (an author name is the combination formed by the last name and the first initial for the purpose of this filter). Second, we parse the citing article data to distinguish between the institutional affiliations of citers, in particular, by flagging the citing articles for which at least one of the addresses recorded by the Web of Science is a corporate address, which we infer from the presence of abbreviations such as Inc., Corp., GmbH, and Ltd. We then aggregate this information at the cited article-year level of analysis. In other words, we can decompose the total number of citations flowing to individual articles at a given point in time into a private and a public set, where public citations should be understood as stemming from academic scientists, broadly construed (this will also include scientists employed in the public sector as well as those employed by nonprofit research institutes). Citations are a noisy and widely used measure of the impact of a paper and the attention it receives. But the use of citation data to trace out the diffusion of individual bits of scientific knowledge is subject to an important caveat. Citations can be made for strategic rather than The set of related articles is composed of journal articles linked to the 589 retracted articles of table 2 through PubMed's "related articles" algorithm (see figure 1) and downloaded using the open source FindRelated software [http://www.stellman-greene.com/FindRelated/]. We exclude from the raw data (a) articles that do not contain original research (e.g., reviews, comments, editorials, letters); (b) articles published outside a time window running from ten years before the retraction event to one year before the retraction event; (c) articles that appear in journals indexed by PubMed but not indexed by Thompson-Reuters Web of Science; (d) articles that we fail to match to Web of Science; (e) articles that we do match to Web of Science but receive no forward citations (exclusive of self-citations) from their publication year up until the end of 2011; and (f) articles for which at least one author also appears on the authorship roster of the corresponding retracted article. For each related article, we select as controls its nearest neighbors in the same journal and issue-the articles that immediately precede or immediately follow it in the issue. By convention, the controls inherit some of the properties of their treated neighbor. substantial reasons (cf. Lampe, 2012, for evidence in this spirit in the context of patent citations). For example, authors of a paper may prefer to reduce the number of citations in order to make larger claims for their own paper; they may be more likely to "get away with it" (i.e., not having editors and referees request to add citations) if the strategically uncited papers are close in intellectual space to a retracted paper. Unfortunately, we do not have the ability to parse the citation data to distinguish strategic from substantial citations, a limitation that the reader should bear in mind when interpreting our results.
Descriptive Statistics. Table 3 provides basic information about the matched sample. By construction, control and treated articles are matched on year of publication and journal, and they appear to match very closely on the length of the authorship roster. Because in many cases, retraction occurs relatively quickly after publication, only 30% of the related articles in the data are published after the publication of the source article and only 7.9% of these articles cite the soon-tobe-retracted source. Conversely, only 6.1% of the articles at risk of being cited by the source (because they were published before its publication) are in fact cited by it. Table 3 indicates that related articles and their nearest neighbors differ slightly in the total number of citations received at baseline (the calendar year preceding the retraction event), with related articles having received 1.7 citations more on average than the controls. Figure 1 compares the distributions of cumulative baseline citations for control and related articles, respectively. The controls appear to be slightly more likely to have received zero or one citation at baseline. This is not necessarily surprising, if articles related to retractions are drawn from fields that draw more attention from the scientific community in the years leading up to the retraction event. Nonetheless, these small differences in the level of citations at baseline could prove problematic for our identification strategy if they translate into preexisting trends in citations for treated articles relative to control articles in the preretraction period. We will carefully document that such pretrends are extremely small in magnitude and undetectable from a statistical standpoint, buttressing the core assumption that underlies our empirical strategy.

D. Field-Level Analyses
To examine the proximate causes of the spillover effects of retractions on their fields, we study whether patterns of entry into these fields, or the funding that accrues to active researchers in these same fields, is altered by the retraction We compute entry rates into the field surrounding a retracted article (or one of its nearest neighbors) by counting the number of PubMed-related articles in a particular year. We measure NIH funding for the same fields by summing the grant amounts awarded in a particular year that yields at least one publication over the next three years that is related to either a retracted article or one of their nearest-neighbor controls. The means and standard deviations are computed over all observations in the resulting retracted article/year panel data set (N × T = 53,451 for related article frequencies; N × T = 42,524 for funding). event. To do so, we create a second data set that collapses the related article-level data onto a retracted article-level panel data set.
As previously, we view scientific fields as isomorphic to the set of articles related (through PMRA) to a given source article. In contrast to the previous section, however, we make use of the related articles published after a retraction event (as well as before). A field is born in the year during which the oldest related article was published. Starting from the set of 589 retractions in the absent shoulders subsample, we eliminate 24 observations for which this oldest related article is "too young"-it appeared less than five years before the retraction event. This ensures that all the fields in the data set have at least a five-year time series before its associated retraction event; each field defined in this way is followed up to the end of 2011. We then select 1,076 nearest neighbor articles that appear in the same journal and issue as the retracted articles, allowing us to delineate 1,076 control fields in an analogous fashion.
It is then straightforward to compute yearly "entry rates" into treated and control fields by counting the number of related articles published in the field in each year. Capturing funding information at the field level is slightly more involved. PubMed systematically records NIH grant acknowledgments using grant numbers, but without referencing the particular grant cycle to which the publication should be credited. To address this issue, we adopt the following procedure: for each related publication, we identify the closest preceding year in a three-year window during which funding was awarded through either a new award or a competitive renewal. We then sum all the funding in the grant year that ultimately generates publications in the focal field.
The descriptive statistics for the field-level analyses are displayed in table 4. The number of observations across the publication frequency data set and the funding data set differ because the funding data are available only until 2007, whereas the publication data is available until the end of our observation period (2011) and we drop from the funding analysis the fields for which there is not a single publication acknowledging NIH funding for the entire 1970-2007 period.

IV. Results
The exposition of the econometric results proceeds in four stages. After a brief discussion of the main econometric issues, we present descriptive statistics and results pertaining to the effect of retractions on the rate of citations that accrue to the retracted articles. Second, we examine the extent of the retraction effect on the set of related articles. Third, we study whether the retraction events altered patterns of entry and funding into the scientific fields associated with the retracted articles. Fourth, we explicate the mechanisms underlying the results.

A. Econometric Considerations
Our estimating equation relates the number of citations that are received by related article j in year t to characteristics of j and of retracted article i: where AFTER denotes an indicator variable that switches to 1 the year after the retraction, RLTD denotes an indicator variable that is equal to 1 for related articles and 0 for control articles, f (AGE jt ) corresponds to a flexible function of article j's age, the δ t 's stand for a full set of calendar year indicator variables, and the γ ij 's correspond to source article or related article (or control) fixed effects, consistent with our approach to analyze changes in j's rate of citations following the retraction of source article i. The fixed-effects control for many individual characteristics that could influence citation rates, such as journal status. To flexibly account for article-level life cycle effects, f (AGE) consists of 32 age indicator variables, where age measures the number of years elapsed since the article was published. 15 15 The omitted category corresponds to articles in their year of publication (i.e., articles' birth year). It is not possible to separately identify calendar year effects from age effects in the within-article dimension of our panel data set in a completely flexible fashion, because one cannot observe two articles at the same point in time that have the same age but different vintages (Hall, Mairesse, & Turner, 2007). In our specifications, the indicator variable corresponding to articles in their thirty-first year also absorbs the subsequent age dummies. Estimates stem from conditional quasi-maximum likelihood Poisson specifications. The dependent variable is the total number of forward citations (exclusive of self-citations) received by each retracted article (or its nearest-neighbor controls) in a particular year. All models incorporate a full suite of calendar-year effects as well as 31 article age indicator variables (age 0 is the omitted category). Exponentiating the coefficients and differencing from 1 yields numbers interpretable as elasticities. For example, the estimates in column 1 imply that retracted articles suffer on average a statistically significant (1-exp[-1.171]) = 68.99% yearly decrease in the citation rate after the retraction event. QML (robust) standard errors in parentheses, clustered around retraction cases. *p < 0.10, **p < 0.05, ***p < 0.01.
Estimation. The dependent variable of interest is extremely skewed. For example, 40.33% of the article-year observations in the data correspond to years in which the related articles or controls receive 0 citations. Following a long-standing tradition in the study of scientific and technical change, we present conditional quasi-maximum likelihood estimates based on the fixed-effect Poisson model developed by Hausman, Hall, and Griliches (1984). Because the Poisson model is in the linear exponential family, the coefficient estimates remain consistent as long as the mean of the dependent variable is correctly specified (Gouriéroux, Monfort, & Trognon, 1984). 16 Inference. QML standard errors are robust to arbitrary patterns of serial correlation (Wooldridge, 1997), and hence immune to the issues highlighted by Bertrand, Duflo, and Mullainathan (2004) concerning inference in DD estimation. We cluster the standard errors around retraction cases in the results we present here.
Dependent variables. Our primary outcome variable is an article's number of citations in a given year. Secondary outcomes include the number of related articles (to either retracted papers or their nearest neighbors) published before and after the retraction event, as well as the amount of NIH funding (in millions of 2007 dollars) flowing to scientists who subsequently publish related articles (either to retracted papers or their nearest neighbors). Though the funding measure is distributed over the positive real line, the Hausman et al. estimator can still be used in this case (Santos Silva & Tenreyro, 2006). Table 5 reports the results from simple difference-indifferences analyses for the sample of 1,037 retractions and 16 In section IV of the online appendix, we find that OLS estimation yields qualitatively similar findings. 1,922 nearest neighbors in the journals in which the retracted articles appeared. 17 Column 1 reports the estimate of the retraction effect for the baseline specification. The result implies that relative to the controls, retracted papers lose 69% of their citations in the postretraction period. The magnitude of the effect is in line with the 60% decline estimated by Furman, Jensen, et al. (2012) in a smaller sample of PubMed-indexed retractions. Column 2 shows that the effect is barely affected when we drop from the sample those observations corresponding to retracted articles for which the retraction reason is missing.

B. Effect of Retraction on Retracted Papers
Column 3 includes in the specifications the main effect of the retraction treatment as well as two interactions with the shaky shoulders and absent shoulders indicator variables. In this model, the main effect implicitly captures the postretraction fate of the retracted papers that still maintain strong shoulders. While this effect is negative and statistically significant (with an implied decrease in the citation rate equal to 38%), its magnitude is markedly smaller than that of the effect corresponding to the shaky shoulders retractions (66%) and smaller still than the effect for the absent shoulders category (73%). Dropping the strong shoulders group from the sample increases the magnitude of the retraction effect in absolute value (to 72%, column 4), while focusing on the earliest retraction event in each case slightly lowers the estimated effect (66%, column 5).
In short, our results confirm the earlier findings of Furman, Jensen et al. (2012). In addition, the results in column 3 provide important empirical validation for the coding exercise detailed in the online appendix. Although the coefficients in this specification are not statistically different from each other, their magnitudes are ordered in an intuitive way, with the postretraction penalty decreasing monotonically with the strength of the shoulders provided to follow-on researchers. Estimates stem from conditional quasi-maximum likelihood Poisson specifications. The dependent variable is the total number of forward citations (exclusive of self-citations) received by each related article in a particular year. All models incorporate a full suite of calendar-year effects as well as 31 article age indicator variables (age 0 is the omitted category). Exponentiating the coefficients and differencing from 1 yields numbers interpretable as elasticities. For example, the estimates in column 1 imply that related articles suffer on average a statistically significant (1-exp[-0.059]) = 5.73% yearly decrease in the citation rate after the retraction event. QML (robust) standard errors in parentheses, clustered around retraction cases. *p < 0.10, **p < 0.05, ***p < 0.01.

C. Effect of Retraction on Related Papers
We now turn to the core of the empirical analysis, examining the effect of retraction on the citation outcomes for the related articles identified by the PubMed Related Citations Algorithm. The first set of results appears in table 6, which is structured analogous to table 5. Column 1 reports the difference-in-difference estimate for the entire sample. We find that related articles experience a precisely estimated 5.73% decline in the rate at which they are cited in the postretraction period, relative to the control articles. Column 2 shows that the estimate does not change after dropping the articles related to retractions for which we were unable to find the underlying reason. Column 3 parses the retraction effect according to our "shoulders" coding. A clear difference emerges between the fate of articles related to strong shoulders retraction and the fate of those related to either shaky shoulders or absent shoulders retractions. The articles related to strong shoulders retractions are essentially immune to the retraction event (in fact, the estimated effect is positive, but also small in magnitude and not statistically different from 0). In contrast, the implied elasticities for the articles related to shaky shoulders and absent shoulders retractions are 8.70% and 6.20%, respectively (the corresponding estimates are not statistically different from each other). In other words, we find evidence of negative spillovers of the retraction event onto the adjacent research area, but only in the cases for which the underlying cause of the retraction suggests that follow-on researchers should proceed with caution (if proceeding at all) before building on the retracted paper's results.
By eliminating from the estimation sample the observations associated with strong shoulders retractions, column 4 further documents that the negative spillovers stemming from the retraction event are of comparable magnitudes for articles related to both shaky shoulders and absent shoulders retractions. Column 5 retains only the first retraction event across retraction cases. Although the magnitude of the treatment effect shrinks somewhat, it remains negative and precisely estimated. 18 The rest of our analysis focuses on the absent shoulders subsample of 589 retractions and 98,160 related and control articles. Figure 2 provides a way of scaling the negative spillovers of retraction events onto their related fields by comparing the postretraction penalty experienced by related articles with the postretraction penalty experienced by the retracted articles themselves. In both cases, the penalty is measured by differencing the log number of cumulative citations between 2011 and the year of the retraction event (using instead a fixed two-year window starting in the year of the retraction yields very similar results). The slope of the regression line is very close to .1, indicating that related articles lose, on average, only one-tenth of the citations lost by the retraction. We note that this ratio dovetails with that of the elasticities estimated in tables 5 and 6, respectively. Moreover, with an average of sixty related papers per retracted article, the aggregate citation consequences of the retraction events for the scientific fields involved are not trivial.
To provide a better sense of the magnitude of these aggregate losses, we estimate an analog of table 6 using OLS in section IV of the online appendix. The dependent variable is the number of citations received in levels. The results are substantially unchanged compared to our benchmark Poisson specification. Furthermore, the citation decline estimated there (-0.173 citation per year) can form the basis of back-ofthe-envelope calculation. Using this estimate of the citation penalty and aggregating to the field level (taking into account both the average numbers of articles per field and the average length of the postretraction period in the sample), we conclude that retraction-afflicted fields experience, on average, a loss of 75 citations relative to control fields. Stated differently, this is as if we deleted from the average field one paper in the top 7% of the distribution for the total number of long-run citations.
Dynamics of the treatment effect. We also explore the dynamics of the effects uncovered in table 6. We do so in figure 3 by estimating a specification in which the treatment effect is interacted with a set of indicator variables corresponding to a particular year relative to the retraction year and then graphing the effects and the 95% confidence interval around them. Two features of the figure are worthy of note. First, there is no discernible evidence of an effect in the years leading up to the retraction, a finding that validates ex post our identification strategy. 19 Second, after the retraction, the treatment effect increases monotonically in absolute value with no evidence of recovery.
Exploring heterogeneity in the effect of retractions. We explore a number of factors that could modulate the magnitude of the retraction effect on intellectual neighbors' citation rates. Table 7 reports the results of seven specifications that include interaction terms between the retraction treatment effect and characteristics of either the retracted article or the retracted/related article dyad. Column 1 evaluates how the cumulative attention to the retracted article affects the reduction of citation to related articles. The rationale for this analysis is that citations are a proxy for the amount of attention that scientists in the field (and other related fields) gave The solid line corresponds to coefficient estimates stemming from conditional fixed effects quasimaximum likelihood Poisson specifications in which the citation rates for related articles and their controls are regressed onto year effects, article age indicator variables, as well as twenty interaction terms between treatment status and the number of years before/elapsed since the retraction event (the indicator variable for treatment status interacted with the year of retraction itself is omitted). The 95% confidence interval (corresponding to robust standard errors, clustered around case codes) around these estimates is plotted with dashed lines.
to the retracted paper prior to retraction and may be a predictor for the amount of collateral damage in a given field. The coefficient on the interaction term shows that highly cited retracted papers-those in the top 25th percentile of citations at the time of retraction-have larger negative spillovers on citations to their related papers (8.0% versus 3.9%). However, the additional decrease is not statistically significant at conventional levels.
Columns 2 and 3 explore how publication trends at the field level moderate the main retraction effect. In column 2, we consider how a field's "hotness"-the extent to which a field experiences elevated rates of entry in the years leading up to the retraction-affects the retraction's effect on related papers, We define a field as "hot" when it is in the top quartile of all fields in terms of the percentage of papers published in either the retraction year or within three years. 20 We find that these very active fields feel the effect of a retraction (-14.4%) more than "colder" fields (-3.4%). Column 3 focuses on how the intellectual concentration of a field intensifies the treatment effect of retraction. Our measure of "crowdedness" relies on the wedge between our ordinal measure of intellectual proximity and the cardinal measure returned by the PMRA. In some fields, the 25th most related paper published prior to retraction will be closely related to the retracted article, whereas in other fields, the 25th most related paper will be only a distant intellectual neighbor of the retraction. We label a field as crowded if this 25th highest ranking related paper lies between the 75th and 100th percentile for the relatedness score. 21 As is the case with hot fields, we see that most of the negative spillover effects occur 20 The field consists of all the related papers, as identified by the PMRA algorithm, published in or before the retraction year. 21 In the rare cases where the field has fewer than 25 papers published in or before the retraction year, the score of the least related paper is used. Estimates stem from conditional quasi-maximum likelihood Poisson specifications. The dependent variable is the total number of forward citations (exclusive of self-citations) received by each related article in a particular year. All models incorporate a full suite of calendar-year effects as well as 31 article age indicator variables (age 0 is the omitted category). Exponentiating the coefficients and differencing from 1 yields numbers interpretable as elasticities. For example, the estimates in column 1 imply that related articles suffer on average a statistically significant (1-exp[-0.040]) = 3.92% yearly decrease in the citation rate after there traction event. Highly cited source articles are retracted papers that are in the top quartile of the citation distribution (assessed at the time of retraction). We define the retracted paper's field as the set of related papers identified by PubMed's PMRA algorithm. We measure recent activity in a field by computing the fraction of papers in that field published in the three-year period leading up to the retraction event. We denote a field as hot if it belongs to the top quartile of this measure. We measure crowdedness in a field using the relatedness score of the 25th highest-ranking related paper that was published in or before the retraction year. In the rare cases where the field has fewer than 25 papers published in or before the retraction year, then the score of the highest-ranked (i.e., least-related) paper in the set is used. We denote a field as crowded if it belongs to the top quartile of this measure. We derive the uncertain intent and intentional deception codes from retraction notices and publicly available information about the retraction event (see section II of the online appendix). QML (robust) standard errors in parentheses, clustered around retraction cases. *p < 0.10, **p < 0.05, ***p < 0.01. in the crowded fields, while the more diffuse fields experience little or no decrease in citations.
Columns 4 and 5 examine whether citation linkages between the related and retracted articles moderate the magnitude of the retraction treatment effect. Recall that relatedness in the PMRA sense does not take into account citation information, only semantic proximity as inferred from MeSH keywords. Related articles can be published before the underlying source, in which case they are at risk of being cited by it, or after the source's publication (but before its retraction), in which case they are at risk of citing the soon-to-be retracted publication. In column 4, we limit the estimation sample to the articles published after the retracted piece but before the retraction. In this subsample, we find the negative retraction response to be especially pronounced (-14.8%) for the 6.1% of articles that were directly building on the retracted articles (as inferred by a citation link). Column 5, in contrast, restricts the estimation sample to the set of related articles (and their controls) that appeared before the retracted articles were published. We find that the related articles that are also cited by the retraction experience a 6.1% boost in the citation rate following the retraction event. This result is consistent with the idea that the researchers who continue to work in the field in spite of the retraction event choose to build instead on prior, unretracted research. The overall effect on the field can still be negative since only a small fraction (7.9%) of articles related to the source are also cited by the source. Column 6 uses our coding of author intent to compare how the treatment effect of retraction differs in clear cases of fraud from fraud-free retraction cases or those with uncertain intent. We see that cases of intentional deception largely drive the negative effect on the field's citations (-7.8%), while fields that experienced retractions with no sign of intentional deception (the omitted category) had no citation decline on average. Figure D of section I in the online appendix explores the extent to which the age of a related article at the time of the retraction event influences the magnitude of the treatment effect. In this figure, each circle corresponds to the coefficient estimates stemming from a specification in which the citation rates for related articles and their controls are regressed onto year effects, article age indicator variables, as well as interaction terms between the treatment effect and the vintage of each related article at the time of the retraction. Since related articles in the sample are published between one and ten years before their associated retraction event, there are ten such interaction terms. 22 The results show that only recent articles (those published one, two, or three years before the retraction) experience a citation penalty in the postretraction period, whereas older articles are relatively immune to the retraction event.
Finally, figure 4 and figure E (section I in the online appendix) investigate the extent to which relatedness (in the sense of PMRA) exacerbates the magnitude of the response. In figure 4, we use the ordinal measure of relatedness-namely, the rank received by a focal article in the list returned by PMRA for a specific source article. We create 22 interaction 22 The 95% confidence intervals (corresponding to robust standard errors, clustered around case codes) are denoted by the blue vertical bars of figure D in section I of the online appendix. The circles correspond to coefficient estimates stemming from conditional fixed effects QML Poisson specifications in which the citation rates for related articles and their controls are regressed onto year effects, article age indicator variables, as well as interaction terms between the treatment effect and indicator variables for the relatedness ranking between the related article and its associated retraction (as per PubMed's Related Articles algorithm). Each circle corresponds to five consecutive ranks (e.g., top 5, top 6-10) with all articles receiving a rank above 100 grouped together in the same bin. The 95% confidence intervals (corresponding to robust standard errors, clustered around case codes) are denoted by the blue vertical bars and their caps.
variables between the retraction effect and the relatedness rank: top 5, top 6-10, . . ., top 95-100, 100 and above. The results show that lower-ranked (i.e., more closely related) articles bear the brunt of the negative citation response in the postretraction event. Figure E is conceptually similar, except that it relies on the cardinal measure of relatedness. We create 100 variables interacting the retraction effect with each percentile of the relatedness measure and estimate the baseline specification of table 7, column 1 in which the main retraction effect has been replaced by the 100 corresponding interaction terms. Figure E graphs the estimates along with the 95% confidence interval around them. The results are a bit noisy, but here too, closely related articles (those for which the relatedness measure is above the 80th percentile) appear to experience a sharper drop in citations postretraction.

D. Effect on Entry and Funding at the Field Level
So far, the empirical exercise has examined cumulative knowledge production by building on ideas that originated before the retraction event, allowing us to hold the quality of these ideas constant over the entire observation window. In order to understand the proximate causes of the negative spillovers documented above, we must examine whether the retraction event influenced the production of new ideas in the affected fields, and assess the extent to which these same events altered the distribution of funding across scientific fields. Table 8 reports the results. Columns 1 through 3b report our estimate of the treatment effect for entry into the retraction-relevant fields, whereas columns 4a and 4b report the treatment effect for funding. A number of interesting patterns emerge. First, the response is consistently negative, indicating that both funding and publication activity decrease in the affected fields following the first relevant retraction event and relative to the patterns observed in control fields. Second, the magnitude of the treatment effect increases when we define the boundaries around fields in a stricter fashion. Third, the effect of retraction on the rate of new publications is not meaningfully different when we look at articles in highimpact versus low-impact journals (columns 3a and 3b). This result implies that the publications "lost" due to retractions do not disproportionately belong to one class of journals. Fourth, the funding response is always larger in magnitude than the publication response. Figure 5 provides event study graphs for both the publication intensity effect (panel A) and funding effect (panel B) using the same approach as that followed in figure 3. In both cases, the magnitude of the retraction effect increases over time without evidence of a reversal.
As a robustness check, we investigated whether the decline in publications might be a result of a "mentor exit" effect, in which the removal of principal investigators reduces the number of new researchers in the field. 23 In section V of the online appendix, we report that more of the lost field-level citations are associated with retractions where first authors rather than last authors are identified as culpable for the retraction. These results suggest that retraction yields the greatest negative citation outcomes not when lab directors (who are typically listed last on scientific papers) are culpable for retractions but when junior investigators (postdocs and graduate students are often listed as first authors) are at fault for retraction. Furthermore, we find that retracted first authors and middle authors are less likely to reappear in fields in which papers have been retracted than are retracted last authors (section VI of the online appendix). These analyses suggest that the strength of the treatment effect is greatest when the author culpable for retraction is the first author (rather than the last author) and that the publication decline is not driven by the exit of principal investigators or lab directors but may be driven by the exit of first authors. 24 To summarize, these results help explain why we observe downward movement in the citations received by related articles highlighted earlier: there are fewer papers being published in these fields and also less funding available to write such papers. While these effects constitute the proximate causes of the negative spillovers that are the central finding of the paper, they beg the question of what the underlying mechanisms are. What explains the flight of resources away from these fields?

E. Underlying Mechanisms of the Retraction Effect
A number of mechanisms may underlie our findings regarding negative citation, entry, and funding. We investigate evidence regarding two possibilities. First, a relative decrease in attention subsequent to retraction may reflect scientists' learning about the limited potential for follow-on 23 We are grateful to an anonymous referee for encouraging us to pursue this explanation. 24 These results align well with the evidence presented in Jin et al. (2013). Estimates stem from conditional quasi-maximum likelihood Poisson specifications. The dependent variable is the total number of related articles published in a particular source/year (columns 1a, 2a, 2b, 3a, and 3b), as well as the number or total dollar amount of NIH funding awarded in a particular year that yields at least one publication over the next three years that is related to either a retracted article or one of their nearest-neighbor controls (columns 4a and 4b). The high journal impact factor (JIF) category includes journals in the top quartile of JIF (indexed by ISI), while the low-JIF category includes journals from the lower three quartiles. All models incorporate a full suite of calendar-year effects. QML (robust) standard errors in parentheses, clustered around retraction cases. *p < 0.10, **p < 0.05, ***p < 0.01.

Figure 5.-Field-Level Dynamics
The solid lines correspond to coefficient estimates stemming from conditional fixed-effects quasi-maximum likelihood Poisson specifications in which the number of related publications (panel A) and NIH funding in millions of 2007 dollars (panel B) associated with a particular source article are regressed onto year effects as well as 20 interaction terms between treatment status and the number of years before/elapsed since the retraction event (the indicator variable for treatment status interacted with the year of retraction itself is omitted). The 95% confidence interval (corresponding to robust standard errors, clustered around retraction cases) around these estimates is plotted with dashed lines. Figure 5A corresponds to a dynamic version of the specification in column 1 of table 8, while figure 5B corresponds to a dynamic version of the specification in column 4b in the same table.
research in retraction-afflicted fields. The case of Jan-Hendrik Schön is consistent with this explanation. Schön's research at Bell Labs initially produced spectacular results using organic materials to achieve a field transistor effect; his results were eventually demonstrated to have been the result of fraudulent behavior, and subsequent efforts building on his work suggest the impossibility of achieving field transistor effects using the materials he employed (Reich, 2009). Second, the field-level declines in citation, entry, and funding we observe could also arise from a fear of reputational association with the "contaminated" fields or authors. The case of Woo-Suk Hwang that we invoke at the beginning of the paper is consistent with this type of explanation: follow-on researchers eschewed all implications of Hwang's work, although when the field revisited his work a few years after the retractions, some would prove promising. Although we may not be able to rule out either explanation entirely, exploring the relative importance of these mechanisms matters because their welfare implications differ. For example, it may be ideal from a social planner's perspective if scientists simply redirect their efforts away from retraction-rich fields after a retraction event demonstrates their unpromising nature. If, however, status considerations inhibit entry into potentially productive fields of research, the risk exists that the negative spillovers we documented earlier reflect underinvestment from a social welfare standpoint. We exploit the finely grained level of detail in the data to provide evidence regarding the relative merits of these explanations. We begin by examining whether the retracted authors' intent influences the citation response to related articles written before the retraction event. Limiting the estimation sample to the set of retractions offering absent shoulders to follow-on researchers, we include in the benchmark specification two additional variables corresponding to the interaction of the retraction effect with, respectively, the uncertain intent and intentional deception indicators mentioned earlier (table 7, column 6). The evidence clearly shows that the postretraction penalty is larger when there is clear evidence of malicious intent. It is possible that retractions associated with misconduct are, even in this restricted sample, more consequential for the field than are retractions associated with honest mistakes. 25 The finding that biomedical research fields apply a greater citation penalty when errors are intentional is consistent with the idea that a stigma attaches to research lines in which fraud has been perpetuated. At least two other explanations are possible, however. First, although the lack of a pretrend in figure 3 suggests that retraction is not the result of the "fishing out" of a research area, intentional fraud may signal its future fruitlessness (i.e., as progress may be possible only through active deception), whereas an honest error may provide no such signal about future research prospects. It is also possible that the differential response to fraud-and mistakesafflicted fields may arise from the rational expectation that fraud could be widespread, while mistakes are more likely to be idiosyncratic. 26 In this view, even if there are no costs of associating with a field going forward (e.g., because journals and referees respond to the retraction by being more vigilant) and no learning about the future prospects of that field, the possibility of undiscovered false science in past work may reduce future work in that area. 27 The evidence in section IV.C that retractions in hot fields have a disproportionate effect on future citations does not lend support to these explanatory mechanisms.
To further investigate the possibility that a reputational mechanism may be at work, we examine heterogeneous responses between academic-and firm-based citers. We start from the premise that scientists employed by profitseeking firms would persist in investigating topics that university-based scientists (and NIH study sections) frown on (postretraction) as long as the possibility of developing a commercial product remains. 28 We parse the forward cita- 25 We also find this effect in models (unreported but available on request) in which we control for retraction size by including in the specification interaction terms between the retraction effect and the quartiles of postretraction penalty at the retracted article level. 26 Another possibility, of course, is that researchers underreact to the discovery of honest mistakes. Though mistakes are likely more idiosyncratic than instances of fraud, one can think of instances where this is not the case, such as with the contamination of reagents or cell lines, as in the famous example of HeLa cells (Lucey, Nelson-Rees, & Hutchins, 2009). 27 We thank one of our anonymous referees for highlighting this alternative interpretation. The reviewer also noted the possibility that the "wild goose chase" effects of false science might contribute to the decreased citations and entry in affected fields, as scientists spend time trying to investigate and verify results related to the retracted paper. 28 We ground our assumptions regarding the potentially differential responses of academic-and industry-based scientists by appealing to prior work on differences in incentives and status concerns among academic and industrial scientists, the former of whom have principally, though not exclusively priority-based, incentives, and the latter of whom face stronger, though not exclusive, financial and organizational incentives that are not tion data to separate the citations that stem from private firms (mostly pharmaceutical and biotechnology firms, identified by suffixes such as Inc., Corp., LLC, Ltd., and GmbH) from those that originate in academia (broadly defined to include nonprofit research institutes and public research institutions as well as universities). Even though we classify as "private" any citing article with a mix of private and academic addresses, almost 90% of the citations in our sample are academic according to this definition. In table 9, columns 1a and 1b, we find that academic and private citers do not differ at all in the extent to which they penalize the retracted articles. Conversely, columns 2a and 2b indicate that private citers hardly penalize related articles, whereas academic citers do to the extent previously documented. 29 The difference between the coefficients is statistically significant ( p < 0.01). These findings are consistent with the view that the retraction-induced spillovers we have documented stem at least in part from academic scientists' concern that their peers will hold them in lower esteem if they remain within an intellectual field whose reputation has been tarnished by retractions, even though these researchers were neither coauthors on the retracted article itself nor building directly on it.
It is possible, however, that these differences arise because industry scientists find it easier to substitute citations within a field because their work is more applied in nature. 30 To investigate this possibility, we have matched the PubMed database with the U.S. patent data to identify the citations received from patents by published scientific articles. 31 Our working assumption is that papers that are cited by patents are more likely to be later stage, whereas those that receive no citations from patents are more likely to correspond to upstream research. In the absent shoulders subsample, 10.7% of retracted articles were ever cited in a patent (table 2), while 8.7% of their related articles and 8.3% of the nearest-neighbor controls were ever cited in a patent (table 3).
We use these data to examine whether the citation patterns of academic and industrial papers also depend on the upstream or downstream character of the research itself (i.e., whether it is specifically cited in a patent). We observe no difference in the case of the retracted articles themselves (table 9, columns 1c and 1d). However, the distinction between upstream and downstream research matters for the rate of citations to related papers. In particular, academic citations to retraction-related articles experience a negligible decline if the related paper was ever cited in a patent (column 2c, sum of the coefficients), but the effect remains strongly negative and significant for related papers not cited in a patent. In other words, the differential response noted directly tied to standing in the research community (Dasgupta & David, 1994;Stern, 2004). 29 The estimation sample is limited to the set of related articles and their controls that receive at least one citation of each type over the observation period. 30 We thank an anonymous referee for this suggestion. 31 See appendix D in Azoulay, Zivin, and Sampat (2012) for more details on the patent-to-publication matching process that provides a foundation for the analyses presented in table 9. above is limited to more upstream research, which makes up over 90% of the retraction-related papers in our sample.
In summary, the available data do not enable us to directly evaluate the relative importance of the learning and status interpretations of the effects we uncover. Viewed in their entirety, however, our analyses suggest that status concerns play an important role in explaining the intellectual atrophy of retraction-afflicted fields. And if participation in these fields is curtailed as a result of these concerns, the conjecture that depressed participation corresponds to underinvestment from a social welfare standpoint is, at the very least, plausible.

V. Conclusion
This paper constitutes the first investigation of the effect of false science on the direction of scientific progress. Our findings show that scientific misconduct and mistakes, as signaled to the scientific community through retractions, cause a relative decline in the vitality of neighboring intellectual fields. These spillovers in intellectual space are significant in magnitude and persistent over time.
Of course, an important limitation of our analytical approach is that though we can document that retraction events cause a decrease in the rate of citations to related articles, we cannot pinpoint exactly where the missing citations go or, more precisely, in which direction scientists choose to redirect their inquiries after the event. Nonetheless, the empirical evaluation has a number of interesting implications. Through the coding scheme we have developed to understand the particular circumstance of each retraction event, we highlight the limitations of the institutional practices that are supposed to ensure the fidelity of scientific knowledge. In particular, the analysis brings systematic evidence to bear on the heightened attention devoted to the topic of scientific misconduct in science policy circles. Some analysts suggest that the scientific reward system has been corrupted and is in need of wholesale, radical reform (Fang et al., 2012). This view points to the increase in detected frauds and errors as a strong indication that much invalid science goes undetected. Acknowledging this possibility, others retort that a system of retractions is precisely what the "Republic of Science" requires: a mechanism that swiftly identifies false science and effectively communicates its implications for follow-on research (Furman, Jensen, et al. 2012). The validity of the more optimistic view hinges crucially on what is signaled by a retraction notice and on how scientists in the affected fields process this information and act on it. Our results suggest that retractions do have the desired effect on the particular paper in question, but also lead to spillover effects onto the surrounding intellectual fields, which become less vibrant.
If these negative spillovers simply reflected the diminished scientific potential of the affected fields, then the collateral damage induced by retractions would not be a cause for concern and would reinforce the belief that the retraction process is a relatively effective way to police the scientific commons (Furman, Jensen, et al. 2012). However, our evidence indicates that broad perceptions of legitimacy are an important driver of the direction of scientific inquiry. Unfortunately, retraction notices often obfuscate the underlying reason for retraction, which diminishes the information content of the signal they provide to follow-on researchers. As a result, there could be high returns to developing a standardized coding approach for retractions that journals and scientific societies could draw on to help the scientific community update their beliefs regarding the nature and scope of false science. While journal editors may understandably balk at the suggestion that it is incumbent on them to make clear determinations regarding the underlying causes of retractions, a clearly articulated schema would increase the incentives of authors to report problems emerging after the publication of an article and provide a more nuanced context within which universities themselves (as well as funding bodies) might investigate and adjudicate instances of false science. 32 A second issue raised by our paper relates to our understanding of what constitutes an intellectual field. As we noted in section I, economists have devoted considerably more time and attention to the study of the rate of inventive activity than to its direction. This gap has arisen in part because of the empirical challenges associated with delineating the boundaries among intellectual fields. Our approach relaxes the data constraint through the systematic use of keyword information. The same approach could also prove itself useful to explore more generally the ways in which researchers, through their publications, choose positions in intellectual space and change these positions over time. At the same time, economists' conceptual grasp of intellectual landscapes remains in its infancy, with a near exclusive focus on vertical research lines (Aghion et al., 2008). We hope that our empirical results will prove useful to economists seeking to understand movement across research lines and the consequences of these movements for cumulative knowledge production and, ultimately, economic growth.