{'fraction_non_alphanumeric': 0.024652953299373954, 'fraction_numerical': 0.0066492981296418716, 'mean_word_length': 5.559041060953838, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Professionalism and Integrity in Research Program: Description and Preliminary Outcomes

Violations of rules and regulations in research can cause significant problems for human participants, animal subjects, data integrity, institutions, and investigators. The Professionalism and Integrity in Research Program (PI Program) provides remediation training that addresses the root causes of violations of rules and regulations in research. Through assessments, a three-day workshop, and follow-up coaching calls, the PI Program teaches evidence-based decision-making strategies designed to help researchers to compensate for bias, uncertainty, and work-related stress, and foster the skills needed to oversee research projects in today’s complex regulatory environments. Across its first three years (2013–2015), the program trained 39 researchers from 24 different institutions in the United States. Participant evaluations of the program’s faculty and workshop content were highly positive (4.7–4.8 and 4.5–4.6, respectively, on a 5-point scale). Preliminary program outcome assessment using validated measures of professional decision making and cognitive distortions in a pre- and postworkshop design indicated significant improvements. A follow-up survey of participants found statistically significant increases in a variety of target behaviors, including training research staff members to foster compliance and research quality, using standard operating procedures to support compliance and research integrity, performing self-audits of research operations, reducing job stressors, actively overseeing the work of the research team, and seeking help when experiencing uncertainty. Assessment of the PI Program was conducted with modest sample sizes, yet evaluation, outcome assessment, and self-reported survey data provided statistically significant evidence of effectiveness in achieving program goals.

When doing research, there are many ways to get into trouble. 1 Some of these involve intentionally engaging in unethical behavior-for example, fabricating data for studies 2 or conducting experiments of unknown safety on humans without informed consent. 3,4 Thankfully, such behavior appears to be rare, and researchers who engage in it are frequently fired and debarred from funding. 3 But there is another, much larger, subset of behaviors that includes less egregious actions that nevertheless can cause serious problems for investigators, institutions, human participants, and animal subjects, and can potentially compromise the integrity of experimental data. Such behaviors include failing to obtain signatures to document informed consent, deviating from anesthesia protocols in animal research, or neglecting to oversee raw data analyzed by trainees (thereby increasing the risk of data falsification). Such behaviors may reflect a lack of attention, rather than an intention to commit wrongdoing; yet, they can lead to serious disciplinary actions from the Food and Drug Administration, the Office of Laboratory Animal Welfare, or the U.S. Office of Research Integrity. Researchers may then find their research privileges suspended, while institutions struggle to identify appropriate actions that will ensure that such behaviors do not recur.
In this article we describe the first remediation program for researchers working in the United States who have violated such rules or regulations in science. We present the rationale behind the program and outcomes from our first nine workshops involving 39 researchers from 24 different institutions throughout the United States.

Program Rationale
A recently conducted needs assessment survey of research administrators showed that institutions confront research violations on a regular basis, often without an effective response. 5 While effective remediation programs exist for physicians who misprescribe, commit boundary violations, or are disruptive, [6][7][8][9][10][11] no remediation program for researchers existed prior to the creation of the program we describe here. This gap was particularly problematic given that standard training programs in responsible conduct of research (RCR) and human subjects protections often fail to achieve their goals. [12][13][14] Many of the most effective programs focus on knowledge rather than professional decision making or behavior, 15,16 when decision making and behavior clearly need to be targeted following disciplinary action.
The Restoring Professionalism and Integrity in Research Program-now called the Professionalism and Integrity in Research Program (or PI Program)was created in 2013 to meet the specific needs of investigators who violated rules or regulations in research. In a recent article, we described the kinds of violations that led to program referrals (most commonly failures to provide lab oversight, informed consent and recruitment violations, plagiarism, and

Abstract
Violations of rules and regulations in research can cause significant problems for human participants, animal subjects, data integrity, institutions, and investigators. The Professionalism and Integrity in Research Program (PI Program) provides remediation training that addresses the root causes of violations of rules and regulations in research. Through assessments, a three-day workshop, and follow-up coaching calls, the PI Program teaches evidence-based decision-making strategies designed to help researchers to compensate for bias, uncertainty, and work-related stress, and foster the skills needed to oversee research projects in today's complex regulatory environments. Across its first three years (2013-2015), the program trained 39 researchers from 24 different institutions in the United States. Participant evaluations of the program's faculty and workshop content were highly positive (4.7-4.8 and 4.5-4.6, respectively, on a 5-point scale). Preliminary program outcome assessment using validated measures of professional decision making and cognitive distortions in a pre-and postworkshop design indicated significant improvements. A follow-up survey of participants found statistically significant increases in a variety of target behaviors, including training research staff members to foster compliance and research quality, using standard operating procedures to support compliance and research integrity, performing self-audits of research operations, reducing job stressors, actively overseeing the work of the research team, and seeking help when experiencing uncertainty. Assessment of the PI Program was conducted with modest sample sizes, yet evaluation, outcome assessment, and self-reported survey data provided statistically significant evidence of effectiveness in achieving program goals.
animal care violations), and why these violations occurred (most commonly due to investigators being overextended, not prioritizing compliance, being unsure of the rules, or failing to communicate effectively). 17 Because the path to research wrongdoing is clearly multifactorial, it is naïve to think that all instances of serious noncompliance or all lapses in research integrity can be prevented through proactive, one-size-fits-all education. Researchers are often overextended as they attempt to balance multiple responsibilities such as conducting research, seeking new funding, teaching, seeing patients, and tending to administrative responsibilities. Moreover, projects may be understaffed, and staff members may not be adequately prepared for their roles. Further, principal investigators are frequently high achievers and creative learners but are not always highly disciplined and detail oriented regarding matters of paperwork and documentation. 17,18 Accordingly, we designed a program that would identify the root causes of individual researcher lapses and that would coach researchers on a range of compensatory and management strategies: • Reducing bias by managing emotions, testing assumptions, and seeking help from others; • Anticipating consequences of actions, including long-term and shortterm consequences to others and themselves; • Holding regular meetings to provide leadership and oversight of research teams; and • Developing standard operating procedures for matters of research integrity and compliance.

Program Development Process
The We four coauthors-all PI Program faculty-then developed the PI Program Manual, with the principal investigator (J.M.D.) producing an initial draft. While content was informed by the work of the development team members described above, all materials were developed de novo to ensure appropriateness for adult professional learning in a small-group, short-term setting using the principles of career coaching, 28,29 which we deemed most likely to facilitate behavior change.
The PI Program is offered three times per year. Here, we share outcomes from the first nine workshops (the first three years), offered from January 2013 through December 2015. Following each workshop, faculty met to evaluate the curriculum and revise the manual.

Program Description
The PI Program consists of preworkshop activities, a three-day on-site workshop, and postworkshop activities. Using an electronic newsletter that is delivered to more than 3,000 research administrators who were identified using publicly available information, the PI Program shares information about its services and upcoming workshops. Contact and registration information are provided in each newsletter and on the PI Program Web site. 19

Preworkshop activities
When an individual or institution contacts the PI Program Coordinator, a brief call is arranged to determine whether the candidate is a good fit for the program. We consider the workshop to be appropriate for individuals who do empirical research at graduate or postgraduate levels. We do not train undergraduates, humanities scholars, or individuals whose difficulties arise from unmet treatment needs for substance use or mental disorders. Thus far, the only individuals who were denied enrollment were reporters and RCR instructors who wanted to observe; institutions have made only appropriate referrals. Upon registration, participants are required to complete an assessment battery that examines knowledge of RCR, professional decision-making skills, levels of compliance disengagement, personal stress, and workplace stress. Baseline data for all measures have been reported in the supplemental materials of a separate paper. 17 Following the assessment battery, we conduct an enrollment interview with the prospective participant and/or an institutional official (depending on participant preferences, which may be influenced by institutional demands). During the interview we assess the nature and scope of noncompliance or other violations, learn about the kind of research being done, and determine whether the institution requires any other actions as part of a remediation plan.

Workshop activities
The heart of the The workshop approach adopted in the PI Program has proven capable of meeting the unique needs of participants despite the fact that they are referred for different reasons. The specific knowledge that participants require is often quite distinct (e.g., informed consent best practices, effective data management strategies, or proper citation practices). Faculty share this knowledge during workshop discussion as appropriate, and we recommend specific CITI training program online training modules as part of participants' professional development. However, most of the program addresses other root causes of problems-poor time management, communication, or data management practices; inadequate leadership on matters of compliance; and failure to use good professional decision-making strategies-and relies heavily on interaction, discussion, and strategizing. Throughout the three-day workshop, participants complete a series of eight worksheets that enable them to identify needs and opportunities to develop new habits, knowledge, skills, and relationships, which become the focus of their professional development plans and subsequent coaching activities. Because the program is tailored to individual needs, we have found little need to change the fundamental design of the program, though we have modified our didactic approach, moving toward greater reliance on interaction (e.g., discussion and roleplay) and worksheets.

Postworkshop activities
In the week following the workshop, participants complete two assessments and finalize their professional development plans with input from program faculty. Participants then complete two to four follow-up coaching calls over the next two to three months. During coaching calls, program faculty provide assistance to participants as they execute their professional development plans. information for participants. Program participants represent diverse disciplines and career stages with a mixture of government and industry funding.
Nearly twice as many participants were born outside of the United States than would be expected based on the percentage of faculty-level researchers in U.S. institutions. 32 This has led to increased workshop discussion of the role of culture and cultural assumptions in research. Such discussions have proven to be relevant to all participants, regardless of country of origin, because each discipline and lab has its own culture with its own attendant assumptions and biases. [33][34][35] Evaluation data Participants complete an evaluation at the end of each workshop day. Table 2 presents mean evaluation scores. On a scale from 1 to 5 with 5 indicating the strongest endorsement, items related to program faculty quality yielded mean scores of 4.7 to 4.8, and items related to the quality of course content yielded scores of 4.5 to 4.6.
After participants complete all program requirements and receive their Certificate of Completion (approximately two months after attending the workshop), we request a final overall program evaluation, which includes several open-ended items pertaining to the value of the program and areas for improvement. Because response rates are understandably lower for this follow-up evaluation (N = 16), we focus on qualitative data. The following are representative responses from several participants to the follow-up evaluation: At the time I came to the course, I was demoralized and convinced that I would be stuck in my situation indefinitely. It was such a relief to find the course was a place to dispassionately examine the factors that led up to problems, realize the roles played by myself and others, and to plan out how I could change constructively to accommodate the institution.
This is a great program for anyone interested in learning new organizational and leadership skills for the high-paced, usually very stressful work that is academic research.
The facilitators are often profound about your specific situation that often leads to positive outcomes for you.
When asked how we might improve the course, most participants made no suggestions. The most common suggestions we received, however, focused on the desire for more diverse case studies, especially cases relevant to a given individual's field of research, as several participants' comments illustrate: Add more diverse case studies.
Would be helpful to have some dedicated material for the physician-scientist. There are areas that are unique to this group of researchers and could be helpful to address some of them specifically.
Broaden the scope of the program to include different types of research concerns other than IRB and medical ethics.
Accordingly, recent iterations of the PI Program have incorporated more diversified cases. It is worth noting that case studies are the only field-specific material in the course; all other modules derive content from the participants' own experiences and work.

Pre-and postworkshop assessment data
Aside from the program evaluation data described above ( 14,37,38 The HIT-Res, on the other hand, correlated moderately with socially desirable responding; however, it contains a built-in measure of "anomalous responding" (the AR scale) that allows the cognitive distortion score to be adjusted for social desirability effects. 35 Once the HIT-Res and PDR were validated (in Years 2 and following), we administered the HIT-Res and the PDR to 24 PI Program participants. Participants completed both measures prior to the workshop and again one week following the workshop. As Table 3 indicates, HIT-Res scores indicated that the use of cognitive distortions to justify noncompliance decreased significantly, and PDR scores increased significantly following training. Interestingly, AR scores on the HIT-Res also decreased significantly, suggesting that participants were more forthright following the workshop. For faculty evaluations, the mean rating across two instructors was first computed, then these variables were averaged across Days 1-3 to get a single evaluation score. For all other scores, Day 1-3 ratings were averaged to get a single score.

Longer-term impact
To examine the longer-term impact of the PI Program on participants' attitudes and behaviors, all participants received a follow-up survey. The mean length of time from workshop completion to completion of the survey was 13 months. At follow-up, all participants were still (or once again) actively engaged in research. Table 4 reports the rest of the questions and findings. Although many of the effects are large and statistically significant, secondary to the small sample size and the use of multiple t tests, results should be considered preliminary at this time. The largest observed effects in self-reported behavior change (all P values < .001) pertained to recognizing rules and regulations in research; choosing to view compliance demands as part of the research process; providing training to research staff members to foster compliance and research quality; anticipating the consequences of decisions for oneself and others; using standard operating procedures to support compliance and research integrity; performing selfaudits of research operations; reducing job stressors; actively overseeing the work of the research team; testing assumptions or motives when making research-related decisions; and seeking help from colleagues, institutional officials, or others when experiencing uncertainty. Only three targeted behaviors did not change following the workshop: communicating with others in a constructive manner, managing emotional responses to research-related challenges, and consulting with a research mentor.

Discussion
In 2013, the PI Program won the Health Improvement Institute's Annual Award for Innovation in Human Research Protections. Not only is it the first remediation program specifically designed for U.S. researchers, but program evaluations to date also indicate that it has been successful in achieving its intended goals. Although our data derive from a small sample, many of the observed effects are large, and we have demonstrated statistically significant improvements in targeted attitudes, problem-solving skills, and selfreported behaviors. Very few behaviors have remained unchanged (such as communicating more constructively); these elements are now emphasized to a greater extent during the workshop.
Overall, as evaluations indicate, the program has been well received by participants. As we have stated elsewhere, PI Program participants were generally successful and productive researchers who did not engage in wrongdoing intentionally, even when the wrongdoing was sometimes serious or persistent. 17 The PI Program thus helps institutions to retain talented researchers while fostering compliance and integrity within their research programs, and also provides a valuable opportunity for struggling researchers to share their experiences. Being investigated for research wrongdoing is highly stressful, and many participants report that Day 2-when they share their stories with each other-has been immensely helpful to them.

Implications for institutions
We believe the PI Program offers an important service to universities. Universities often have the tools to address violations at two extremes. In the most severe cases (e.g., serial data fabrication), institutions may terminate employment; in the mildest cases (e.g., failure to update a conflict of interest form in a timely manner), they may send a written reminder of expectations and require that researchers repeat a training module. However, universities may struggle with moderately severe or repeated violations of RCR-for example, the publication of false data where intention to fabricate or falsify data appears absent; persistent failures to obtain signatures on consent forms or to report serious adverse events in a timely manner; or plagiarism that arises from improper citation practices rather than intentional theft of words or ideas. These behaviors must change to protect data, human participants, and animal subjects.
Developing a remediation program that identifies and addresses the root causes of such diverse difficulties, however, requires a significant investment of time; a curriculum informed by the best available evidence on research integrity and behavior change; and independent, highly trained faculty. Additionally, because the most effective remediation programs use a small-group, face-to-face format, [8][9][10] an institution would need regular cohorts of researchers requiring remediation to effect optimal change. Few institutions can provide such remediation in-house. In such cases, the PI Program offers a reasonable training option.
At the same time, lessons from the PI Program might be used to inform RCR instruction and mentoring at research institutions. Lack of knowledge is only one reason why researchers deviate from appropriate conduct. Other reasons include poor oversight and management of teams and a failure to prioritize matters of compliance and integrity. A new emphasis is needed on good practices such as holding regularly scheduled research team meetings, developing standard operating procedures for matters of research compliance, explicitly discussing with teams the importance of compliance and research integrity, and backing up all data to a shared server that principal investigators can access. These may seem like commonsense activities, yet not all investigators engage in them.

Limitations and next steps
The PI Program has been effective in meeting many goals such as improving attitudes toward compliance, fostering the use of good decision-making strategies, and increasing adoption of best practices for running a lab or research program. Nevertheless, it is costly for participants in terms of time and expense. Over the next two years, the PI Program plans to roll out new training options for behaviors that may be simpler to address than persistent noncompliance (such as proper citation practices and strategies for avoiding plagiarism). The program is also planning new recruitment activities: We believe that far more researchers could benefit from the three-day workshop than the relatively modest number who have enrolled to date.
Until now, the PI Program's one-year follow-up survey has been conducted anonymously to encourage participation by reducing the risk of participant identification. However, this limits our ability to examine potential links between demographic factors and behavior changes. Similarly, although our sample sizes have been large enough to detect many statistically significant changes (e.g., in attitudes, decision-making strategies, and research practices), they are too small to enable analysis at the level of subgroups-whether by training cohorts or by demographic variables. Finally, our one-year follow-up survey is limited to self-reported behavior and would be more robust if augmented with institutional feedback. Unfortunately, past efforts to obtain such data from institutions have been unsuccessful, possibly because research wrongdoing and other employee behaviors are considered to be confidential human resources matters. Alternatively, the lack of institutional feedback may derive from the fact that institutional officials rarely work closely with researchers on a day-today basis. We respect the confidentiality of these matters and the need for voluntariness in disclosing information to third parties, while acknowledging that this limits the quality of long-term data we can obtain about participants. Nevertheless, the PI Program will continue to gather assessment data to identify factors that might increase the risk of violating rules and regulations in science and to establish the short-and long-term outcomes of the program.
Acknowledgments: The authors wish to thank Tessa Gauzy, who served as PI Program Coordinator during the project period, for her support with evaluations and assessment data. The authors thank the participants who permitted use of their evaluation and assessment data in this project.  Rate the degree to which you engaged in each of the following behaviors before and after participation in the PI Program (1 = not at all, 5 = routinely) Seeking help from colleagues, institutional officials, or others when experiencing uncertainty  Using standard operating procedures to support compliance and research integrity (e.g., procedures for consenting participants, data cleaning, or data storage) Integrity provided funding to conduct outcome assessment of the PI Program (ORIIR140007). The PI Program workshop (but not research) was sponsored by the Collaborative Institutional Training Initiative (CITI) program from January 2014 to March 2017.