{'fraction_non_alphanumeric': 0.029987437695019655, 'fraction_numerical': 0.013372776269400656, 'mean_word_length': 5.3050587634133874, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Validating curricular competencies in innovation and entrepreneurship for biomedical research trainees: A modified Delphi approach

Introduction: Biomedical researchers need skills in innovation and entrepreneurship (I&E) to efficiently translate scientific discoveries into products and services to be used to improve health. Methods: In 2016, the European Union identified and published 15 entrepreneurial competencies (EntreComp) for the general population. To validate the appropriateness of these competencies for I&E training for biomedical researchers and to identify program content, we conducted six modified Delphi panels of 45 experts (6–9 per panel). Participating experts had diverse experience, representing such fields as entrepreneurship, academic research, venture capital, and industry. Results: The experts agreed that all 15 EntreComp competencies were important for biomedical research trainees and no additional competencies were identified. In a two-round Delphi process, the experts identified 120 topics to be included in a training curriculum. They rated the importance of each topic using a 5-point scale from not at all important (1) to extremely important (5) for two student groups: entrepreneurs (those interested in starting their own ventures) and intrapreneurs (those wanting to be innovative and strategic within academia or industry). Consensus (mean importance score >4) was reached that 85 (71%) topics were of high importance for the curriculum. Four topics were identified by multiple panels for both student groups: resiliency, goal setting, team management, and communication skills. Conclusions: I&E training for biomedical trainees should address all 15 EntreComp competencies, including “soft skills,” and be flexible to accommodate the needs of trainees on different career trajectories.


Introduction
Today's biomedical research trainees often need skills in innovation and entrepreneurship (I&E). Those who choose an academic career need I&E skills to equip them for success in an increasingly impact-orientated funding environment. Those who choose a career in industry or government need I&E skills to help them to identify, assess, and capitalize on opportunities to improve human health. To keep pace with the changing training needs to support multiple career options for biomedical researchers, the National Institutes of Health (NIH) has encouraged breadth and flexibility in research training programs [1]. We received a grant from the NIH to develop a competency-based program to introduce biomedical research trainees to innovation and entrepreneurial thinking. To achieve this goal, we first sought to identify the core I&E competencies a successful biomedical researcher should possess and to identify topics to be included in a curriculum to introduce these skills to biomedical research trainees.
In a preliminary review of the literature, we found that few publications addressed competencies for I&E; and we found none that specifically addressed the needs of biomedical researchers. Our study team considered the literature and their combined experience in I&E, to develop an initial working draft of I&E competencies shown in Table 1. After the grant was awarded, we updated our literature review and discovered the work of another group, "EntreComp: The Entrepreneurship Competence Framework" [27]. The EntreComp Framework is unique in the literature in that the aim was to identify competencies that would generate an "entrepreneurial mind-set" for all citizens rather than training entrepreneurs. Fifteen high-level competencies were identified using a rigorous mixed-methods approach, including a literature review and in-depth case study analyses leading to the development of draft competencies and a conceptual model, with validation via expert and stakeholder consultation [27]. Entrepreneurship is defined broadly as "the capacity to act upon opportunities and ideas to create value for others. The value that is created can be social, cultural, or economic" [28]. The authors of the EntreComp Framework suggest that it can be used as a "starting point for the interpretation of the entrepreneurship competence" and that it should be adapted and tailored to address the needs of specific groups. As of March 2018, the Framework had been used in at least 74 training programs [28]. The EntreComp competencies are provided in Table 2.
We decided to use the EntreComp Framework as a starting point to develop our I&E training program for biomedical research trainees for the following reasons: (1) the aim of the EntreComp Framework to generate an entrepreneurial mind-set was in line with our intent; and (2) all the competencies in our working draft were contained within the Framework. To validate the appropriateness of the EntreComp competencies for I&E training for biomedical research trainees in the USA and to identify course content for our program, we conducted six modified Delphi panels. Delphi panels are typically used to establish group consensus about priorities when many options exist, and modified Delphi processes have previously been used to develop curricula [29,[30][31][32][33][34][35], which was our purpose. We needed expert opinions upon which to build our curriculum.

Compliance
The project was presented to the Washington University in St. Louis (WU) Institutional Review Board who determined that it did not constitute research because it aimed at producing consensus among experts rather than generalizable knowledge from subjects. However, all participants freely agreed to participate in the process and provided permission to publish their names and biographical details, which they reviewed and approved (see Appendix A).

Participants
Panelists were selected using purposeful, non-probability sampling with the goal of recruiting a heterogeneous group of experts from the USA. Forty-five experts were identified (from academia, 25; venture capital, 11; industry, 9). Many had experience in biomedical entrepreneurship as shown in their biosketches (see Appendix A). All experts agreed to participate in a modified Delphi process over 2 months, with a total time commitment of less than 2 hours. A $100 Amazon gift card was offered as an honorarium.

Procedures
Recognizing the breadth of the I&E competencies, and to avoid overburdening panel members, we grouped the EntreComp Framework high-level competencies into five domain areas (Table 2). Six Delphi panels were planned, one for each domain area (Panels 1-5) and one to validate the 15 EntreComp competencies for use in our program (Panel 6).
The project team allocated the 45 experts to the six panels based on their area of expertise and entrepreneurial experience to ensure that each panel had a mix of both content experts and experienced entrepreneurs. Panel size ranged from 6 to 9 and is provided in Table 2. Between April 9, 2018 and July 18, 2018, each panel worked independently and simultaneously with panelists blinded to the identity of other participants. For each round of the Delphi process, individual panelists accessed the surveys hosted in Qualtrics through unique links. Each round allowed a 2-week window to submit responses with reminder e-mails and a 1-month gap between rounds.
For all panels, at the beginning of round 1, all experts were asked to review background information to understand the context and purpose of the Delphi panel project. First, they were introduced to the program goal: to better equip biomedical research trainees for their future careers by teaching them I&E skills. Second, they were introduced to the EntreComp Framework and the 15 high-level competencies that we had grouped into five overarching categories: management, vision and imagination, social skills, psychological skills, and ethical and decision-making skills. Finally, the experts were asked to make the following assumptions: "(1) All course participants are enrolled in a training program to pursue a career in biomedical research. (2) They may work in the university, government, industry, and other settings.
(3) They may or may not become entrepreneurs. (4) The course they were considering would be introductory, and approximately 20 hours long."

The EntreComp Framework Panel
The goal of this panel was to validate the EntreComp Framework as a whole for biomedical researchers. The seven panelists were provided a list of all 15 EntreComp Framework high-level competencies and a brief description of each (Table 2). They were asked whether each competency was relevant for biomedical research trainees (yes/no); in the case of a "no" vote, they were asked to provide a rationale. Panelists were also asked if any additional competencies should be added to the list. A priori, we defined agreement among the experts that the EntreComp competency was relevant to biomedical research trainees as a yes vote from 5/7 panel members. This approach is consistent with other Delphi methodology [36]. We chose to use this variation on the Rand criteria due to our sample size and dichotomous choice. If new competencies were identified by individuals, they would be presented to the panel in a second round to evaluate for consensus.

Panels 1-5
The goal for Panels 1-5 was to identify topics to be included within the training program for the EntreComp competencies assigned to that panel. In round 1, participants were presented with their panel's EntreComp high-level competencies and brief definitions ( Table 2). Then they were asked "What content do you think should be taught? Content can be described as topics, knowledge, Cross-cutting personal competencies • Creativity and cognitive adaptability [2][3][4][5] • Ethical problem-solving [6][7][8] • Self-leadership and selfmanagement [9][10][11][12][13][14][15] • Social intelligence [16][17][18] Team and project management competencies • Communication [3,16] • Leading creative and diverse teams [2,[19][20][21] • Leading for innovation [2,3,[22][23][24][25] • Project management skills [3] Business competencies • Business acumen and technical knowledge • Negotiation • Managing ethical and legal issues in business [26] 166 Jane Garbutt et al. skills, or attitudes. Please list as many content areas that you think should be included." For each panel, responses from round 1 were analyzed to eliminate redundancy and the responses were summarized. In round 2, panelists were asked to review the summarized list and rate the importance of each topic for an introductory course in entrepreneurial thinking for biomedical research trainees using a 5-point scale from not at all important (1) to extremely important (5). For each topic they provided their rating for two student groups: entrepreneurs (those with an interest in starting their own ventures) and intrapreneurs (those who want to be innovative and strategic within pre-established companies or an academic career) [37]. For each panel, the mean importance score of all expert panelists was calculated for each topic. We defined topics with a mean importance score > 4.0 (very or extremely important) as having a consensus that the topic was "highly important" to teach. This approach is consistent with other Delphi methodology that defines a consensus using a mean score [36]. We adopted a 5-point rating system (rather than 9 point) to make it easier to label each option. Participants were also asked to use open-ended textboxes to provide other content areas they thought were missing from the list.
After analyzing results from round two, it was decided that a third round for Panels 1-5 was not required as the results were intended to guide curriculum development by providing an expert generated list of topics, rather than build consensus.

Validity of EntreComp Competencies for Biomedical Research Trainees
The seven experts on the EntreComp Framework panel agreed that the 15 EntreComp competencies were all important for biomedical trainees. After analyzing results from round 1, we determined that a second round for this validation panel was not required as no new high-level competencies were identified.

Panels 1-5
Altogether 207 topics were generated across the five panels in round 1, resulting in 120 summarized topics (17-31 per panel) for analysis in round 2. Overall, 36/38 (95%) experts submitted responses for both rounds of the Delphi process.

Course Content
The five panels achieved consensus that 85 (71%) of the 120 topics ranked in round 2 were of high importance to include in the curriculum. These included 42 (49%) topics considered to be of high importance for all biomedical research trainees regardless of future career direction, 35 (41%) additional topics for entrepreneurs only, and 8 (9%) topics for intrapreneurs only (Table 3). In Table 4, topics that reached consensus for entrepreneurs and intrapreneurs are listed by panel grouped by importance for trainee's career direction (both entrepreneur and intrapreneurs, entrepreneur only, and intrapreneur only). The complete data for all panels are provided in Appendix B.
Although each panel was assigned different EntreComp competencies in rounds 1 and 2, several topics were identified as high importance to include in the curriculum for both career groups by multiple panels. For entrepreneurs, these included resiliency/self-management (four panels), goal setting (three panels), team management (three panels), and communication skills (three panels). These same topics were identified by multiple panels for intrapreneurs, resiliency/self-management (two panels), goal setting (three panels), team management (three panels), and communication skills (four panels).

Discussion
The six modified Delphi panels reached our two aimsto identify the I&E competencies a biomedical researcher should possess and to identify topics to be included in the core curriculum to introduce I&E skills to biomedical research trainees. Our findings suggest our introductory training program should address all 15 EntreComp competencies, yet be flexible to accommodate variation in needs of trainees on different career trajectories. One approach to meet the need for breadth and depth of course offerings is to provide both core and elective courses.
Our expert panelists provided guidance about topics to be included in core and elective courses. Half of the 85 topics identified as high importance were ranked as high importance for all trainees, regardless of their career trajectory. Reviewing this topic list suggests that core courses should be designed to teach trainees the following I&E skills: how to identify opportunities for innovation; how to determine their potential for success; how to communicate about your innovation idea to various audiences; how to build and manage teams; how to be aware of the ethical consequences of your decisions and actions; and self-management and resiliency. Topics identified as high importance for only one career group could be considered as electives, allowing trainees to tailor their program to meet their needs. Skills of particular interest to entrepreneurs might be learned through elective courses focused on commercialization such as identifying your funding needs and opportunities, and building a business plan. Elective 168 Jane Garbutt et al. courses of interest to intrapreneurs might include managing up and social entrepreneurship including cultural competencies. Elective courses also provide opportunity for more in-depth coverage of topics that might be relevant only for some trainees.
With the goal of improving human health through translational innovation, I&E skills are important for the translational research workforce of the future [38]. An important initiative in this regard is the Clinical and Translational Science Award (CTSA) program with the goal of efficiently translating research knowledge into improved health [39]. Several core I&E skills identified by our experts have previously been identified as core competencies for clinical and translational science, including the "soft skills" needed to function as a professional and to work in multi-and interdisciplinary teams [38,[40][41][42]. Courses in communication and team science are provided at many CTSA sites across the country as part of degree programs in clinical and translational research [38,40,41]. Training in the more traditional entrepreneurial skills such as design thinking and commercialization is offered through engineering and business schools [43]. For some there is a tension between the goals of medicine and science and entrepreneurship. Medicine and science are founded on goals such as improving health, creating generalizable knowledge, openness, and transparency [44-46]. These goals may conflict with goals of entrepreneurship where profits and financial motives may lead to secrecy, proprietary claims, and competition rather than collaboration [28,29,47]. Evidence has shown that financial incentives and motives can affect decision making, change behaviors, and potentially lead to unethical actions, for instance through conflicts of interests [29,46,48,49]. At the same time, collaborations between industry and academia are now the norm, alongside an emphasis on translational science, suggesting we should not abandon these relationships but rather we need to ensure that individuals are aware of these tensions [47,49,50]. This is our rationale for including ethics experts in our panels and for requiring an ethics course for all students, regardless of track. Our web-based introductory program will increase opportunity for pre-and post-doctoral students to access training in I&E skills that are relevant for translational researchers. The final program will include a bootcamp and a team-based capstone project to provide learners opportunities to integrate their new knowledge and skills.
It is notable that the topics of self-management and resilience were rated as highly important for both entrepreneurs and intrapreneurs across multiple panels. Our experts recognized resilience as an important skill to deal with the ups and downs of innovation. Indeed, entrepreneurial resilience has been shown to have a significant positive relationship with success for individuals and businesses [51, 52]. Overall, research suggests that resilience is a modifiable construct and not an inherent, immovable trait [53]. Encouraged by a recent metanalysis that found "a modest but consistent benefit of resiliency training programs in improving a number of mental health outcomes within three months of follow-up" [53], we plan to have a core course to improve resilience skills. We will carefully evaluate the courses as we note that the authors commented that the 13 randomized controlled trials included in the meta-analysis were small and generally of poor methodological quality [53].

Limitations
Our panelists each had I&E experience relevant to biomedical research, but it cannot be assumed that a different panel of experts would reach the same conclusions regarding competencies and program content for an I&E curriculum for biomedical research trainees. We based the definition of highly important topics for learner groups on common sense and common practice [36], and changing this definition might change the conclusions drawn from the study. We provide the complete study data in Appendix B to allow the reader to review all topics suggested by the expert panelists, not just the ones that we identified as being highly important. We focused on identifying topics to include in our curriculum and did not ascertain the level of mastery expected of learners. Our program will be an introductory course for pre-and postgraduate students, and the learning objectives will determine the level of mastery expected.

Conclusion
The six modified Delphi panels identified topics to be included in a basic training program to encourage an entrepreneurial mind-set in biomedical research trainees. We will use these findings to inform the development of our introductory program in I&E training targeting this group, specifically to inform learning objectives, course content, and designation as a core or elective course. While these training recommendations are based on the expert consensus, we will need to assess learner outcomes and reactions to evaluate the success of our program. Additional considerations such as how the course should be implemented (in-person, web-based, team-based), course duration, and the roll of additional components such as mentoring and a capstone project to synthesize learning need further exploration. 22 Warren Seering, PhD Seering@MIT.edu Dr. Seering is the Weber-Shaughness Professor of Mechanical Engineering. He has helped to establish the MIT Machine Dynamics Laboratory, was a member of the MIT Artificial Intelligence Laboratory, and cofounded and for a time directed the Center for Innovation in Product Development at MIT. His research has spanned machine dynamics, engineering system design, and product development. Dr. Seering has won several awards for his teaching excellence, and he has mentored over 150 advanced degree-seeking students and taught courses on design, product development, applied mechanics, system dynamics, instrumentation, and computer software. He earned his PhD in Mechanical Engineering from Stanford University. worked as the Director of QA/RA at Vention Medical, Inc., and the VP of Regulatory Affairs and Quality Assurance at Aspen Surgical Products. Her experience includes executive management responsibility for global medical device manufacturing and contract manufacturing companies. She had responsibility for quality and regulatory due diligence and integration, along with collaboration and planning with all functional areas, for several successful acquisitions during her career. Her responsibilities included design control, document control, change control, risk management, inspection, manufacturing, assembly, packaging, labeling, sterilization, biocompatibility, environmental management, validation/ verification, distribution, marketing, post-market surveillance activities, internal audits, customer audits, supplier audits, regulatory audits, supplier management, global submissions, licensing, registrations, and certifications for Classes I, II, and III medical devices including implantable devices. She has also worked in the clinical laboratory arena and IVD in the areas of Chemistry, Hematology, Microbiology, Virology, Mycology, Immunology, and Blood Banking. Her current areas of responsibility include CMC plant transfer activities, post-market variations, document control, special project group, and a full service of medical device service offering.   178 Jane Garbutt et al.     Financial and economic literacy 6 1 Financial planning is a managerial role and not necessarily the role of a basic scientist. This is not needed by everyone and in fact should be a task allocated only to the finance staff of a company, start-up, or university Mobilizing resources 7 0

Panel 3: Social Skills
Spotting opportunities 6 1 While this seems like a "yes" answer, in fact basic scientists or managers with a definite task should focus on that and not be looking for "opportunities" wherever they exist.
Vision 5 2 (1) While important to start with a vision, this needs a better definition in entrepreneurship and innovation. A vision may need to be very flexible in this setting, but are sometimes rigid and can be a hinderance for innovation.(2) Again, many people work in a focused area without the need to visualize the future -I want my staff to get the analysis done, and not worry about how or if we will market certain areas for future growth Valuing ideas 6 1 Value to me is a monetary return and often i do not need staff to worry about this. It is often sufficient for the boss to say do this and not needed for everyone to be able to demonstrate this Creativity 6 1 Is your future job to plow through a defined scientific problem using established tools, or is it to develop an independent group (lab, start-up, core facility)?
Self-awareness and self-efficacy 7 0 Mobilizing others 6 1 This is an important leadership trait, but may be delegated to a manager. Effective entrepreneurs are not always the best managers Working with others 7 0 Learning through experience 7 0 Taking the initiative 6 1 Not everyone needs or has leadership skills Motivation and perseverance 6 1 Not everyone needs or has leadership skills Coping with uncertainty, ambiguity, and risk 7 0 Ethical and sustainable thinking 6 1 I don't believe everyone has the cognitive ability to analyze the consequences of their work