{'fraction_non_alphanumeric': 0.01832452643614227, 'fraction_numerical': 0.005809462006736504, 'mean_word_length': 5.599102773246329, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data Management Tools to Measure the Impact of Core Facilities

The Biomolecular Research Center at Boise State University is a research core facility that supports the study of biomolecules with an emphasis on protein structure and function, molecular interactions, and imaging. The mission of the core is to facilitate access to instrumentation that might otherwise be unavailable because of the cost, training for new users, and scientific staff with specialized skills to support early-stage investigators, as well as more established senior investigators. Data collection and management of users and their research output is essential to understand the impact of the center on the research environment and research productivity. However, challenges are often encountered when trying to fully quantify the impact of a core facility on the institution, as well as on the career success of individual investigators. This challenge can be exacerbated under the conditions of unprecedented growth in biomedical research and shared core facility use that has been experienced at Boise State University, an institution of emerging research excellence. Responding to these challenges required new approaches to information management, reporting, assessment, and evaluation. Our specific data management, evaluation, and assessment challenges included 1) collection and management of annual reporting information from investigators, staff, and students in a streamlined manner that did not lead to reporting fatigue; 2) application of software for analyzing synergy among programs' management strategy and investigator success; and 3) consolidation of core facility management, billing, and reporting capabilities into 1 cohesive system. The data management tools adopted had a beneficial effect by saving time, reducing administrative burden, and streamlining reporting. Practices implemented for data management have facilitated effective evaluation and future program planning. The substantial burden of assessment requirements necessitates early consideration of a strategy for data management to allow assessment of impact.


INTRODUCTION
The Biomolecular Research Center (BRC) at Boise State University is a research center with shared core facilities located at an institution of emerging excellence in biomedical research. The focus of the BRC is the study of biomolecules with an emphasis on proteins and their molecular interactions. The BRC is also the administrative center for multiple programmatic biomedical research grant awards that emphasize mentored career development and student training. To evaluate the performance of the center as a whole, as well as its individual shared core facilities and specific programs with respect to their contributions to both the research and educational missions of Boise State University, we adopted specific evaluation and assessment data management tools and evaluation approaches. These measures were put in place so that the center could create required reports in a timely manner for the institution and for individual funding agencies. These efforts support program management and promote sustainability of the center, so the research infrastructure provided can continue to serve an integral role in the research and research training mission at Boise State. These practices may apply to other primarily undergraduate institutions with a rapidly growing and emerging biomedical research emphasis.
Within the BRC, small shared core facilities (,$200,000 of recharge income annually) support research. The National Institutes of Health (NIH) defines core facilities as centralized shared resources that provide access to instruments, technologies, and services, as well as expert consultation for scientific and clinical investigators. 1 Core facilities are discrete units within an institution and may have dedicated personnel, equipment, and space for operations. In general, core facilities recover the cost, or a portion of the cost, of providing services in the form of user fees that are charged to an investigator's funds, often to NIH or other federal grants. 1 Instrumentation within the BRC core laboratories supports recombinant protein expression and purification, mass spectrometry, analytical instrumentation for characterization of molecular interactions, histology and microscopy, and cell/molecular biology investigations. Although the focus of the BRC is the study of biomolecules with an emphasis on proteins and their molecular interactions, the BRC has an additional level of emphasis on extracellular matrix molecules, cell-matrix interactions, and both the biochemical and biomechanical properties of extracellular matrices and those used for biomimetic scaffolds as part of one of the programs administered by the center, the NIH Center of Biomedical Research Excellence (COBRE) in Matrix Biology Program.
In addition to the COBRE in Matrix Biology, the BRC is the administrative center for statewide and regional programmatic biomedical research grant awards for which Boise State University is a partner institution, including the NIH-sponsored Idaho Institutional Development Awards (IDeA) Network of Biomedical Research Excellence (INBRE) and the Mountain West Clinical Translational Research Infrastructure Network programs. These programs share the goals of workforce development, career mentoring, and expanding research capacity for biomedical and behavioral research and are programs within the IDeA programs at NIH.
The BRC also provides administrative support for equipment grants that provide instrumentation for the regional research community from the National Science Foundation (NSF) and the M. J. Murdock Charitable Trust. Additionally, the BRC administers a limited number of individual researcher-initiated awards from agencies, including the American Heart Association, the National Aeronautics & Space Administration, and NSF. These awards have made a significant impact on research and research infrastructure growth at Boise State.
The NIH INBRE program is designed to foster the development, coordination, and sharing of research resources and the expertise that will expand research opportunities and increase the number of competitive investigators in IDeA-eligible states. The program is also designed to enhance the caliber of scientific faculty at research institutions and undergraduate schools to continue to attract talented students to augment the science and technology knowledge of the state's workforce. A multidisciplinary research program within a broad biomedical thematic focus creates an environment that strengthens research expertise and infrastructure and fosters collaboration.
The NIH COBRE program strengthens biomedical or behavioral research capacity in institutions from IDeA states. COBRE Phase I provides support to develop research infrastructure and to foster independence of junior investigators. COBRE Phase II is designed to continue the progress toward building an independent research center that is competitive for support from NIH and other funding agencies. The COBRE Phase III awards provide support for maintaining COBRE research cores developed during phases I and II that are essential for the continuing conduct of basic, clinical, translational, and community-based research at the institution and to sustain a collaborative, multidisciplinary research environment for research pilot projects and mentoring. Each phase takes place over a 5-y period. Boise State has recently completed phase I (2014-2019) and has transitioned to phase II (2019-2024).
The growth in biomedical research and shared core facility use that we experienced required new approaches to information management for reporting, assessment, and evaluation that were not available at the university previously. Here, we report on our approach to streamline administrative functions and increase the efficiency of information management and the ability to assess and evaluate our center's impact on investigators' success, as well as programmatic success. Specifically, the data management challenges included 1) collection and management of annual reporting information from investigators, staff, and students; 2) implementation of software for analyzing synergy between our program's management strategy and investigator success; and 3) consolidation of core facility's management, billing, and reporting capabilities into 1 cohesive system.
The results indicate that the adoption of an effective data management tools can lessen the administrative burden and allow a more effective day-to-day management of center operations while increasing the effectiveness of evaluation, assessment, and reporting. The substantial burden of data management and assessment requirements for research centers and programs necessitates early consideration of a thoughtful and meaningful strategy for data management. Flexibility must be maintained to allow continuous improvement to approaches used for assessment and evaluation in ways that streamline reporting and dissemination of outcomes to all stakeholders. Here, we present lessons learned for institutions of emerging research excellence.

Acquisition of graduate program information
The 10th day of the fall semester enrollment numbers were reported by the Registrar's Office at Boise State University. 3 Collection and management of annual reporting information from investigators, staff, and students The BRC assisted in the development of a customized data management system to collect information from individuals to fulfill the reporting requirements of funding agencies and the reporting requirements of Boise State University. Forms were created specifically for individual roles within the BRC and its associated programs. Students, research staff, faculty members, project investigators, and core managers completed unique forms with customized interfaces. This information was self-reported annually, and collected information was automatically deposited into a database.
Implementation of software for analyzing synergy between programmatic components and investigator success Publication data queries for grant P20GM109095 were made through the MyNCBI software system, a tool that retains user information and database preferences using features that save citations and manage peer-reviewed article compliance with the NIH public access policy. 4 This information was added to the bibliography section of MyNCBI, and then bibliometric data were exported from MyNCBI and imported into Gephi to derive the statistical information on the coauthorship network. 5 Visualization of the resulting network was rendered using the Forced Atlas 2 algorithm. 6 Consolidation of core facility's management, billing, and reporting capabilities into 1 cohesive system Agilent iLab Operations Software (iLab) was used for core facility management and billing. The web-based system Logic model. The logic model presents the long-term goals of the BRC and information about the resources available to the center (input), activities that are planned, types of output, the manner in which output is monitored, information that is collected (output collection), and the anticipated effect of the planned activities for each of the aims of the center. ACS, American  facilitates service requests, project management, billing, and invoicing for shared research core facilities. The combination of these methods streamlined administrative functions and increased efficiency of information management and, in turn, our ability to assess and evaluate the center's impact on investigators' success, growth of research programs, and core facility sustainability.

Logic model
The design of evaluation strategies for programmatic assessment was outlined in a logic model. The logic model shown in Fig. 1 represents the relationship between programmatic resources, planned activities, and the potential outcomes of the program. The logic model includes information about the resources available to the center (input), activities that are planned, types of output, the manner in which output is monitored, information that is collected (output collection), and the anticipated effect of the planned activities for each of the aims of the center. This logic model serves as a framework for monitoring and evaluation to assess the extent to which goals are achieved.

History
Boise State University has maintained a robust undergraduate teaching mission since its establishment and an everincreasing focus on research growth since 2000. Growth in research programs from 1993 to 2019 exceeded a 10-fold increase, as shown in Fig. 2. Fig. 2 illustrates the growth of research activity over this time frame, with the annual level of research expenditures at Boise State shown as a function of year. Boise State received NIH IDeA funding starting in 2001, and total NIH funding is shown in blue in Fig. 2. Fig.  3 shows the levels of IDeA compared with total NIH funding at Boise State. Representing a small percentage of the total funding, biomedical and behavioral research at Boise State also increased over this time. Future growth will be due in part to the infrastructure investments made by the NIH IDeA Program to create shared core facilities.
Biomedical research growth fueled by NIH investments in core facility infrastructure The NIH-sponsored Idaho INBRE program acted as the catalyst for the creation of the BRC through the establishment of infrastructure, funding, leadership, and mentorship starting in 2001. Like other institutions, Boise State University established shared core facilities to support the growth of independent and collaborative research efforts by providing access to shared instrumentation. According to Hockberger and colleagues, 7 sharing services is driven by the high cost of instrumentation and the technical expertise needed to operate this equipment. Consequently, researchers have become increasingly reliant on core facilities to access state-of-the-art instrumentation and services in a safe, productive, and cost-effective manner. 7 The increase in biomedical research at Boise Dotted lines represent the best fit for the data points and projection for near-future NIH funding levels.
State is due in part to the availability of core research facilities for all researchers.
As an institution of emerging excellence, Boise State University is a developing doctoral research institution. For the academic year 2019-2020, 187 out of 382 students were enrolled in recently established Boise State University doctoral degree programs that have the potential to contribute to biomedical research. Several of the relatively new programs have the potential to contribute to research and research training with relevance to biomedical research, and these are shown in Table 1.
Collection and management of annual reporting information from investigators, research staff, and students Performance standards and metrics for evaluation are critical in the assessment of outcomes and evaluating the need and effectiveness of core facilities. 8 Evaluation of the growth in biomedical research and BRC shared core facility use required new approaches to information management.
A customized system was developed to collect information from individuals. This information was essential to fulfill the reporting requirements of funding agencies and Boise State University. Forms were created specifically for unique roles within the BRC and its associated programs. Students, research staff, faculty members, project investigators, and core managers completed unique forms with customized interfaces. This self-reported information was collected annually, checked for accuracy and completeness by the program directors and then by the INBRE and COBRE evaluators, and deposited into a database. The collection of information and having the ability to store it in a searchable database were critical to allow management of information as the center data volume grew over time. Additionally, the system enabled longitudinal reports and analysis of trends over time ( Table 2).
Collaboration between the Idaho INBRE program and the COBRE program for information collection reduced the reporting burden on participants who were required to report much of the same information to several entities by providing 1 custom system for self-reported information.
The custom system allowed for program-specific information tailored to individuals based on assigned roles. The expectation was that reduction of reporting fatigue would result in more reliable self-reported data.  Collection and management of annual reporting information uses the following steps: 1) create a yearly roster of participants and designate appropriate role(s), 2) assign appropriate modules/forms, and 3) request that all individuals on the roster complete assigned modules/forms and update personal profile by a specific date. Self-reported information is collected in a central database by a pre-established reporting year, and database information can be exported in various ways depending upon need and specific question. The gathered information is used for federal reporting requirements such as the Research Performance Progress Report (RPPR) and the Scientific Information Reporting System (SIRS). Additional reports to key stakeholders such as the External Advisory Committee, Boise State University Institutional Center Report, annual activity reports for faculty and staff, renewal of externally funded programs, and other reports can also be generated as needed. Fig. 4 illustrates the workflow for individual data collection and the flow of feedback for improvements to the process for future years.

Application of software for analyzing synergy between programmatic components and investigator success
Social network analysis can be used to map collaboration and produce evidence for effective team science. 9 We applied social network analysis methods to analyze the outputs of research from the participant user base. Data were retrieved for a 5-y period from MyNCBI, resulting in 96 publications that cited one of our programmatic grants, the COBRE in Matrix Biology (P20GM109095). Publication data were exported from MyNCBI and imported into Gephi to derive statistical information on the coauthorship network. 5 Visualization of the resulting network shown in Fig. 5 was generated using the Forced Atlas 2 algorithm. 6 Nodes representing each of the 377 authors within the coauthorship network were sized based on the number of publications for that specific author, ranging from 1 to 15 for our network. Lines connecting nodes (edges) were weighted to indicate the strength of connection; the thicker the edge, the more connections between those 2 nodes. Workflow for the collection of self-reported information. Collection and management of annual reporting information uses the following steps: create a yearly roster of participants, designate appropriate role(s), assign appropriate modules/forms, request all individuals on roster complete assigned modules/forms and update personal profile by a specific date, collect selfreported information in a central database by a pre-established reporting year; and exported database information in various ways depending on need. Information is used for federal reporting requirements such as the RPPR and the SIRS. Additional reports are created from this database to disseminate information to other key stakeholders such as the External Advisory Committee and the Boise State University Institutional Committee for Centers and Institutes. The information is also useful at the time of renewal of externally funded programs.
Analysis of core facility staff within the network was included in addition to individual researchers that represent users of the shared core facility to assess the role that shared core facility scientists play in supporting research productivity represented by publication in peer-reviewed journals.
Core facility research staff members (orange nodes) support research productivity and output. The network grew during the 2014-2019 time frame, as shown by comparing Fig. 5A, B. Output and productivity increased over this time frame.
Consolidation of core facility's management, billing, and reporting capabilities into 1 cohesive system Core facility management and billing software was adopted for the shared core facilities within the BRC. This approach streamlined administrative functions and increased efficiency of information management and the ability to assess and evaluate the center's impact on investigators' success and overall programmatic success.
With the adoption of core facility management and billing software, the BRC more efficiently managed day-today operations as well as the processing of billing and invoicing. The core facilities now provide a uniform and consistent customer experience from initial contact to receipt of payment, as shown in Fig. 6. Details of projects are managed and maintained through this software. Communications between the core and customers are managed through this system as well. Researchers who work independently on equipment have the flexibility and transparency of scheduling on a shared calendar. There are variations of the workflow that include independent customer use of the facility as well as training opportunities provided by the core.
Using our customized data collection and management system, we could assess and evaluate the number of researchers that rely on the BRC to support their research programs and the productivity of investigators that utilize the BRC. Since 2014, we tracked the growth of users of shared core facilities that support biomedical research. Figure 7 shows the increase in the number of biomedical investigators as well as the increase in investigators using the BRC each year over the 5 years from 2014 to 2018.

Productivity of investigators that utilize the BRC
Researchers that utilize the BRC cite both the Idaho INBRE and COBRE in Matrix Biology grant awards. Instilling the importance of award citation and training in the correct manner in which to cite the funding source allowed the programs to meet rigorous compliance requirements. Additionally, we were able to conduct comprehensive searches in NIH databases for journal articles, thereby enabling the collection of information in a systematic manner that did not rely on individual self-reporting to obtain the majority of cited work. Figure 8 illustrates growth over time of cited peer-reviewed journals from 2014 to 2018. The center's administrative staff diligently monitors publications to ensure that NIH public access policy is observed and that all investigators funded by the NIH submit publications to the National Library of Medicine's PubMed Central. 10 Shared core facility management software With the adoption of core facility management and billing software, the BRC more efficiently managed day-to-day facility operations as well as the processing of billing and invoicing. BRC customer workflow. The shared research core facilities standardized the workflow required for the customer interface. This workflow supported more efficient management of day-today operations and billing/invoice processing.
Investigators experience uniform and consistent support from initial contact to receipt of payment.
The core facility management software includes a robust reporting system that is available for the generation of reports delineated by an individual investigator, the core management team, or at an institutional level to address a specific aspect of the core facility. Reports can span specific periods of time, compare individual functions within a core, generate figures, and export detailed information in a Comma Separated Values (CSV) file format. These data can be filtered at a granular level by time frame, funding source, research lab, individual, institution, or equipment use. The data can also help to aid in the evaluation of the rate review process, analyze equipment usage, increase the understanding of core staffing needs, and provide detailed information about individual users or lab activity. Reports, figures, spreadsheets, and data can be generated and used for program evaluation. The reports can also provide valuable data to the institution, stakeholders, or potential grantees. Streamlining the process for core facility management and office administration allows for a more effective assessment of the core facility. The office administrators benefit from having a cohesive, transparent, and automated way of invoicing customers and processing payments. Research activity from 2014 to 2018. Data management tools were used to monitor the number of users of shared core facilities for biomedical research. An increase in the number of biomedical investigators (participants, shown in blue bars), as well as an increase in the number of investigators using the BRC (red bars) and the Biomedical Research Vivarium (BRV) (green bars) each year, was observed over the 6 y from 2014 to 2018. Analysis of the data assists the core facility personnel as they make informed long-and short-term business decisions. Figure 9 demonstrates the departments supported by the BRC core facilities (Fig. 9A) and the services that were used (Fig. 9B). This information is vital to the planning of sustainability of shared research core facilities.

Lessons learned
Over time, we used several custom data management tools to collect individual annual reporting information for stakeholders. Regardless of which tool we adopted, there was always an administrative burden associated with it, especially for faculty, who were required to report much of the same information to many entities. We recognized that most of the BRC-affiliated faculty members were required to report similar information to both the BRC and Idaho INBRE programs annually. To reduce the administrative burden that individual reporting requires, we collaborated with the Idaho INBRE program to create a unique custom database system. This system allowed for individuals to report all requested information from both program rosters into 1 database. Therefore, individuals only report the same information-such as funding, presentations, and publications-once while having programspecific information tailored to individuals based on assigned roles. Some information from the previous year carries forward and autopopulates the current year's report, further reducing the reporting burden experienced by the investigator. We expected that we would obtain more reliable individual reports while simultaneously reducing reporting fatigue. After the first round of reporting with the new system, feedback indicated that this approach was well received by faculty members, and we plan to continue collecting individual reporting information in the shared database. Collection and management of annual reporting information uses the following steps: create a yearly roster of participants, designate appropriate role(s), assign relevant modules and forms, and request that all individuals on the roster complete assigned modules and forms and update their personal profile by a specific date. Self-reported information is collected in a centralized database by a preestablished reporting year, and database information can be exported in various ways depending on need. As noted earlier, the data are used for federal reporting requirements such as the RPPR and the SIRS. Additional reports to key stakeholders such as the External Advisory Committee, Boise State University Institutional Center Report, annual activity reports for faculty and staff, renewal of externally funded programs, and other reports can be generated as needed.

Future directions
The substantial burden of data management and assessment requirements for research centers and programs necessitates early consideration of a thoughtful and meaningful strategy for data management. Adoption of these tools and best practices lessens the administrative burden and increases effectiveness in managing day-to-day center operations while also increasing transparency of core facility activity and our ability to report more comprehensively, making assessment and evaluation more meaningful. We will continue to improve our application of software tools for data collection, assessment, evaluation, and information retention. Dissemination of outcomes will progress in an efficient, streamlined manner to all stakeholders as our center continues to grow.
Like all share core facilities, the BRC must investigate changing dynamics over time at multiple levels, including FIGURE 9 Distribution of customers supported by shared core facilities. Data management tools were employed to monitor the customer base for shared core laboratories within the BRC and the types of services used by investigators. A) Disciplines supported by the shared core laboratories. B) Laboratory use within the shared core facility. A) Distribution of users and their specific discipline or department is illustrated as a pie chart, indicating that users are distributed among 5 general areas, Chemistry and Biochemistry, Health Sciences, Engineering and Physics, Biologic Sciences, and External Users, which include investigators from nearby 2-and 4-y colleges and universities, as well as research institutes and industry. B) Services provided by laboratories within the shared core facility include Mass Spectrometry, Biostatistics, Histology, and Microscopy, Bioinformatics, Cell Culture and Recombinant Protein Expression, and Other Analytic Services, which includes the characterization of molecular interactions and protein structure. The relative size of the wedge is indicative of the percentage of the total users for each type of service.
overall programs such as the COBRE in Matrix Biology, individual laboratories within the BRC, specific research projects, and personnel, including junior investigators, who play the role of mentored young scientists. Social network analysis tools provide a more in-depth understanding of relationships among faculty, staff, and student participants as we include analysis of the overall size of the research community and the connectivity within the network. We anticipate that network analysis will yield information about the role of core scientists within research projects of junior investigators and their career development, as well as the impact that program grants like the COBRE in Matrix Biology and the Idaho INBRE program have on establishing an environment that promotes research growth and productivity at an institution of emerging excellence.

CONCLUSIONS
The BRC at Boise State University is a research center that includes core facilities and administers multiple programmatic and individual researcher-initiated biomedical research grant awards. These awards have made a significant impact on research and research infrastructure at Boise State, an institution of emerging excellence. The Idaho INBRE program acted as the catalyst for the creation of the BRC through the establishment of infrastructure, funding, leadership, workforce development, and mentorship. As a result, Boise State increased its NIH-funded programs and received its first COBRE grant in 2014. The growth in biomedical research and shared core facility use required new approaches to information management for reporting, assessment, and evaluation.
Data management approaches to support our rapid growth had a beneficial effect on time management, reduction of administrative burden, and efficiency of reporting. In addition, the approach supported ease of data collection, evaluation, assessment, and project management decisions. Adoption of best practices has lessened the administrative burden, more effectively managing day-today center operations while increasing transparency of core facility activity.
The substantial burden of data management and assessment requirements for research centers and programs necessitates early consideration of a thoughtful and meaningful strategy for data management. Flexibility must be maintained to allow continuous improvement to approaches used for assessment and evaluation in ways that streamline the dissemination of outcomes to all stakeholders.