{'fraction_non_alphanumeric': 0.024740471319239222, 'fraction_numerical': 0.012502841554898841, 'mean_word_length': 5.302531041069723, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
California's COVID-19 Virtual Training Academy: Rapid Scale-Up of a Statewide Contact Tracing and Case Investigation Workforce Training Program

Case investigation (CI) and contact tracing (CT) are key to containing the COVID-19 pandemic. Widespread community transmission necessitates a large, diverse workforce with specialized knowledge and skills. The University of California, San Francisco and Los Angeles partnered with the California Department of Public Health to rapidly mobilize and train a CI/CT workforce. In April through August 2020, a team of public health practitioners and health educators constructed a training program to enable learners from diverse backgrounds to quickly acquire the competencies necessary to function effectively as CIs and CTs. Between April 27 and May 5, the team undertook a curriculum design sprint by performing a needs assessment, determining relevant goals and objectives, and developing content. The initial four-day curriculum consisted of 13 hours of synchronous live web meetings and 7 hours of asynchronous, self-directed study. Educational content emphasized the principles of COVID-19 exposure, infectious period, isolation and quarantine guidelines and the importance of prevention and control interventions. A priority was equipping learners with skills in rapport building and health coaching through facilitated web-based small group skill development sessions. The training was piloted among 31 learners and subsequently expanded to an average weekly audience of 520 persons statewide starting May 7, reaching 7,499 unique enrollees by August 31. Capacity to scale and sustain the training program was afforded by the UCLA Extension Canvas learning management system. Repeated iteration of content and format was undertaken based on feedback from learners, facilitators, and public health and community-based partners. It is feasible to rapidly train and deploy a large workforce to perform CI and CT. Interactive skills-based training with opportunity for practice and feedback are essential to develop independent, high-performing CIs and CTs. Rigorous evaluation will continue to monitor quality measures to improve the training experience and outcomes.


INTRODUCTION
In March-April 2020, California was experiencing a rapid increase in COVID-19 incidence, and the State's Department of Public of Health needed to rapidly develop a coordinated pandemic response that included case investigation (CI) and contact tracing (CT). CI and CT, in combination with isolation of cases and quarantining and testing of contacts, are key elements of the COVID-19 response (1)(2)(3)(4)(5)(6). CI and CT have been a cornerstone of the public health response to other infectious disease outbreaks for decades; as evidence of this, they have been used in various countries for HIV (7), Ebola (8), tuberculosis (9,10), and hepatitis C (11) outbreaks. Moreover, the observation that countries with robust and expansive CI and CT programs have effectively controlled transmission of SARS-CoV-2 (e.g., Singapore and New Zealand) (12,13) has led to calls for expansion of similar programs in the United States. Despite recognition that a large workforce was needed, few jurisdictions in the U.S., including California, had a comprehensive CI and CT training plan, a clear assessment of ongoing training needs, a plan to operationalize training on a large scale or adequate funds budgeted for training. Stay-at-home orders and widespread shifts to working remotely during the COVID-19 pandemic introduced additional challenges to training, which is typically conducted in-person.
We describe the experience in California, where a unique partnership between the University of California, San Francisco (UCSF), University of California, Los Angeles (UCLA) and the California Department of Public Health (CDPH) rapidly implemented a training program for CI and CT. The description outlined below highlights three mutually related facets that illustrate the uniqueness of the effort, but also provide lessons for other jurisdictions. Firstly, the training program was designed and implemented to be delivered rapidly and at scale. The state's target of training several thousand contact tracers and case investigators by mid-summer demanded an approach to training that leveraged online learning resources. Secondly, the training program design was informed by a conviction that asynchronous didactic learning would be insufficient to prepare a workforce to effectively contain COVID-19. The initial audience for this training was redirected civil servants, such as census workers and librarians, who had very limited public health experience. As a consequence, the training needed to focus both on theory and the practice of CI and CT (14). As we highlight, the curriculum was distinct from other available CI/CT training tools that focused primarily on teaching principles without also providing training on the practical skills critical to reaching those impacted by COVID-19. Thirdly, the training program was developed with early recognition that learners needed not only the knowledge and skills to perform this essential function, but also sensitivity and empathy to reach those communities most impacted by COVID-19. Accordingly, we embedded a module in the core training focusing on cultural humility and the social context of people's lives (15,16). The main objective of this paper is to highlight the different components of the curriculum development process and implementation, so that our approach can serve as a model of evidence-informed COVID-19 specific CI/CT instructional design.

METHODS AND PEDAGOGICAL FRAMEWORK
To guide the design and implementation of the project, a Steering Committee was formed consisting of 15 infectious disease epidemiologists, clinicians, community health scientists, instructional designers, learning specialists and evaluation experts from UCSF, UCLA, and CDPH. This group had expertise in professional training of both a public health workforce and others who are not public health professionals. This group took a systemic approach to curriculum design as outlined below.
From April 27 to 30, 2020, the Steering Committee undertook a rapid asset and needs assessment to determine goals, objectives and strategies of the new training program. The rapid asset and needs assessment consisted of: (1) Identifying areas of expertise of the team. Several task forces were created to manage the many logistics of this large and comprehensive effort ( Table 1). (2) Identifying existing training resources, both those available freely online and those previously created by the team members. (3) CDPH conducting a statewide survey of the 61 local health jurisdictions to assess the needs to support COVID-19 CI and CT programs and demand for a training program. This survey detailed assessment of each jurisdiction's CI/CT workforce, including the size of their workforce, the digital platform(s) used and the training program(s) used to equip people to do CI and/or CT.
The needs assessment identified an urgent need for trainingmost jurisdictions had limited capacity to provide training-as well as a paucity of high-quality training materials focused on teaching COVID-19 CI and CT. Subsequently, between April 27 and May 5, 2020, the team rapidly developed a training curriculum that would enable learners from diverse backgrounds and experiences to quickly acquire the competencies necessary

Course Platforms
The initial four-day CT curriculum consisted of approximately 13 h of synchronous live web meetings and 7 h of asynchronous, self-directed online study (

Webinars
With an initial plan to train 2,500 learners per week, instructorled presentations were delivered using the Zoom video webinar function, which enables up to 3,000 participants to join simultaneously (including learners, observers and facilitators). These webinars addressed much of the core content, delivered in a didactic "presentation format" by module presenters, who were Steering Committee members who had expertise in the topic being presented. Module presenters were encouraged to make the webinars as interactive as possible using four techniques: (1) poll questions using the Zoom poll function; (2) live audience response using the Zoom chat function (speaker posed question and quickly acknowledged themes and notable responses gathered from learners in the chat); (3) real-time texting using the Zoom Q&A function (supporting faculty fielded written Q&A during the presentation and curated questions for live Q&A discussion); and (4) live Q&A discussion with speaker at end of each presentation.

Skill Development Labs
In addition to knowledge delivered through didactic presentations, the team developed a curriculum emphasizing learning practical interviewing skills through four 1.5-h "skill development labs." Skill sessions covered (1) role-play practice using a CT script, (2) applying interviewing and health coaching principles to challenging situations, and (3) two CI-track skill sessions for role-play focused on challenging segments of the CI script (see Supplementary Material). Skill development labs were led by a lead facilitator, who introduced role-play activities to the whole group, after which learners were separated into small breakout groups of 5-10 learners. Trained facilitators were typically assigned to each breakout group, although in some cases when enrollment was high, the facilitators rotated between breakout groups. During the small breakout groups, learners practiced delivering the CI or CT script using the interviewing skills and health coaching techniques described during the webinars. For example, in the first CT skill development lab, learners practiced reciting the introduction of the CT interview script, conducting an interview with a contact and addressing common concerns a contact might have about being notified or interviewed. The facilitators' role included leading the small group in introducing themselves, providing instructions for each activity, ensuring each learner had an opportunity to practice, providing tailored, in-depth feedback to each learner to strengthen their skills, keeping track of the time and answering learner questions. The opportunity to practice interviewing skills is what distinguished the VTA courses from other available online CI and CT courses. Skill development labs met on Days 2 and 3 of the 4-day course using the Zoom meeting function. CT learners were required to participate in two labs, and CI learners were required to participate in four labs. Because the Zoom meeting function allowed a maximum of 300 participants, we limited the number of learners in each skill development lab to 250, which allowed for 10-50 facilitators and production staff to join as well ( Table 4). With a maximum of 2,500 learners joining the course per week, ten daily skill development labs were required to accommodate all learners. On Day 4, we offered optional skill development labs where learners could experience extra practice or make up a skill development lab missed during the week.

VTA Facilitators and Production Staff
Given the large number of learners each week and the need to engage them in small group practice sessions, the VTA required a large number of trained facilitators. Facilitators were recruited  rapidly among public health graduate students, staff and faculty of UCLA and UCSF, as well as retired health workers. A total of 104 facilitators were initially recruited and trained for 4 h over two consecutive weeks in May 2020. Training consisted of an overview of the CI and CT courses, navigating the Canvas platform and Zoom, review of facilitator roles, content of the skill development labs, and best practices for facilitating small groups. The training was conducted on Zoom, which also allowed new facilitators to practice some of the Zoom functions. Facilitators worked for the VTA part-time between 2 and 15 h per week. As new facilitators joined the VTA, they attended an onboarding meeting on role and logistics, participated in the CI and CT training, and then shadowed experienced facilitators during a skill development lab before facilitating on their own. All facilitators met during a bi-weekly "facilitator check-in" consisting of an overview of the number of learners who had completed the course, announcements and updates to learning materials, and ongoing training and development. All activities of the VTA facilitators were coordinated by the Facilitator Task Force consisting of nine members who met twice weekly to coordinate the activities described above. In addition to facilitators, implementing the VTA required production and technical support staff. Production staff managed the Zoom webinars and meetings by setting up waiting rooms for participants, recording the presentations, sharing slides and videos, enabling or disabling the chat function, facilitating poll questions and managing breakout rooms, among other duties. Production staff were primarily college students hired on a part-time basis. Technical support was provided by UCSF Educational Technology Services and UCLA Extension and included helping learners and staff with technical problems (such as audio and video) and answering technical questions using the Q&A function of Zoom. The Production Task Force hired and managed the production and technical support staff, in addition

Selection of Learners
The VTA training courses were offered to the following groups of people performing or training to perform contact tracing or case investigation activities: (1) California state employees re-directed to COVID-19 work; (2) local health department employees, contractors and partners designated to provide support to local health department COVID-19 contact tracing activities; (3) tribal nations. Because of the high demand, the courses were not open to the general public in order to save space for learners who would be doing case investigation and contact tracing activities. Learner registration and enrollment was managed by the Registration Task Force and UCLA Extension using Destiny, a registration portal system. Eligibility for each person enrolled in a course was verified by VTA staff before access to the course was given.

Course Evaluation
A program evaluation plan, aligned with the course goals and objectives, was designed to capture learners' sociodemographic characteristics, course participation, knowledge and selfassessment of skills. Learners who passed the knowledge assessment with ≥ 70% correct were emailed a record of completion within 2 weeks of course completion. Learners provided feedback on the course content and delivery at the time of training and again 2 weeks after completion of the training. Feedback was used to inform iterative improvements of the course content and delivery and logistical changes to improve learner experience.

Pilots and Move to Canvas Platform
The VTA training program was piloted in early May 2020 to two groups of learners affiliated with the San Francisco Department of Public Health (n = 31) before being rolled out to a statewide audience. Feedback was collected from these learners through a Qualtrics survey; learners wanted less didactic training and more time for practicing interviewing skills. Incorporating feedback from the pilot, we implemented a second and expanded CT training to a group of 547 learners on May 7-13, adding more time for small group skill development labs. From May 18 onwards, the CI and CT courses were delivered on the UCLA Canvas platform; the first cohort to use Canvas consisted of 443 learners.

Sociodemographic Characteristics of Learners
An average of 520 learners a week were trained (range 163-1,120) during the first 16 weeks of training (May 7-August 31, 2020) ( Table 5). Most learners held a non-public-healthrelated job title, such as librarian or city attorney, before being redirected to COVID-19 work (71.9%). Responses from selfreported demographic questionnaires indicated that the majority of learners were female (71.3%), and many were college graduates or had more advanced degrees (68.3%), were White/Caucasian (48.2%), and not of Hispanic, Latinx or Spanish origin (69.7%). The age distribution of learners was spread evenly across age groups from 25-64 years, with a smaller proportion of learners aged 18-24 (7.6%) and 65 or older (8.0%). Spanish (14.8%) was the most frequently spoken language in addition to English (63.4%), with 18 other languages selected by at least one learner. At the start of each week, the Monitoring and Evaluation Task Force analyzed and distributed the demographic data of the learners to VTA staff in order to help presenters and facilitators better prepare for incoming cohorts.

Iteration and Development of New Content
The VTA continued to redesign the course content and format based on feedback from learners and facilitators. Module presenters were invited to make updates to the slides or materials from week to week based on updated guidelines and evaluation results (such as knowledge scores on specific module-related questions). Substantial revisions of the course content were made in July 2020 and were based on learner and external CDPH observer feedback. Revisions included expansion of cultural humility content (see Supplementary Table 1), improved skill development activities and scripts, additional skill development labs (optional practice and development of Spanish-speaking practice labs and materials), a reduction in live didactic time with concomitant increase in self-study video options, and expanded use of real-life stories from CIs and CTs in the field  (live Q&A and video testimonies). In August 2020, in response to increased demand for case investigator training, the VTA created a new CI course for alumni of the VTA CT course that allowed these learners to attend only CI webinar modules and skill development labs, but not repeat content they had previously covered.

DISCUSSION
To our knowledge, this is the first time that a COVID-19 contact tracing skills-based training program has been adopted at scale in the U.S. The program is unique given both its focus on equipping learners with the interviewing and health coaching skills necessary to perform this essential task and the rapidity with which it was implemented. The iterative approach to curriculum development and the emphasis on cultural humility are also central tenets of the California COVID-19 contact tracing training effort. We assert that the California contact tracing program offers not only a model in how to scale COVID-19 contact tracing training, but also what core components should be included in such a training program. Outlined below are several reasons why the California CI and CT training program offers a model for other jurisdictions: Firstly, the CI and CT training program, targeted to learners with limited public health experience, was implemented with considerable speed. Within 16 weeks of inception, the program had trained 7,499 unique learners who could then be deployed as part of the state's COVID-19 public health response workforce. This rapid scale-up was achieved through a unique partnership between CDPH and the University of California, leveraging CDPH's ability to quickly mobilize learners from across the state and the university's capacity to quickly implement an online training program. While the need for social distancing precluded in-person training, it offered a unique opportunity to use the UC's existing online educational capability. Moreover, using an online format allowed the training team to reach learners from across the large state of California.
Secondly, a key facet of the California CI and CT training program is its central emphasis on equipping learners with critical practical skills necessary to perform case investigation and contact tracing effectively. While other available contact tracing training curricula highlight the importance of developing interviewing skills, few provide the infrastructure to support learners to develop these skills in live, interactive web-meetings. Given many of the learners undertaking contact tracing in California had no public health experience, we assert that this 'hands-on approach' to training was critical. Moreover, this approach is supported by data from contact tracing training programs for other diseases including syphilis and tuberculosis, where skills-based modules are essential to enhancing the proficiency of learners (18,19). The training also included modules on health coaching skills. While health coaching is recognized primarily as a method to improve chronic disease outcomes, we adapted its core skills to equip learners with basic skills to establish a collaborative relationship with contacts and cases, to assess and build on contact knowledge of COVID-19 prevention strategies, and to explore and address ambivalence around recommendations such as quarantine and isolation.
Implementing "hands-on" skill development labs for up to 2,500 learners a week presented logistic challenges and required a large workforce of more than 100 facilitators to ensure that breakout groups were small enough to allow effective skill development. Nonetheless, the skill development labs afforded opportunities for practice and feedback necessary to the development of independent, high-performing contact tracers and case investigators. Further research is ongoing to understand better the extent to which these practical sessions affected learners' confidence and competency and therefore improved public health.
Thirdly, the description of California's contact tracing training program also highlights the importance of ensuring that the COVID-19 CI and CT workforce appreciate diversity, inclusion, and cultural humility as central tenets of the public health response. Substantial research has already documented the way that COVID-19 has disproportionately affected specific ethnic and racial communities in the U.S. (20)(21)(22). The profound deleterious impact of structural racism on the COVID-19 public health response is also well described (23,24). In concert with strategies to increase COVID-19 CI and CT workforce diversity, optimize minority community engagement in CI and CT efforts and target the social determinants of health (25), we assert that the training program highlighted here can play an important role in addressing the legacy of health inequities in the U.S.

Further Extensions and Expansions of the VTA
Beyond CI and CT training, the VTA team is now leveraging the training expertise and the educational infrastructure to respond to other COVID-19 public health training needs and to enhance the existing content in response to the evolving epidemic in California. For example, to address an increasing demand for outbreak investigators in local health jurisdictions, the VTA team developed and implemented an outbreak management course. This was piloted for 38 learners on August 25-27, 2020, and was subsequently incorporated into the standard VTA course options. Given the huge impact that COVID-19 is having in Latinx communities in California, a weekly optional skill development lab in Spanish, with Spanish-speaking facilitators and translated materials, was added in early August 2020. This lab equips CIs and CTs to more adeptly reach the Latinx community. Content on respecting context and cultural humility was expanded to encourage learners to explore implicit bias and how it might affect the CI and CT interview. Learners were encouraged to examine their own beliefs, cultural identities and values in order to recognize and challenge power imbalances and better address the needs of cases and contacts. In addition, to support continuous learning in the ever-changing world of COVID-19, the VTA has created several "communities of practice" for groups of CIs and CTs in local health jurisdictions, as well as a group for state redirected employees to meet online weekly to create community and share experiences, updates and best practices. Finally, the VTA developed school-specific initiatives to support schools' reopening, including technical assistance to local health jurisdictions and risk communication training.

ACKNOWLEDGMENT OF CONSTRAINTS
A limitation of the existing training program is the limited breadth of topics covered within the training curriculum. Notably, training to date has not included content on mental health and/or the importance of trauma-informed care. As the epidemic in California has unfolded, it has become increasingly apparent that knowledge of these domains is critical for CIs and CTs and thus they need to be incorporated into the curriculum. Furthermore, the VTA offered a general training in CI and CT and did not offer specific training on the standard operating procedures of each local health jurisdiction. Therefore, local health jurisdictions needed to offer specific training and onboarding for their CIs and CTs. Finally, as more was learned about COVID-19 and VTA trainings were updated with current information, there was not an easy way to communicate these updates to earlier cohorts. Again, this fell to local health jurisdictions to keep their local workforces updated. Some of these issues may be addressed through ongoing inservice training opportunities, combined with the development of local communities of practice for CIs and CTs.

CONCLUSIONS
CI and CT have been used for multiple diseases to contain outbreaks (7)(8)(9)(10)(11) and are now at the core of COVID-19 containment activities. A robust and resilient response to COVID-19 depends on local and state public health departments having the capacity to develop and maintain CI and CT capability. Time is of the essence given the need for workforce training. The lessons from California can inform how other states respond. The VTA curriculum was designed to equip CIs and CTs with essential skills to provide culturally sensitive care by incorporating interactive learning strategies focused on cultural humility, rapport-building and health coaching techniques (15,16).

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Files, further inquiries can be directed to the corresponding author/s.