{'fraction_non_alphanumeric': 0.029428206356812012, 'fraction_numerical': 0.021482191343235906, 'mean_word_length': 5.065633325260354, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Transition from in-person to telehealth standardized patient encounters: A retrospective analysis of 18 cases

Introduction Due to the COVID-19 pandemic, many pedagogical activities shifted from in-person to virtual modalities, including standardized patient (SP) encounters. While many academic institutions may not have previously been conducting SP encounters virtually, this change provided student pharmacists with new opportunities to learn and practice telehealth techniques, which are important with the continuously evolving healthcare landscape. Methods This retrospective study analyzed pharmacy student performance on and perceptions of SP cases in fall 2019 (in-person) and fall 2020 (telehealth) semesters. The primary objective was to compare average total scores on SP cases from fall 2019 (in-person) and fall 2020 (telehealth) for first-, second-, and third-year pharmacy students (P1, P2, and P3, respectively). Secondary objectives were to compare average scores from case-specific and relationship and communication (R&C) checklists, as well as to describe student perceptions of the pedagogy. Results Total scores on SP cases decreased from in-person to telehealth for the average of all P1 SP cases, but not for P2 or P3 cases. Case-specific scores did not change, but R&C scores decreased significantly across all three years. Student survey data revealed difficulty with communicating effectively and building rapport via telehealth. Conclusions Overall, student performance on in-person vs. telehealth SP encounters was similar, though P1 students performed more poorly on SP cases conducted via telehealth compared to P2 or P3 students. Thus, faculty could consider focusing on in-person encounters for early learners before introducing virtual encounters. This study also highlights the difficulty in establishing rapport when communicating with patients via telehealth.


Introduction
Telehealth can be defined as the use of electronic information and telecommunication technologies to support and promote longdistance clinical healthcare, patient and professional health-related education, public health, and health administration. 1 Due to the COVID-19 pandemic, many services and activities shifted from in-person to virtual modalities including higher education and healthcare services. [2][3][4][5][6][7][8][9] Standardized patients (SPs) in pharmacy education are individuals who portray the role of a patient, caregiver, or healthcare professional when interacting with students, which allows students to advance their communications skills, apply clinical knowledge, practice patient assessment, and perform a variety of other activities that simulate pharmacist encounters. [10][11][12] A literature review by Smithson et al 13 found that most studies reported SP activities as beneficial and provided students with exposure to relevant clinical scenarios where they could safely develop knowledge, clinical skills, and professional attributes required for pharmacy practice. While many academic institutions may not have previously been conducting SP encounters in a virtual environment, the pedagogical shift due to the COVID-19 pandemic provided student pharmacists with new opportunities to learn and practice telehealth techniques.
After the mandatory adjustment to deliver SP encounters via telehealth, it was imperative to reflect on changes to teaching methodologies to discern how this content should be taught and assessed moving forward. Furthermore, the use of telehealth in the larger medical arena is expected to continue, 2,5,7,14 so the continued provision of these educational opportunities could help align pedagogy with the evolving healthcare landscape. Existing literature on telehealth modalities used during or after COVID-19 to deliver SP simulation activities is limited. Telehealth simulations were incorporated into pharmacy programs rapidly out of necessity during this time, which was likely outside the normal processes of curricular change. 6,[15][16][17][18] Further research is needed to determine best strategies to incorporate telehealth into the pharmacy curriculum to prepare students for their future careers.
The purpose of this study was to describe the transition from in-person SP encounters in fall 2019 to telehealth encounters in fall 2020 and to compare student performance and perceptions between the two pedagogies. The hypothesis was that there would be no difference in student performance.

Methods
Between the fall 2019 and fall 2020 semesters, 18 of 20 SP cases were converted from in-person to telehealth modalities including videoconference and telephone calls (Table 1). Two cases assessed students' ability to measure vital signs, and therefore could not be converted to telehealth. The 18 cases included both standalone weekly formative cases and summative end of semester objective structured clinical examinations (OSCEs). OSCEs were composed of two to four individual cases assessing unique student competencies. All cases in this study were previously used in fall 2018. Cases were written and reviewed by faculty to ensure quality content and reviewed and scored by standardized clients. Inter-rater reliability was not established.
To prepare for telehealth encounters, students were provided with learning materials, including an example telehealth video encounter and instructions for utilizing the telehealth software. Videoconference cases occurred in virtual rooms delivered by a simulation software management platform. Students were on campus in simulated hospital, clinic, and community pharmacy practice setting rooms, while SPs were at a remote location during the encounters. Telephone call cases occurred in a similar format with students on campus in a simulated healthcare environment. The content of cases did not differ from fall 2019 to fall 2020, except for the addition of telehealth-focused items on the grading checklist, which assessed the student's ability to address telehealth-related concerns such as recognizing technological barriers to communication, confirming a patient's current physical location in case of emergency, and obtaining a call-back number in case of disconnection. In fall 2020, two to three min were added to the time allotted for all SP cases to accommodate for anticipated technical difficulties.
This retrospective study analyzed student performance on and perceptions of SP encounters completed by first-, second-, and thirdyear pharmacy students (P1, P2, and P3, respectively). For fall 2019 and fall 2020, these students were members of the graduating Classes of 2021 (P3 in fall 2019), 2022 (P2 in fall 2019 and P3 in fall 2020), 2023 (P1 in fall 2019 and P2 in fall 2020), and 2024 (P1 in fall 2020). Assessment checklists were used to assess performance on three domains: knowledge, communication, and overall impression of the student as a future pharmacist. The first domain on knowledge was graded using a case-specific checklist developed by faculty case authors (100 points per case). The second domain of communication was graded using a general pharmacist-specific relationship and communication (R&C) checklist (100 points per case) that is utilized for all SP cases across the curriculum. The total score was calculated based on the weighted average between the case-specific score and the R&C score as illustrated in Table 1. Although the weighting of each component varied throughout the curriculum, all cases were consistent between 2019 and 2020 except for four cases with minor variations in weighting. To ensure equal comparisons between in-person and telehealth cases for these four cases, weighting from 2019 was utilized to calculate the total score for analytical comparisons. Finally, the third domain on overall impression was assessed using the Global Assessment Item (GAI), which was one question with a dichotomous answer of "Yes" or "No" for "Would you feel comfortable having this student as your future pharmacist?" No grade points were assigned for this domain.
The primary outcome was the difference in average total scores on SP cases from fall 2019 to fall 2020 for each class cohort (i.e., P1, P2, and P3). Four cases were included in analysis from the P1 fall semester; hence, the total scores from each case were added and divided by four to calculate the average total score of a P1 student's performance. Seven cases were administered in the P2 fall semester; hence, the total scores from each case were added and divided by seven to calculate the total average score of a P2 student's performance during the P2 year. Similarly, seven cases were administered in the P3 fall semester, and the total score was calculated using the methods described previously. Between the assessment years of fall 2019 and fall 2020, students who were previously P1 students progressed to P2 year, thus their scores contributed to the control arm (in-person) in P1 year and intervention arm (telehealth) in P2 year. To avoid a repeated sample measurement issue, statistical analyses comparing scores between fall 2019 and fall 2020 were restricted to each class cohort. Given the primary outcome, the average total score was captured as a continuous scale of score range from 0 to 100 and a two-samples t-test was used with a priori significance level of <.05. Secondary outcomes included average scores from case-specific and R&C checklists (both continuous variable with possible score range 0-100). Hence, a two-samples t-test was used to compare the scores between the in-person and telehealth group with a priori significance level of <.05. For the percentage of students who received "No" to the GAI in each cohort, Fisher's exact test was utilized to compare the fall 2019 and fall 2020 groups with a priori significance level of <.05. Data were analyzed in Stata, version 17.0 (StataCorp).
Student demographic data was collected from Pharmacy College Application Service applications and included age, gender, ethnicity, race, undergraduate grade point average (GPA) at admission, and completion of prior degrees. Demographic information was compared across the four cohorts (i.e., graduating Classes of 2021, 2022, 2023, and 2024) to provide general information about the demographic characteristics of students included in the study. For the subgroup analysis, Pearson correlation coefficient was used to compare the relationship between the two scale forms of GPA at admission and the average total score per class cohort with a priori significance level of .05.
Lastly, qualitative data from end-of-semester surveys regarding student perceptions were reviewed for overarching themes. Qualitative data included comments from P2 and P3 students on an optional end-of-semester survey in fall 2020 that was conducted through Qualtrics to receive student feedback on the structure of lab that semester. These students were able to provide a unique perspective of the telehealth transition as they completed in-person SP cases previously. The survey included eight (for P2) or nine (for P3) questions, one of which was a free-response question specific to SP encounters. Qualitative data was analyzed using an inductive approach in Microsoft Excel for Microsoft 365 MSO, version 2110 (Microsoft Corp.). Two investigators reviewed the survey results and systematically arranged data into themes using an investigator triangulation approach. 19 The qualitative data was then organized into three categories of negative, positive, and neutral/mixed student responses. To validate this data, two additional investigators were involved to resolve any discrepancies identified at the initial survey response review. This study was approved by the local institutional review board.

Results
Demographic characteristics among the four cohorts (Classes of 2021, 2022, 2023, and 2024) were comparable in age (P = .68), gender (P = .31), ethnicity (P = .33), previous degrees (P = .77), and GPA at admission (P = .92). There was a statistically significant difference in race composition in that 13.2% of students were reported to be black in the Class of 2021, 21.8% in the Class of 2022, 29.1% in the Class of 2023, and 37.7% in the Class of 2024 (P = .02). Racial diversity in class composition and its relationship between student performance on SP cases is not known; hence, this statistically significant difference in class composition was reported but not adjusted for during analysis.
SP case scores are displayed in Table 2. The total score for the average of all P1 cases demonstrated a statistically significant decrease when transitioning from in-person to telehealth (92.21 ± 5.2 to 89.53 ± 6.73, P = .02). However, the total score did not differ significantly for P2 or P3 cases (P = .07 and P = .05, respectively). When comparing case-specific scores only, there was no difference among P1, P2, or P3 cases (P = .09, P = .32, and P = .84, respectively). For the R&C scores, a statistically significant decrease in performance was seen for all years (P = .002, P = .001, and P < .001, respectively). Within three class cohorts (i.e. P1, P2, and P3) included in fall 2019 and fall 2020, all six groups demonstrated no statistically significant correlation between GPA at admission and total score (six bivariate correlation comparisons resulting in P value > .05).
For the GAI (Table 3), there was not a statistically significant difference in the P1 cohorts between in-person and telehealth for the percentage of students who received feedback of "No" for at least one or at least two of the four cases. For the P2 cohorts there was a significant increase in the number of students who received a "No" for at least one of the seven cases (23.2% vs 43.1%, P = .02), but this did not reach significance when comparing the number of students who received a "No" for at least two cases. The P3 cases also indicated a statistically significant difference in the number of students who received a "No" for at least one case (4.3% vs. 20.7%, P = .004). There were no students in either P3 cohort that received a "No" for two or more cases.
Of 116 second-and third-year students in the fall 2020 semester, 54 (46.6%) completed the end-of-semester survey. Within the survey, 50 (43.1%) answered the question specific to SP cases and 24 (20.7%) provided comments related to the delivery of SP encounters via telehealth; the remainder of responses were excluded due to being unrelated to the telehealth transition. The four themes that emerged were: (1) transitional pedagogy (14 comments), (2) student perceptions of telehealth encounters (21 comments), (3) student-perceived benefit of telehealth encounters (six comments), and (4) technological barriers (six comments). Students mentioned needing more time for telehealth vs. in-person encounters, difficulty with establishing rapport with a patient over the phone, and anxiety related to technological issues. Students also commented that the telehealth encounters provided a beneficial experience that may be more realistic to future practice (Table 4).

Discussion
The effects of the COVID-19 pandemic required a shift in pharmacy education, and this study aimed to analyze a large-scale transition to telehealth patient simulations and its impact on student performance and perceptions. This project is unique in that although it analyzes SP encounters delivered via telehealth due to mandatory curricular shifts related to the COVID-19 pandemic, students were on campus during these activities, which simulates a pharmacist's work environment to communicate with a patient via telehealth. As such, results can be applied in a meaningful way to pedagogy moving forward. Furthermore, this study analyzed 18 cases including eight OSCE cases; to the authors' knowledge, no other studies have included this high number of SP cases, either before or after the COVID-19 pandemic.
This study found that the total score on SP cases decreased from in-person in 2019 to telehealth in 2020 for the average of all P1 SP cases, but it did not significantly change for P2 or P3 cases. While the total score average for P1 cases decreased, the GAI did not. P1 students likely struggled more than P2 and P3 students to establish rapport with SPs as they were new to participating in SP cases and had to do so for the first time via telehealth modalities. This, however, did not significantly change results of the GAI, indicating that SPs still felt comfortable to have their students as future pharmacists based on their performance on any given telehealth case.
There were no statistically significant changes when comparing case-specific scores across all three cohorts. This may provide evidence that the virtual environment did not impact students' ability to apply clinical knowledge and relay specific medicationrelated information to patients. Thus, even with a virtual environment to resemble telehealth, students were still able to demonstrate knowledge about medication and disease state counseling. Similar findings regarding in-person vs. virtual SP encounters have been reported in the literature. 20,21 Furthermore, when GPA at admission was compared to total score, there was no correlation between the two variables, suggesting that the SP assessment is focused on evaluating a communication skill that differs from the application of knowledge typically accounted for in GPA. This may provide further evidence for the support of teaching varied communication techniques, such as those required for telehealth encounters. The R&C score average decreased significantly across all three years. This may be due to a lack of experience building rapport with SPs through telehealth modalities. The addition of telehealth-related items in the R&C grading checklist may have also negatively affected performance. Furthermore, technological issues may have served as barriers to optimal student outcomes. One student reported that it was harder to establish rapport with the patient in telephone encounters since they were not able to make eye contact. Another student expressed feeling that they had to focus on asking the telehealth-related items more so than spending time on the encounter itself. Several students stated that technological barriers took away valuable time from the encounter, caused anxiety, and made it difficult to effectively communicate with SPs. This qualitative data from the student survey supports the R&C results.
The GAI results revealed that the P2 and P3 cohorts, but not the P1 cohort, experienced a statistically significantly higher number of "No" responses to the question "Would you feel comfortable having this student as your future pharmacist?" when comparing results from fall 2019 to fall 2020. This may be related to the overall difficulty of P2 and P3 SP cases compared to P1 cases. P2 and P3 cases are increasingly challenging as they introduce more complex patients and disease states and involve more provider consultations. These advanced cases rely heavily on students' communication and soft skills that may be more difficult to convey in virtual settings.
Qualitative results for the transition from in-person to telehealth cases were mixed. Some students expressed feeling faculty did well with the incorporation of telehealth and were glad to participate in such learning experiences, while others disliked the transition. According to literature published prior to the COVID-19 pandemic, implementation of SP encounters via telehealth had also been met with mixed results. A study by Skoy et al 22 demonstrated that students performed better regarding patient counseling in the face-toface format compared to telehealth. Lempicki et al 23 randomized students to a virtual or face-to-face SP encounter and reported that despite enjoying the virtual encounter and viewing it as a positive learning experience, some students would prefer not to complete such an encounter again and would not recommend it to others.
A limitation of the study was the inability to control for differences in the teaching of related lecture content from 2019 to 2020, and thus the SP-specific transitional pedagogy may not have been the only factor that affected student outcomes. While cases compared in this study were similar, students may have prepared for cases differently based on revisions of course content, changes in delivery of materials and lecturers, or modifications in learning activities provided. When reviewing historical data on these cases, student performance on SP encounters was not necessarily consistent prior to the index year of 2019. While total scores for the four included P1 cases was less than one percentage point different from 2018 to 2019, performance on the seven included P2 cases decreased 5% from 2018 to 2019 and P3 case performance increased 5%. Lastly, one must take into consideration that students were enrolled in a professional program amidst a pandemic, which may have immeasurable effects on their overall performance.

Conclusions
Overall, study results support the hypothesis that student performance would be similar between in-person and telehealth SP encounters, though P1 total scores declined while P2 and P3 total scores remained consistent. Thus, faculty should balance the use of in-person and telehealth encounters and consider focusing on in-person encounters for early learners before adding in virtual encounters. Although there was no impact on case-specific scores, R&C scores demonstrated a small decline in performance, echoing statements from student survey responses. This highlights the difficulty communicating and building rapport via telehealth, which reveals an opportunity for faculty to target their teaching on establishing rapport with patients when communicating via telehealth. Future studies could analyze individual cases in more detail, such as cases that require establishment of rapport with the SP for success in the encounter, as these may be especially challenging in the virtual environment. With ever-changing advancements in healthcare, there is a need to offer opportunities for students to participate in simulated patient activities via telehealth, and educators should feel empowered to utilize telehealth to provide realistic SP encounters. .004 "No" on at least 2 of 7 cases 0% 0% NA NA = not applicable; P1 = first-year pharmacy student; P2 = second-year pharmacy student; P3 = third-year pharmacy student. a The Global Assessment item included is a single subjective question, "Would you feel comfortable having this student as your future pharmacist?", to provide an overall assessment of performance. The responses to this question were not included in the total score percentage. • Standardized clients went great this semester. I was really impressed with the way the faculty and clients handled the transition to online encounters. • …I appreciate the quick adaptations to phone and web. • The added time has helped since we have to acquire some information just in case we lose connection but time seems to go by faster because at moments it is difficult to hear them.
• I do like how some are calls and some video to make it seem appropriate for the actual encounter. • Telehealth SC were much different but I think overall they went well. • The standardized clients went really well especially with the transition onto a virtual session.
• SC were fine as we got to a whole new experience that I think is beneficial to us as students. • As always, standardized client encounters are a great way to learn how to talk with future patients/prescribers about certain topics. This semester having them online in a virtual setting definitely will be beneficial for the future especially since this is a growing part of healthcare.
Not applicable based on theme

Negative Example Statements
• I think there should have been extra time given due to the circumstances. • The virtual experience needs work...
• I completely understand that technology was the reason for the phone calls, but it did make it harder to establish rapport with the patient/caregiver without being able to make eye contact with them.
Not applicable based on theme • The only issue I had was the video quality sometimes. One time I felt like I was going to have to call my client due to lagging video quality… • It was super stressful on the students when technology didn't work and I think it caused more stress than benefit. • When there were tech issues, we weren't informed until we arrived upstairs, which causes anxiety to skyrocket, leading us to make more mistakes. virtually, but I realize it was the best alternative. Overall, I think our standardized client encounters are a great asset to the pharmacy program.
• Overall it was realistic and even with technical difficulties due it prepared us for what may happen in real life since everything is not 100% perfect.
None in addition to above OSCE = objective structured clinical exam; SC = standardized client. a Statements may include multiple themes.