{'fraction_non_alphanumeric': 0.03187342352671406, 'fraction_numerical': 0.01597492929756172, 'mean_word_length': 5.0799256505576205, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Time of clerkship rotations’ interruption during COVID-19 and differences on Progress Test’s scores

SUMMARY OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student’s scores as the dependent variable. RESULTS: The comparisons of the students’ scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools’ average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students’ scores (p<0.001) and 0.9% of the 6–year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students’ scores cannot be attributed to the time when the schools paused the clerkship rotations.


INTRODUCTION
The COVID-19 pandemic has led many medical schools to interrupt their activities to keep their students safe and minimize the spread of infection 1 . For medical schools, this decision was more challenging because the students, especially those in the final years of their undergraduate training, need to be trained in real scenarios. However, many just-graduated physicians would eventually look after patients with COVID-19 without adequate training 2,3 . In the beginning of the pandemic, less was known about the clinical and epidemiologic characteristics of the disease and, therefore, most of the schools and universities stopped giving classroom lectures and initiated remote teaching. Conventional lectures were replaced by synchronous or asynchronous online classes, and clinical teaching was replaced by case-based online discussions. Upon gaining some understanding about the causes and methods of prevention of the disease, some schools and universities decided to reopen, provided that social distancing and personal protective equipment were adopted 4 . The decision for reopening was not unanimous across countries and even within the same geographical region 5 . In Brazil, the lack of national guidelines for health and sanitary measures led the states and cities' leaders to decide the most appropriate time to reopen scholar activities. Accordingly, the universities resumed their presential classes at different times 6,7 .
The effects of online teaching on the students' knowledge are yet to be properly evaluated. Although several studies show that faculty members and students have adapted to the new situation, the literature is scarce on objective demonstration of equivalence between traditional and online teaching for clinical training [8][9][10] . In a previous study, we showed that knowledge acquisition was impaired among final-years medical students during the pandemic. A possible explanation is that even after resuming clerkship, the students had less exposure to real clinical scenarios due to the closure of outpatient clinics and reduced variety of diseases in the wards (that were allocated to COVID-19 patients) Considering that different Brazilian medical schools resumed their clerkship rotations at different times and that clinical teaching is pivotal for knowledge acquisition, we aimed to investigate whether the time to resume clerkship rotations was associated with the performance of the students on a Progress Test.

METHODS
An annual Progress Test was conducted at nine medical schools of Brazil for all enrolled students. In this cross-sectional study, we included information of the 5-and 6-year medical students (clerkship years). We analyzed the time of clinical training interruption as the independent variable and the students' scores as the dependent variable.

Ethical considerations
Since we dealt with secondary data and no student was identified, ethics committee approval was not necessary, according to Brazilian legislation. The Progress Test consists of 120 multiple-choice items equally divided into 6 content areas: basic sciences, internal medicine, pediatrics, surgery, obstetrics and gynecology, and public health. The items are clinical vignette-based aiming for applied knowledge rather than knowledge recall. In 2020, the exam was administered in November and was computerized with a safe exam browser instead of the traditional paper-based question book. The test has only formative purposes (i.e., the student's score does not have high-stake implications for progression in the educational program) 11 .

Data analysis
The scores of all clerkship rotations were eligible to enter in the analysis. However, tests with less than 30 correct answers (casual mark) were excluded. The data of students who did not apply for the test were also excluded.
To compare the performances between the schools, the Kruskal-Wallis test was used. It was followed by Dunn's test for post-hoc analysis.
After verifying whether there was a difference in scores between the schools, we analyzed the correlation between the time of clinical interruption and the schools' average scores and the students' individual scores. For parametric and nonparametric data, the Pearson's and Spearman's correlation were used, respectively.
In the second step, we analyzed if a linear model could explain the relationship between the variables. The correlation between the residuals was tested with the Durbin-Watson test and acceptable values should range between 1.5 and 2.5. In the final step, we analyzed which equation model could better explain the relationship and provide the coefficient of determination, i.e., how much of the time of interruption would explain the students' scores. We set the statistical significance level at a p-value of 0.05.
The statistical analyses were performed using the IBM Statistical Package for the Social Sciences (SPSS), version 24.0 (IBM SPSS Statistics for MacBook, IBM Corp., Armonk, NY, USA).

RESULTS
Scores of 1,470 students (728 and 742 from the fifth and sixth years, respectively) were included in the study. The mean time of clerkship interruption was 141.6±52.0 and 111.4±49.1 days for the fifth and sixth years, respectively. Table 1 summarizes the number of students and the time of interruption for each school.
The comparisons of the students' scores between the schools showed that there are differences for both clerkship years (i.e., fifth and sixth years), with p<0.001 (Figure 1). Post-hoc analysis showed significant differences for some pairwise comparisons.
The correlation between the schools' average scores and the time to resume clerkship rotations was not significant both fifth year (r= -0.298, p=0.436) and sixth year (r= -0.440, p=0.240).
The correlation between the students' score and the time to resume clerkship rotations was significant for the fifth year (r= -0.082, p=0.026), though not for the sixth year (r= -0.064, p=0.084). A linear model showed R 2 values of 0.8 and 0.5% for the fifth and sixth years, respectively. However, it was unable to explain the relationships between the variables (score and time) because of the lack of homoscedasticity (constant variance of errors). Durbin-Watson values were 0.268 and 0.223 for the fifth and sixth years, respectively.
A curve estimation analysis showed a better fit of model using a cubic regression (higher R 2 values and lower p values) for both the clerkship years. The model summary and parameters estimates are presented in Table 2. The R 2 values for the fifth and sixth years are 3.4 and 0.9%, respectively, with nonsignificant value for the sixth year. The parameters estimates are too low, demonstrating that the dependent variable (students' scores) is poorly explained by the independent variable (time). For both the years, a cubic regression has the better fit. The equation is y=k+b 1 *x+b 2 *x 2 +b 3 *x 3 , where y: score (dependent variable); k: constant (intercept); x: time (independent variable); and (b 1 *x+b 2 *x 2 +b 3 *x 3 ): slope.

Model summary Parameter estimates
Year

DISCUSSION
The COVID-19 pandemic has changed several aspects of human life, especially the social interactions. Health and education were affected and, due to the unprecedented situation, some decisions were difficult to make, such as the interruption and resumption of clinical and educational activities of medical undergraduates 11,12 .
The effects of the pandemic on the medical students' learning are yet to be understood properly. Initial observations concluded that the transition from face-to-face to remote activities did not decrease the students' knowledge. An American survey with 19 students enrolled in a third-year surgical clerkship during COVID-19 concluded that scores of students did not change in comparison with those who studied in the year before the outbreak of the pandemic 13 . However, this sample size is too small to provide unequivocal and generalizable conclusions. Another American study, with 335 first-year students, also showed in differences on the students' scores before and during the COVID-19 pandemic 14 .
An Indonesian cross-sectional study with 270 fourth-and fifth-year medical students in surgery clerkship found no differences between the multiple-choice questions examinations conducted before and during the pandemic. However, in the objective-structured clinical examination, the students evaluated during the pandemic achieved higher scores than those evaluated before the pandemic 15 .
Importantly, none of the aforementioned studies described equating methods to compare different tests administered to different students reliably. Equating is an important tool to guarantee a reliable comparison 16,17 . By linking and equating different tests, we have previously demonstrated that the clerkship students' knowledge acquisition was impaired during the pandemic [unpublished data].
It is possible that pre-clerkship and clerkship students have been affected differently by the pandemic and the transition to remote teaching 18 . Much of the educational material developed during the pandemic was helpful and replaced some traditional face-to-face activities, such as lectures and tutorial sessions, quite well 19,20 . In some instance, the pandemic induced a positive effect on developing such material.
However, the scenario may be a little different for clerkship students who need direct contact with patients and scenarios to fully develop their professional competence with knowledge, skills, and positive and empathic attitudes toward the patients 21 . Therefore, the clerkship students were the first to resume their educational programs (even in the new normal context). Even so, the decisions on the most appropriate time to resume the clerkship rotations were different across many schools.
We hypothesized that the students who were away from clerkship rotations for a longer period would have lower scores on the test. Our findings do not support this hypothesis. Since there are differences in the median scores of the schools, we believe that other factors such as the nature and quality of how remote teaching was employed play more important roles in determining the students' knowledge scores. Besides, we cannot discard the possibility of previous differences of the students' scores across the schools before the pandemic.
A limitation of this study is that we are unable to provide information regarding the direct impact of the social distancing and remote teaching for each school individually, as no comparison with the pre-pandemic scores was done.
Another concern to be discussed is the possible interference of the computer-based exam on students' performance, which could have changed the difficulty of the exam. However, there are several previous studies demonstrating that there is no difference on students score by using either computer-or paper-based exams, including for Progress Test 22,23 . Besides, as our Progress Test has no high-stake implications, cheating behavior is minimal and, if present, might be homogeneous across the schools.

CONCLUSIONS
We found that the students' scores in the Progress Test were different across the schools. However, these differences cannot be attributed to the time during which the students were kept away from the clerkship rotations.