{'fraction_non_alphanumeric': 0.03130154051303646, 'fraction_numerical': 0.018563357546408393, 'mean_word_length': 5.085415332052103, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Impacts of Remote Learning in Secondary Education: Evidence from Brazil during the Pandemic


 Background: The transition to remote learning in the context of Covid-19 could lead to dramatic setbacks for school enrollment and learning outcomes, especially in developing countries – where a multiplicity of challenges, from limited connectivity to little support from parents, are bound to limit its effectiveness. To date, however, no study has rigorously documented the educational impacts of remote learning relative to in-person classes within primary and secondary education. Quantifying the extent of those losses, as well as the extent to which resuming in-person classes in the pandemic could at least partially offset them, is urgent, as governments worldwide struggle evaluating the trade-offs between the health and educational risks of reopening schools, with vaccination rates still dragging.Methods: Taking advantage of the fact that São Paulo featured in-person classes for the lion’s share of the first school quarter of 2020, but not thereafter, we estimate the effects of remote learning on secondary education, using a differences-in-differences strategy that contrasts variation in dropout risk and standardized test scores between the first and the last school quarters in 2020 to that in 2019, when all classes were in-person. We estimate heterogeneous effects by grade, student characteristics and school characteristics. We also estimate intention-to-treat (ITT) effects of reopening schools in the pandemic through a differences-in-differences strategy, contrasting differences between middle- and high-school students within municipalities that authorized in-person classes to partially return for the latter over the last quarter of 2020, to those within municipalities that did not.Findings: Dropout risk increased by 365% under remote learning. While risk increased with local disease activity, most of it can be attributed directly to the absence of in-person classes:we estimate that dropout risk increased by no less than 247% across the State, even at the low end of the distribution of per capita Covid-19 cases. Average standardized test scores decreased by 0.32 standard deviation, as if students had only learned 27.5% of the in-person equivalent under remote learning. Learning losses did not systematically increase with local disease activity, attesting that they are in fact the outcome of remote learning, rather than a consequence of other health or economic impacts of Covid-19. Authorizing schools to partially reopen for in-person classes increased high-school students’ test scores by 20% relative to the control group.Interpretation: Results show that the societal costs of keeping schools closed in the pandemic are very large. While the learning losses that we document are at least as large as those documented in developed countries on the aftermath of the first Covid-19 wave, the dramatic surge in dropout risk is unique to developing countries. Such massive impacts are likely to bring about long-lasting effects on employment, productivity, and poverty levels. Our findings highlight that reopening schools under safe protocols can prevent those costs from growing even larger. They also caution against recent enthusiasm for remote learning in primary and secondary education outside the context of Covid-19.Funding: Research funded by the Inter-American Development Bank (IADB) as part of a partnership between IADB and the São Paulo State Education Secretariat.


Introduction
In the decades before the Covid-19 pandemic, developing countries were making strides towards universal basic education: by 2019, enrollment rates for primary education had reached over 90% in Latin America and over 75% in Sub-Saharan Africa. (1 ) Having said that, UNESCO and other international organizations described the global education outlook as a 'learning crisis' (2 ): in developing countries like Brazil, the setting of our study, even though most children are now in school, over half of 10-year-olds still cannot read age-appropriate texts (3 ), and 70% finish high school without minimum math and language skills (4 ). School closures in the context of the Covid-19 pandemic are expected to not only detrimentally affect such already fragile learning outcomes, but also, to upset recent progress in enrollment rates. (5 , 6 ) Despite widespread efforts to transition from in-person classes onto remote learning(7 -10 ), a multitude of factors combine to make the latter presumably much less effective in developing countries -limited internet access, lack of dedicate spaces to study at home, and little support from parents who often have not attended school for as long as their children have -above and beyond additional detrimental factors in the context of the pandemic, from demand for child labor to violence against children in a context of psychological distress (11 -14 ). With over 1.6 billion children left without in-person classes for a prolonged period of time, international organizations estimate that at least 7 million additional students will no longer be in school when in-person classes return (15 ).
Quantifying learning losses due to remote learning within primary and secondary education is urgent, as governments need to make informed decisions when trading off the potential health risks of reopening schools in the pandemic (16 ) against its potential educational benefits. Nevertheless, the evidence base for the impacts of remote learning is thin: either focused on post-secondary education (17 -19 ), or on very specific interventions within primary or secondary schools, connecting the best teachers in the country to its most vulnerable students (20 , 21 ) -both with severe limitations when it comes to informing the current debate. Even worse, rigorously documenting the impacts of remote learning is hard. In-person classes have not yet returned in many settings, especially in developing countries where vaccination is dragging. (22 ) Many school systems did not even conduct assessments during the period of remote classes. (23 ) Even in the few that did, inferring the effects of remote learning from differences between average test scores at in-person and remote exams would conflate differences between assessments, or differences in the profile of students whose scores are reflected by each of them. Last, even if those differences could be parsed out, schools tend to close when disease activity is higher; as such, separately identifying the impacts of remote learning from those of the health or economic effects of Covid-19 is challenging. This paper leverages unique data and two natural experiments in São Paulo State during the pandemic to provide first-hand evidence on the educational impacts of remote learning in secondary education in developing countries.

Research Design
São Paulo State provides a unique opportunity to study this question for two reasons. First, the State Secretariat of Education, responsible for middle-and high-school students in the State, conducted quarterly standardized tests (Avaliações de Aprendizagem em Processo, AAPs) throughout 2020, in the same scale as in the years before the pandemic. Supplementary Materials show that while teachers tried to manipulate scorecard grades in 2020 (presumably in an attempt to prevent massive grade repetition), the same did not happen for the standardized tests. Such tests are not mandatory (although heavily promoted by the State, with a take-up no lower than 80% even during school closures).
Second, State schools transitioned to remote learning only at the very end of the first school quarter, when basically there were only exams left to be taken before the start of the second quarter. This provides a natural experiment: in São Paulo, classes were in-person only in the first quarter (Q1), and remote thereafter (Q2-Q4), but exams were remote across all school quarters. Remote exams could be taken either online, through a zero-rating proprietary platform powered by the Secretariat of Education, or picked up and handed back in person at the school gate.
As such, we estimate the impacts of remote learning through a differences-in-differences strategy, contrasting variation in dropout risk and standardized test scores between Q1 and Q4 in 2020 relative to that in 2019, when all classes were in-person. Computing within-year variation not only absorbs teacher effects, but also, holds exam characteristics constant -as all exams were remote in 2020, while all exams were in-person in 2019. We also present results of naive comparisons between Q4-2020 and Q4-2019, and of differencesin-differences analyses contrasting variation in dropout risk and standardized test scores between Q4-2019 and Q4-2020 relative to that between Q4-2018 and Q4-2019 -both of which conflate the effects of other changes in 2020. All estimates are Ordinary Least Squares regressions (OLS), absorbing grade fixed effects. We cluster standard errors at the school level, allowing random shocks to the outcomes of interest to be arbitrarily correlated within schools.
While there were other changes in standardized tests between 2019 and 2020 -in particular, the simplified curriculum recommended for Brazilian schools during the pandemic (24 ) was reflected in 2020 standardized tests(25 ) -, most importantly, such changes were not differential across school quarters: the Q1-2020 AAP already reflected the simplified curriculum, benefiting from re-planning efforts that happened early on, as the state of the pandemic worsened in the country.
We refine our differences-in-differences strategy for the effects of remote learning on test scores by matching student characteristics across years, to parse out the effects of selection. We estimate a propensity score within each grade and quarter, based on student characteristics (see Supplementary Materials), and control flexibly for it (with a cubic polynomial) to parse out selection. We also use these propensity scores to re-weight observations (by the inverse of their selection probability) to ensure that treatment effects on standardized test scores reflect those on the universe of students in each grade. We allow our estimates to vary non-parametrically with municipal-level per capita Covid-19 cases over that period, to gauge the magnitude of treatment effects in the absence of other effects of disease activity on learning outcomes.
We also estimate heterogeneous treatment effects of remote learning by students' gender and race (coded in the Secretariat of Education's administrative data), by the average per capita income of the neighborhood where the school is located (according to the 2010 Demographic Census), and by whether the school offered online academic activities even before the pandemic (according to the 2019 Educational Census).
Last, we estimate the educational impacts of resuming in-person classes in the pandemic, taking advantage of a second natural experiment in São Paulo State: over Q4-2020, some municipalities allowed in-person optional activities to return for middle-school students (psycho-social support and remedial activities for students lagging behind), and inperson classes to return for high-school students (16 ). Since in-person classes only returned for the latter, we implement a triple-differences strategy, contrasting differences between middle-and high-school students within municipalities that authorized schools to reopen to those within municipalities that did not, before and after school reopening. Because we do not have data on which schools actually reopened, we estimate intention-to-treat effects (ITT) based on municipalities' authorization decrees, through Ordinary Least Squares regressions. Importantly, since neither Covid-19 cases nor deaths varied systematically across municipalities that reopened schools and those that did not (16 ), this strategy further validates our previous estimates of the causal educational impacts of remote learning in secondary education.

Data and Definition of Outcomes
We have access to quarterly data on student attendance, math and Portuguese scorecard grades, and standardized test scores, for the universe of 6th to 12th graders in São Paulo State between 2018 and 2020. Restricting attention to 2019 and 2020, our data comprises 4,719,696 observations for middle-school students and 3,791,024 for high-school students. Because of selection into exams each year, we have data on standardized tests scores for 83.3% of observations.
Tracking student dropouts in the pandemic is challenging: many education secretariats in Brazil automatically re-enrolled students at the beginning of 2021(16 ); as a leading example, while middle-and high-school dropouts in São Paulo State average 10% in a typical year, in 2021, dropouts were officially 0%. Instead, we focus on dropout risk, which presumably looms very high regardless of the official enrollment status. We define high dropout risk equal to 1 if a student had no math and no Portuguese grades on record in that school quarter, and 0 otherwise; this proxy reliably predicts actual aggregate dropouts in São Paulo State (see Supplementary Materials). When it comes to standardized test scores, we average math and Portuguese scores within each quarter (for Q4-2020, only overall standardized test scores are available). Table 1 presents different estimates of the effects of remote learning on high dropout risk (in p.p.; Panel A) and standardized test scores (in s.d.; Panel B). The first two columns present naive comparisons, as a benchmark: Column (1) contrasts the last school quarters of 2019 and 2020; and Column (2) contrasts variation between the last school quarters of 2019 and 2020 to its 2018-2019 counterpart. Next, Columns (3-5) present our differencesin-differences estimates; Column (4) controls flexibly for the propensity score to parse out selection effects; and Column (5) further re-weights observation by the inverse of their selection probability. Column (3)(4)(5) also parse out the effects of school reopening over Q4-2020.  (3) to (5), we estimate difference-in-differences comparing variation in outcomes between Q1-and Q4-2020 to that between Q1-and Q4-2019. In Panel A, the dependent variable is high dropout risk (= 1 if the student had no math or Portuguese grades on record for that school quarter, and 0 otherwise). In Panel B, the dependent variable is scores from quarterly standardized tests (AAPs), averaging math and Portuguese scores for that school quarter. All columns include grade fixed-effects and an indicator variable equal to 1 for municipalities that authorized schools to reopen from Sep-2020 onward, and 0 otherwise (allowing its effects to vary at Q4). In Columns (4) and (5) The table shows that, by all accounts, dropout risk has increased dramatically during remote learning, by roughly 365% (significant at the 1% level; Columns 3-5). In turn, when it comes to learning losses, naive comparisons across years would lead one to incorrectly conclude that test scores have actually increased during remote learning (Columns 1-2). The differences-in-differences strategy, however, uncovers dramatic losses, of 0.32 s.d. (significant at the 1% level; Column 5), a setback equivalent to 72.5% of the in-person learning equivalent. Figure 1 displays heterogeneous treatment effects by grade (estimates based on Table  1, Column 5). Panel A showcases effect sizes on dropout risk, normalized with respect to its Q4-2019 average within each grade, and Panel B, those on standardized test scores, normalized with respect to the average difference between Q1 and Q4-2019 within each grade. The figure shows that, except for high-school seniors, dropout risk increased by at least 300% across all grades (Panel A). In Panel B, learning losses are also homogeneously distributed: 60% or higher across all grades, with no distinctive differences between middleand high-school grades. grades Notes: The figure displays heterogeneous treatment effects of remote learning on educational outcomes, by grade. Each effect size stands for a differences-in-differences estimate from grade-specific OLS regressions.

Effects of Remote Learning
All regressions follow the specification in Column (5) of Table 1, only restricting observations to each grade.
In Panel A, the dependent variable is high dropout risk (= 1 if the student had no math or Portuguese grades on record for that school quarter, and 0 otherwise). In Panel B, the dependent variable is scores from quarterly standardized tests (AAPs), averaging math and Portuguese scores for that school quarter.
We normalize each effect sizes by its baseline mean, to express them in % changes: in Panel A, we divide estimates by the variation in the % of students with dropout risk=1b e t w e e nQ 1 -a n dQ4-2019 within each grade; in Panel B, we divide estimates by variation in standardized test scores between Q1-and Q4-2019 within each grade. All columns include an indicator variable equal to 1 for municipalities that authorized schools to reopen from Sep-2020 onward, and 0 otherwise (allowing its effects to vary at Q4), a third-degree polynomial of propensity scores, and re-weight observations by the inverse of their propensity score. Confidence intervals based on standard errors clustered at the school level.
Supplementary Materials showcase that the negative effects of remote learning are significantly concentrated on girls and non-white students, and on schools located in poorer neighborhoods and those that did not offer online academic activities prior to the pandemic. Last, Supplementary Materials also show that allowing treatment effects to vary with local disease activity barely changes our conclusions. To study heterogeneity, we first residualize variation in educational outcomes and in Covid-19 cases with respect to student and school characteristics (allowed to influence learning outcomes differentially in Q4). For Covid-19 cases, this approximates quasi-random variation in disease activity within the State, conditional on all characteristics that we observe (26 ). Then, we estimate heterogeneous treatment effects by flexibly regressing variation in these residualized educational outcomes across Q1 and Q4 on residualized municipal-level variation in per capita Covid-19 cases over that period, with local polynomial regressions. We find that learning losses did not systematically increase with local disease activity over that period. In turn, while dropout risk does seem to have increased with local disease activity (despite no statistically significant patterns), effect sizes were very large even at the low end of distribution: we estimate that dropout risk during remote learning increased by no less than 247% in the State. Table 2 presents intention-to-treat (ITT) estimates of the effects of resuming in-person classes on student attendance (in p.p.; Column 1), standardized test scores (in s.d.; Column 2) and dropout risk (in p.p.; Column 3). The table contrasts municipalities that authorized schools to reopen to those that did not, before and after school reopening, restricting attention to middle-school students in Panel A, and to high-school students in Panel B. Panel C further contrasts those differences, as in-person classes only returned for the latter. . Quarterly data on attendance reflects online or in-person attendance and/or assignment completion (handed in online or in-person) over each quarter (in p.p.), averaged across math and Portuguese classes; standardized test scores from quarterly standardized tests (AAPs), averaging math and Portuguese scores for that school quarter; and high dropout risk = 1 if the student had no math or Portuguese grades on record for that school quarter, and 0 otherwise. In Panels A and B, we estimate treatment effects through difference-in-differences, contrasting variation in outcomes between Q1-and Q4-2020 within municipalities that authorized schools to reopen and those that did not. Panel A restricts attention to middle-school students, and Panel B, to high-school students. Panel C estimates treatment effects through a tripledifferences estimator, which contrasts the differences-in-differences estimates for middle-and high-school students (for whom in-person classes could return within municipalities that authorized schools to reopen in Q4-2020). Column (2) controls for a third-degree polynomial of propensity scores, and re-weights observations by the inverse of their propensity score. All columns are OLS regressions, with standard errors clustered at the municipality level. * if p<0.1, ** p<0.05 and *** if p<0.01.

Effects of Resuming In-person Classes
The table shows that school reopening increased student attendance by the same extent for middle-and high-school students. Effect sizes are small (Column 1), since attendance in 2020 captured a combination of in-person attendance, online attendance, and assignment completion (through an app or handed in at the school gate) -averaging over 90% across all school quarters. In face of that, the fact that school reopening increases attendance at all is testament that reopening indeed allowed some students to return to in-person activities. Most importantly, treatment effects on learning are fully driven by high-school students: in municipalities that authorized high-school classes to return from Nov-2020 onward, test scores increased on average by 0.023 s.d. (significant at the 1% level; Panel C, Column 2), 20% higher than in municipalities that did not. Incidentally, the absence of treatment effects on middle-school test scores rules out differential trends in educational outcomes across municipalities that authorized in-person classes to resume and those that did not.

Discussion
Several studies have tried to approximate the educational impacts of remote learning during the Covid-19 pandemic. (5 , 10 , 27 -31 ) Most of those studies use natural variation in school recess across different geographical units or that induced by previous epidemics (32 , 33 ). Such variation is not a good approximation for the impacts of remote learning: schools were not in recess over most of 2020. Most countries quickly transitioned academic instruction onto remote classes. In São Paulo State, over 90% of students stated that they studied at least 2 hours a day during the pandemic (34 ). Recent evidence from the Netherlands leveraging variation from the first Covid-19 wave suggests that learning losses during remote learning might be very large (27 ). In developing countries, however, such costs might be even larger, as dropout risk might increase dramatically without in-person classes (35 ). We provide first-hand evidence that remote learning in secondary education threatens to not only impose severe learning losses for students who remain in school by the time in-person classes return -a nearly 75% setback relative to in-person classes -, but also, to increase student dropouts dramatically, by a factor of 2.5 or more, reversing decades of efforts to ensure nearly universal basic education in those countries. São Paulo State's response to the pandemic was typical: it closed schools from March 16, 2020, and did not reopen them until September that year. Remote learning strategies were rolled out from April onwards, heavily based on broadcasting content on open television. The State's educational response in the pandemic was rated close to the median quality in the country. (34 ) As such, we expect these findings to generalize to other developing country settings. Besides its impacts on learning outcomes, remote learning is also expected to affect a multiplicity of other child development indicators, including their psychological well-being. (36 , 37 ) Recent evidence for Brazil indicates that learning losses through 2020 were dramatic (38 ). Nevertheless, that study focused only on test scores, and did not decompose learning losses into those caused by remote learning and those caused by other changes in 2020, including other direct effects of Covid-19 -such as its economic and health consequences over the course of the pandemic. These other effects are presumably large; our findings indicate that increasing per capita Covid-19 cases from the low to the high end of the State distribution would nearly double the surge in dropout risk during school closures. Moreover, while the impacts of remote learning on learning losses did not vary systematically with local disease activity, recent studies show that students might present difficulties in concentration, insomnia or neurological disorders in the aftermath of Covid-19 infections, even 60 days after the diagnosis. (39 -41 ) Without rigorous evidence to quantify the contribution of remote learning to those educational impacts, decisions about what to do about in-person classes in the pandemic have been largely influenced by the potential health costs of reopening schools, without weighing those against the educational costs of keeping them closed. (42 , 43 ) In effect, other economic activities for which it is easier to quantify losses from closures -such as shopping centers, bars and restaurants -have reopened systematically earlier than schools (and for longer) in developing countries. (44 ) Our results for the effects of resuming in-person classes also contribute to this important debate. While it has been questioned whether in-person classes would contribute to learning at all in such a complex scenario, we causally show that they do: average test scores deteriorated by nearly 3/4 in municipalities that did not reopen schools, while by only 2/3 in those that did -even if only partially, and for a rather short period of time. While reopening schools only for optional in-person activities was not enough to prevent learning losses (as we find no improvements in learning outcomes of middle-school students, relative to the control group), in-persons classes were. Having said, neither were enough to mitigate the dramatic effects of school closures on dropout risk.
As such, the public debate should move from whether schools should be open or not to how to reopen them safely (45 ): whether to prioritize school staff in the vaccination schedule, what is the appropriate but feasible personal protection equipment in place, what safe capacity limits should be, what changes in school infrastructure are required (e.g. to ensure appropriate ventilation), and how to adapt public transportation to mitigate contagion risks on the way to school -the very same discussions that have been part of the public debate about shopping centers, bars and restaurants since the onset of the pandemic.
Outside the context of the pandemic, many countries (from Brazil to the US) have shown recent enthusiasm for online classes in primary and secondary education, in particular when it comes to the homeschooling debate (46 , 47 ). The evidence base for the impacts of remote learning is thin even with respect to post-secondary education -with mixed causal evidence for its impacts on learning outcomes relative to in-person classes (17 , 18 ), on the one hand, and a more positive outlook for its effects on the set of individuals who pursue further education (19 ), on the other. For primary and secondary education, in turn, our study provides the first rigorous body of evidence on the impacts of remote learning to inform that debate. Related studies(20 , 21 ) that evaluate interventions connecting top teachers in the country to students with the help of technology do not reflect how remote learning is typically implemented; in effect, they find sizeable learning gains from those interventions, in sharp contrast to the dramatic learning losses that we document.
Although we do not have data on private schools -who serve only 20% of primary and secondary students in Brazil, but tend to be better equipped and to serve wealthier families relative to public schools -, our finding that even schools located in relatively wealthier neighborhoods and better equipped to offer online academic activities suffer enormous learning losses under remote learning suggests that our conclusions might extend to a broader student population. Having said that, further research is needed to better understand heterogeneity in the impacts of remote learning, especially by parental education, which we do not observe in our data.