Pandemic Schooling Mode and Student Test Scores: Evidence from US States (NBER Working Paper 29497)

Summary

NBER Working Paper 29497 by Clare Halloran, Rebecca Jack, James C. Okun and Emily Oster, dated November 2021, estimates the impact of district-level schooling mode on test scores. The abstract states that the authors combine Spring 2021 state standardized test score data with data on schooling in the 2020-21 school year across 12 states. It states that pass rates declined compared to prior years and that the declines were larger in districts with less in-person instruction. It reports that passing rates in math declined by 14.2 percentage points on average, a decline the authors estimate was 10.1 percentage points smaller for districts fully in-person.

Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used

Issuer: Source documents
Document type: Https Www The74Million Org Article Roza Without Clear Rules Theres No Way To Judge How Sch
Case: Https Www The74Million Org Article Roza Without Clear Rules Theres No Way To Judge How School Relief Funds Are Being Spe

Cite this document

“Pandemic Schooling Mode and Student Test Scores: Evidence from US States (NBER Working Paper 29497).” Original at www.nber.org. Archived at Pandemic Darlings, https://pandemicdarlings.com/source-documents/https-www-the74million-org-article-roza-without-clear-rules-theres-no-way-to-judge-how-school-relief-funds-are-being-spe/, SHA-256 02824c1714ae38861f58c7ab6f6626aaec6111622220a3a0965c4593c94fda5c.

Full text

NBER WORKING PAPER SERIES

PANDEMIC SCHOOLING MODE AND STUDENT TEST SCORES:
EVIDENCE FROM US STATES

Clare Halloran
Rebecca Jack
James C. Okun
Emily Oster

Working Paper 29497
http://www.nber.org/papers/w29497

NATIONAL BUREAU OF ECONOMIC RESEARCH
1050 Massachusetts Avenue
Cambridge, MA 02138
November 2021

We are grateful to Hannah Orlove for excellent research assistance. The COVID-19 School Data
Hub was produced with funding from Emergent Ventures at the Mercatus Center, the Chan
Zuckerberg Initiative, and Arnold Ventures. Contact: Emily Oster at emily oster@brown.edu.
The views expressed herein are those of the authors and do not necessarily reflect the views of the
National Bureau of Economic Research.

At least one co-author has disclosed additional relationships of potential relevance for this
research. Further information is available online at http://www.nber.org/papers/w29497.ack

NBER working papers are circulated for discussion and comment purposes. They have not been
peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies
official NBER publications.

© 2021 by Clare Halloran, Rebecca Jack, James C. Okun, and Emily Oster. All rights reserved.
Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission
provided that full credit, including © notice, is given to the source.
Pandemic Schooling Mode and Student Test Scores: Evidence from US States
Clare Halloran, Rebecca Jack, James C. Okun, and Emily Oster
NBER Working Paper No. 29497
November 2021
JEL No. I14,I21,I24

ABSTRACT

We estimate the impact of district-level schooling mode (in-person versus hybrid or distance
learning) on test scores. We combine Spring 2021 state standardized test score data with
comprehensive data on schooling in the 2020-21 school year across 12 states. We find that pass
rates declined compared to prior years and that these declines were larger in districts with less in-
person instruction. Passing rates in math declined by 14.2 percentage points on average; we
estimate this decline was 10.1 percentage points smaller for districts fully in-person. Changes in
English language arts scores were smaller, but were significantly larger in districts with larger
populations of students who are Black, Hispanic or eligible for free and reduced price lunch.

Clare Halloran James C. Okun
Brown University MIT
clare.halloran@gmail.com jokun@mit.edu

Rebecca Jack Emily Oster
University of Nebraska at Lincoln Brown University
1400 R St Department of Economics
Lincoln, NE 68588 64 Waterman Street
rebecca.s.jack@gmail.com Providence, RI 02912 and
NBER
emily_oster@brown.edu

A Learning Mode Database is available at https://www.covidschooldatahub.com/
Pandemic Schooling Mode and Student Test Scores: Evidence from
US States ∗

Clare Halloran Rebecca Jack James Okun
Brown University University of Nebraska-Lincoln MIT

Emily Oster
Brown University and NBER

November 14, 2021

Abstract

We estimate the impact of district-level schooling mode (in-person versus hybrid or distance
learning) on test scores. We combine Spring 2021 state standardized test score data with
comprehensive data on schooling in the 2020-21 school year across 12 states. We find that pass
rates declined compared to prior years and that these declines were larger in districts with less
in-person instruction. Passing rates in math declined by 14.2 percentage points on average; we
estimate this decline was 10.1 percentage points smaller for districts fully in-person. Changes in
English language arts scores were smaller, but were significantly larger in districts with larger
populations of students who are Black, Hispanic or eligible for free and reduced price lunch.

1 Introduction

Over the course of the 2020-21 school year, students across the United States experienced educational
disruptions as schools and districts alternated between learning modes in an effort to mitigate the
transmission of Coronavirus Disease 2019 (COVID-19). Uncertainty about the role of schools in
the spread of COVID-19 as well as the impact of different schooling modes forced school leaders to
make difficult decisions about how to appropriately support both their students and staff (McLeod
and Dulsky, 2021). School districts utilized a range of schooling modes (sometimes called ”learning
∗
We are grateful to Hannah Orlove for excellent research assistance. The COVID-19 School Data Hub was
produced with funding from Emergent Ventures at the Mercatus Center, the Chan Zuckerberg Initiative, and Arnold
Ventures. Contact: Emily Oster at emily oster@brown.edu.

1
models”) throughout the year: some remained closed and offered virtual learning options, others
opened for full-time in-person learning, and some combined these approaches through a “hybrid”
schooling mode (Kaufman and Diliberti, 2021). Approaches to both hybrid and virtual learning
varied, including different numbers of days in school and variation in the degree of synchronous
instruction (Marshall, Shannon, and Love, 2020).
There has been some effort to compare overall student learning during the pandemic to pre-
pandemic years, without reference to schooling mode (Lewis et al., 2021). Other research has relied
on parent or teacher perspectives of students outcomes based on reported schooling mode (Chen
et al., 2021; Verlenden et al., 2021). To date, however, little is known about how outcomes vary
for students by school districts’ predominant schooling mode.
In this paper, we use data on district schooling modes alongside Spring 2021 test score data
from 12 states to analyze the relationship between schooling mode and test scores. We estimate the
relationship between the share of the year that in-person learning was offered (as opposed to hybrid
or virtual learning) and student standardized test scores, both overall and interacted with district
demographics. Our main regression specification uses a standard panel data approach, with data at
the district-year level, to estimate the impact of the time spent in-person on 2021 test scores. These
regressions include district-specific trends and a full set of demographics and year interactions.
We demonstrate first that, across all 12 states in our sample, there were large reductions in test
scores between 2020-21 and previous years. The average decline in math is 14.2 percentage points,
versus 6.3 percentage points in English language Arts (ELA). These changes are well outside typical
year-on-year variation.
Following this, we show our main finding: the decline in students’ 2021 test scores as compared
to prior years was significantly larger in districts which offered less access to in-person schooling.
For example, we estimate that offering full in-person instruction rather than fully hybrid or virtual
instruction reduces test score losses in math by 10.1 percentage points (on the base of 14.2
percentage points). In ELA, the loss is reduced in fully in-person settings by 3.2 percentage points.
These interactions between test score losses and schooling mode are highly significant.
We demonstrate, as well, that access to in-person learning varied by demographics and baseline
test scores. Districts with lower baseline test scores and a higher share of Black students offered
less in-person learning. Within states, districts with a higher share of Hispanic students also offered
less in-person learning (although across states this effect is reversed).
Finally, we observe that the impact of access to in-person learning on 2021 test scores also

2
varies by subject and demographics. Specifically, among districts with a larger share of Black and
Hispanic students, districts with less in-person schooling saw a greater decline in ELA test scores
than those with more in-person schooling. Although the impact of schooling mode on ELA is fairly
small for districts which are majority white, it is large for those districts with a majority of students
of color. Meanwhile, the impact of access to in-person learning had a similar effect on math scores
for all districts, regardless of their racial composition.
A key consideration for our results is the role of test participation. Test participation was
lower in the 2021 testing cycle than previous years. This was true for several reasons, one of
which includes changes in federal participation requirements (Gewertz, 2021). According to state
reports, the lost participation seems to disproportionately come from groups with generally lower
test scores1 ; this would suggest our estimates understate test score losses.
To account for this, we use data on participation rates and re-estimate our models with varying
assumptions about the performance of students who do not participate in testing. We find our
results are robust. In our preferred specification, in which we calibrate the relationship between
participation and counterfactual test scores, we find our baseline results underestimate both overall
test score losses and the impact of in-person learning in ameliorating them.
A second concern is that it is not variation in schooling mode but rather other pandemic-related
factors that drive our results, such as changes in parents’ or caregivers’ working conditions due to the
pandemic, changes in access to child care, or changes in access to other services such as tutoring
resources. In general, districts were more likely to offer in-person learning in areas with higher
COVID-19 rates, suggesting the pandemic itself is a confound in the other direction. Nonetheless,
given that the lack of in-person schooling was also likely associated with other lockdown behaviors,
it is challenging to fully separate these effects.
We contribute to a large literature on how students’ time out of school, typically due to summer
break, impacts their academic achievement (McCombs et al., 2011; Alexander et al., 2016; von
Hippel, Workman, and Downey, 2018). In their synthesis of the research on summer learning loss
(SLL), McCombs and colleagues (2011) conclude that while all students experience summer learning
loss, on average, this is particularly true for low-income students, and that losses accumulate over
time. Other research has focused on learning disruptions resulting from long and unplanned school
closures from events such as weather emergencies (Harmey and Moss, 2021).
1. See, e.g., Colorado Department of Education, 2021; Connecticut State Department of Education, 2021; Hanson,
2021; Hilton and Linnane, 2021; Littlehales, 2021; Ohio Department of Education, 2021; Rhode Island Department
of Education, 2021

3
More narrowly, we contribute to the small but growing literature on the impacts of the COVID-
19 schooling disruptions on students (Angrist et al., 2020; Bacher-Hicks et al., 2021; Dee et al.,
2021; Musaddiq et al., 2021). In many cases, these papers highlight the possibility of significant
inequalities in schooling losses during the pandemic (Kuhfeld et al., 2020; Kuhfeld and Tarasawa,
2020; Azevedo et al., 2021; Wyse et al., 2020; Fuchs-Schundeln et al, 2021), although they have
limited data on outcomes. Of note are several papers which use test scores to directly study impacts
of Spring 2020 closures on learning, largely in Europe (Tomasik et al., 2021; Engzell et al., 2021;
Ardington et al., 2021; Contini et al., 2021; Maldonado and De Witte, 2021; Schult and Linder,
2021). Individually and in a systematic review, these papers tend to find significant learning losses
as a result of limited in-person schooling access (Hammerstein et al., 2021; Storey and Zhang,
2021).
Relative to this existing literature, the wide variation in schooling access in the United States
provides both more variation to exploit and a longer period of disrupted schooling.
Overall, our data suggest significant learning losses as measured by state assessments as a result
of the pandemic, and larger losses in districts with less access to in-person learning. From a policy
standpoint, our results highlight the non-health implications of the pandemic, which may be long-
lasting. There is significant funding at both the federal and state levels to address these issues; our
findings suggest the possibility of targeting certain districts and students in these efforts. These
data also highlight the value of in-person learning and may provide a caution when considering
school closures in the future.

2 Data

Our analyses use three sources of data: 1) district-level schooling mode data from the 2020-21
school year; 2) district-level test score data from 2015 to 2021; and (3) demographic data from the
NCES. We explain these data sources below.

2.1 Schooling Mode Data

District-level schooling mode data are drawn from the COVID-19 School Data Hub (CSDH). This
is a public database, produced by our research team, which uses state-sourced data to document
the schooling modes used by school districts during the 2020-2021 school year.
The CSDH sources raw data from State Education Agencies on schooling modes across schools or

4
districts over the 2020-21 school year. 2 The CSDH reviewed all school or district-level information
and classified each school or district time period as: 1) ”in-person” (all or most students had access
to traditional, 5-day-per-week, in-person instruction); 2) ”virtual” (all or most students received
instruction online, five days a week); and 3) ”hybrid” (schooling modes that did not fall into one
of these approaches). More details about the data construction in each state are available at the
CSDH.3
We included states in our analyses if the state had schooling mode available during the 2020-21
school year. When schooling mode was available at the school district level only, we used this mode
in our analyses. In cases where schooling mode was available at the school or district-grade-band
level, we calculate an enrollment-weighted average across schools or grade levels.
Our resulting data file is at the district-time period level. This classification represents our best
attempt at capturing the typical schooling mode for students in a particular district and time period.
However, this does not mean that this was the only way that students learned within a district,
as changes occurred on a daily basis throughout the year at the district, school, and classroom
level due to county-level COVID-19 case rates, school- and district-level quarantine procedures,
and close-contact policies.

2.2 Test Score Data

We draw test score data from state-level testing in spring 2021. We included states in the analysis
if they met the following test score criteria: 1) at least 2 years of pre-pandemic test data; 2)
No significant testing changes over this period, and 3) statewide 2021 participation rates above
50%. We focus on test scores for grades 3 through 8 in English language arts (ELA) and math.
Our primary outcomes are subject-area pass rates, which measure the share of students who score
proficient or above in ELA or math on state assessments. We present each state’s testing details
in Appendix B.
Our final sample includes 12 states: Colorado, Connecticut, Florida, Massachusetts, Minnesota,
Nevada, Ohio, Rhode Island, Virginia, West Virginia, Wisconsin and Wyoming. In several of these
cases, states have produced reports documenting learning losses using these raw data (Colorado
Department of Education, 2021; Connecticut State Department of Education, 2021; Hanson, 2021;
Hilton and Linnane, 2021; Littlehales, 2021; Ohio Department of Education, 2021; Rhode Island
2. The CSDH also contains data on student enrollment by schooling mode, if available from State Education
Agencies. We do not use those data here.
3. https://www.covidschooldatahub.com/

5
Department of Education, 2021).
Of note is the fact that participation rates for tests in most states in spring 2021 were lower
than in previous years. In a robustness analysis we use test participation rates to create bounds.

2.3 Demographic and Pandemic Data

In addition to these primary data sources, we draw information on school demographics from the
National Center for Education Statistics (NCES). These data include information on the share of
enrolled students by race and ethnicity as well as the share of students who are eligible for free
and reduced price lunch (FRPL)-eligible. NCES also provides enrollment data; we use 2019-20
enrollment data as it is the most recent available. To capture the possible role of variation in
COVID-19 case rates in driving district opening decisions, we use district-level COVID-19 case
numbers from USA Facts4 . We focus on the average level of COVID-19 over school year in the
central zip code of the school district, and the rate in August 2020, around school opening.

2.4 Summary Statistics and Determinants of Schooling Mode

Table 1 shows each state’s summary statistics on schooling mode, pass rates, and demographics. Of
the states in our analyses, in-person learning rates are highest in Florida and Wyoming, and lowest
in Minnesota and Virginia. Virginia and Colorado also have the highest share of district-time spent
in fully virtual learning. Across states, pass rates are, on average, lower in 2021 than 2019.
Table 2 estimates the relationship between the share of time a district offered in-person learning
and the district’s pre-pandemic test scores, demographics, and COVID-19 rates. Columns (1) and
(2) estimate these regressions without state fixed effects, and Columns (3) and (4) include state
fixed effects.
Overall, higher baseline test scores appear to be associated with greater in-person learning
shares; this is especially true within states. A greater share of Black students in a district is
associated with less in-person learning, both within and across states. In contrast, conditional on
race shares, a higher share of students with FRPL status correlates with greater in-person learning.
The share of Hispanic students in the district correlates positively with in-person share across
states, but negatively within states; this reflects overall access differences across the states in our
sample (i.e. a high in-person rate in Florida).
4. Available at https://usafacts.org/.

6
Focusing on the within-state results, the overall picture suggests that more vulnerable districts
– lower prior test scores, more minority students – were less likely to have access to in-person
schooling.
On pandemic metrics, we find that both within and across states, higher case rates (on average
or in August) were associated with greater in-person schooling during the year. Much has been
written on the possible role of schools in driving COVID-19 cases, most of which suggests schools
were not significant drivers of COVID-19 (Goldhaber et al., 2021; Harris et al., 2021; UNICEF,
2020). The positive correlation here likely reflects differences in other pandemic restrictions which
were correlated with schooling mode choice and influenced COVID-19 rates.

3 Results

We turn now to using the above data to estimate the impact of the pandemic schooling mode on
student test scores. Our primary treatment is the share of the school year that districts offered
full-time in-person learning options, with the untreated group being a combination of hybrid and
fully virtual districts. Effectively, we can view these analyses as identifying the possible losses
resulting from deviation from a traditional school schedule, characterized by in-person learning.

3.1 Overall Test Score Changes

Figure 1 illustrates the changes in pass rates across states during the pandemic period (the 2020-21
school year) and in several previous periods (the 2015-16 school year through 2018-19). Panel A
shows the scores in math and Panel B presents the scores in ELA. The solid triangles represent the
change in pass rates between the 2018-19 school year (the last pre-pandemic year) and the 2020-21
school year. The open circles show prior year-on-year changes.
Our analysis finds a decline in pass rates from the 2018-19 school year to the 2020-21 school year
in all states. The decline ranges from -31.9 percentage points (Virginia, math) to -2.3 percentage
points (Wyoming, ELA). Based on the prior year changes, typical year-on-year variation is very
small. The pandemic year is a significant outlier.
The changes in pass rates for math (Panel A) are considerably larger than the changes in scores
for ELA (Panel B). This is broadly consistent with a larger literature which shows math scores are
more responsive to schooling differences (Betts and Tang, 2011; Angrist et al., 2013).

7
3.2 Schooling Mode and Test Score Changes

To analyze changes by schooling mode, we divide districts into three groups: districts that offered
in-person learning to students for at least two-thirds of the school year (”Most In-Person”), districts
that offered in-person learning to students between one-third and two-thirds of the year (”Intermediate
In-Person”), and districts that offered in-person learning to students for less than one-third of the
school year (”Least In-Person”). We then analyze year-on-year changes in test scores for each
group, by state.5
Panel A of Figure 2 shows the year-on-year changes in math test scores by group for pre-
pandemic years (left figure) and for the pandemic year (right figure). In most cases, the test
score declines in the pandemic year are larger in the districts with less in-person instruction. This
difference is largest in Virginia. There is no evidence that this pattern aligns with the pre-pandemic
years. Prior year-on-year changes do not seem to systematically vary with 2020-21 schooling mode.
Panel B shows the results for ELA. While the overall changes are smaller in ELA as compared
to math, the pattern is similar.
We note that the states in the graph are organized left-to-right by the state-level share of in-
person instruction. In addition to the within-state variation, there appears to be some relationship
between average test score declines and average in-person share by state. In both subjects, there are
sizable reductions in scores even among areas which have the most in-person instruction. Locations
places with less in-person schooling drop more, but the overall test score reductions go beyond
virtual learning.
These graphs look at changes in scores across years; one interpretation of the larger reductions in
districts with less in-person learning is that they started at higher pass rates, on average. However,
this is inconsistent with the evidence in Table 2, which demonstrates that higher pre-pandemic test
scores are associated with greater in-person learning access.
Next, we estimate standard difference-in-difference regressions at the district-year level, including
district-specific trends and interactions between district demographics and time (equation (1)
below). In addition to the primary regressions, we analyze the relative impact of schooling mode
by demographics (equation (2) below). To do so, we include an interaction of in-person share, a
dummy for 2021 and either (a) the share of Black or Hispanic students or (b) the share of students
eligible for free or reduced price lunch.
5. We exclude state-groups which contain fewer than 5% of districts; in addition, Florida is excluded as it is entirely
in the ”Most In-Person” group. All districts, however, are included in regression analyses.

8
More specifically, our estimating equations can be stated:

Yat = Opal (t = 2021) + ByXa + bat + Ve + Cat €at (1)

Yat = (apa + Opata + pra) (t = 2021) + BrXae +7 + Ca + eat (2)

where d represents districts, t is the year, yg is the pass rate for a specific subject in district d
during year t, pg is the share of days in-person in 2021, Xq@ is a matrix of district d’s demographics
(share White, share Black, share Hispanic, and share FRPL) in year t, ¢at is a district specific
linear time trend, y; is a time fixed effect, Cg is a district fixed effect, and €y is a idiosyncratic
shock to pass rates in district d during year t. In the second model, xg is a specific demographic
characteristic (either share of students of color or share FRPL).

We present these analyses in Table |3| with Columns (1)-(3) focusing on math and Columns
(4)-(6) on ELA. Columns (1) and (4) echo the results from Figure [2| The regression suggests that
moving a district from 0% to 100% access to in-person learning would mitigate test score loss by
10.1 percentage points in math and 3.7 percentage points in ELA. Both are highly significant.

In Columns (2) and (5), we explore interactions with the share of Black and Hispanic students in
the district. In the case of math, there is no significant difference in the effect of in-person learning
by districts’ race or ethnicity. In ELA, however, we find that the benefit of in-person learning is
significantly higher in districts with a higher share of students who are Black or Hispanic.

Aggregating the coefficients together, we find that for a district with no Black or Hispanic
students, moving from fully in-person to only hybrid or virtual would be expected to decrease pass
rates by 4.3 percentage points. However, for a district with a 50% Black and Hispanic student
population, that same move from fully in-person to fully not in-person would be expected to
decrease pass rates by 9.0 percentage points.

Columns (3) and (6) do the same for FRPL-eligible students. In this case, students in districts
with a larger share of FRPL-eligible students benefit more from in-person learning in both math
and ELA. The relative gains are larger in ELA versus math. For a district with no FRPL-
eligible students, our estimates suggest full in-person learning would reduce test score losses by
2.9 percentage points. However, for a district where 100% of students were FRPL-eligible, these
advantage to full in-person learning is 10.3 percentage points.

Putting this together, the regressions paint a picture in which average test score losses are larger
in math as compared to ELA. Yet, we see substantial ELA losses among districts with larger Black,
Hispanic, or FRPL-eligible student populations.
We focus here on in-person learning, with the omitted category as a combination of hybrid
and distance learning. Appendix Table 1 replicates Columns (1) and (4) of Table 3 including
both in-person and hybrid learning shares; the omitted category is distance learning. We observe
an intermediate coefficient on hybrid learning share, consistent with the observation that hybrid
districts had an intermediate amount of school time.

3.3 Robustness: Participation Changes

A key empirical issue in evaluating these changes is the role of participation rates. In all states,
test participation was lower in the pandemic year than in previous years. We observe that the
2021 participation rate drops are typically larger in districts with more virtual learning. One
interpretation of both lower pass rates overall and the differences by mode is that they are driven
by these participation changes. To evaluate this, we consider the explicit assumptions which drive
our conclusions.
Consider that there exists an underlying distribution of test scores in the entire population of
students, whether or not they take the test. In a typical non-pandemic year, test participation
rates are extremely high, and it is reasonable to assume we see virtually the entire distribution. In
2021, test taking rates are lower (see Appendix B for details by state). If the 2021 test-takers are
chosen at random from the distribution, however, then the changes we observe reflect the overall
distribution and are unbiased.
If there is negative selection in test-takers – that is, if the bottom of the distribution is more
likely to test – then our estimates will overstate the test score losses. In contrast, if there is positive
selection in test-takers – if the bottom of the distribution is less likely to test – then our primary
estimates will understate test score losses. We explore robustness to this in three ways: 1) by
discussing what is known about test score selection, 2) by using a back of the envelope selection
assumption, and 3) by modeling participation selection explicitly.
On the first point: based on state reports, participation declines during the pandemic appear to
be larger in more vulnerable groups (Colorado Department of Education, 2021; Connecticut State
Department of Education, 2021; Hanson, 2021; Hilton and Linnane, 2021; Littlehales, 2021; Ohio
Department of Education, 2021; Rhode Island Department of Education, 2021). This includes
students from lower socioeconomic status (SES) backgrounds, English language learners (ELL),

10
and others. These groups typically have lower test scores than other student subgroups, such as
their higher SES peers or students who are English proficient. To the extent that these are the
excluded groups, our estimates will understate test score losses.
This argument suggests our results may be conservative. To be more precise, we consider three
approaches to modeling the counterfactual test scores for the absent individuals. First, we create
a bound which assumes a form of positive selection in test takers. Specifically, we assume those
who do not take the test are drawn randomly from a distribution with a one standard deviation
lower test score than the district mean. Second, we create a parallel bound which assumes negative
selection in test takers. This assumes those who do not take the test are drawn randomly from a
distribution with a one standard deviation higher test score higher than the district mean.
Our preferred approach is the third, which explicitly models the relationship between pass rates
and participation. Conceptually, we attempt to estimate the relationship between participation
rates and pass rates, and then infer the expected pass rate if participation was higher. Practically,
we do this by regressing pre-pandemic pass rates on pandemic-year participation rates, interacted
with schooling mode. These regressions also control for schooling mode and for a rich set of
demographics. In the end, the regressions aims to estimate the expected change in pass rate with
a change in participation rate. We then apply these coefficients to the 2021 data to calculate a
counterfactual pass rate under full participation.
Appendix Figure 1 replicates Figure 1 for the pandemic year, showing the changes under
our baseline assumption and the three alternative assumptions about participation; note these
regressions exclude Virginia, where we do not currently have participation data. Under all three
alternative assumptions, we continue to observe declines in test scores. Of note, the calibrated
counterfactual is close to the positive selection bound, suggesting that the calibration supports the
observation that test takers were positively selected.
Appendix Table 2 replicates Columns (1) and (4) of Table 3 under these varying assumptions.
(The baseline effect is lower due to the exclusion of Virginia). The calibrated assumption suggests
a larger advantage for in-person districts, which is consistent with positive selection of those taking
the test. That is: the model underlying the calibrated assumption implies the marginal participants
in the less-in-person districts would be expected to perform worse.
If we assume significant negative selection in test-takers, the effect of in-person learning is
eliminated. This is important in illustrating the possible sensitivity of results, although we emphasize
that both the state-level reports and the calibration point to positive selection in testers.

11
Overall, we suggest based on this analysis that, if anything, the estimates we present in the
baseline model in Table 3 may understate both the overall test score declines and the relative
importance of in-person learning in stemming them.

4 Discussion and Conclusion

A remaining key threat to the validity of these results is endogeneity in schooling mode. This is
the concern that districts with less in-person learning were more affected by other aspects of the
pandemic. The pre-pandemic data in Figure 2 does not suggest differential pre-trends, but cannot
speak to changes which resulted from the pandemic.
The most obvious concern is that less in-person learning may reflect higher community COVID-
19 rates, and that the health impacts of the COVID-19 pandemic are directly driving students’
academic outcomes. However, this explanation does not align with the fact that in-person learning
shares are actually higher in areas with higher COVID-19 rates.
Alternatively, given the clustering of lockdown measures, there is a concern that other non-school
lockdowns – such as closures of local businesses or after-school activities – impacted learning. If
this were the case, it will bias our results. Given the co-linearity between these restrictions, it is
difficult to separate schooling changes from these other pandemic-related changes.
Taken together, the data here suggest that there were considerable declines in test scores overall
during the 2020-21 school year, and these declines were larger in school districts with less in-person
instruction. There are consequences for inequality in outcomes in these results. Students in districts
with larger populations of Black and Hispanic students, for example, were less likely to have access
to in-person learning. In addition, in ELA in particular, the consequences of distanced learning
were larger in districts with these demographics.
Test scores are only one measure of student learning during the 2020-21 school year. This paper
cannot capture ways that students learned that were not reflected on such assessments; we also
cannot account for pandemic-related changes in students’ lives beyond schooling mode. However,
these results can serve as a starting point for education leaders and policymakers as they weigh
where to target funding moving forward in order to support student learning. Specifically, our
analyses suggest that a focus on areas which had less in-person learning over the 2020-21 school
year would be critical. More generally, our analyses demonstrate that that virtual or distanced
schooling modes cannot support student learning in the same way as in-person schooling. As such,

12
educational impacts of schooling mode on students’ learning outcomes should be a critical factor
in policy responses to future pandemics or other large-scale schooling disruptions.

13
References

Alexander, Karl, Sarah Pitcock, and Matthew C Boulay. 2016. The summer slide: What we know
and can do about summer learning loss. Teachers College Press.

Angrist, Joshua D, Parag A Pathak, and Christopher R Walters. 2013. “Explaining charter school
effectiveness”. American Economic Journal: Applied Economics 5 (4): 1–27.

Angrist, Noam, Peter Bergman, and Moitshepi Matsheng. 2020. School’s out: Experimental evidence
on limiting learning loss using “low-tech” in a pandemic. Technical report. National Bureau
of Economic Research.

Ardington, Cally, Gabrielle Wills, and Janeli Kotze. 2021. “COVID-19 learning losses: Early grade
reading in South Africa”. International Journal of Educational Development 86:102480.

Azevedo, João Pedro, Amer Hasan, Diana Goldemberg, Koen Geven, and Syedah Aroob Iqbal.
2021. “Simulating the potential impacts of COVID-19 school closures on schooling and learning
outcomes: A set of global estimates”. The World Bank Research Observer 36 (1): 1–40.

Bacher-Hicks, Andrew, Joshua Goodman, and Christine Mulhern. 2021. “Inequality in household
adaptation to schooling shocks: Covid-induced online learning engagement in real time”. Journal
of Public Economics 193:104345.

Betts, Julian R, and Y Emily Tang. 2011. “The Effect of Charter Schools on Student Achievement:
A Meta-Analysis of the Literature.” Center on Reinventing Public Education.

Chen, Li-Kai, Emma Dorn, Jimmy Sarakatsannis, and Anna Wiesinger. No date. “Teacher survey:
Learning loss is global—and significant”. mckinsey.com.

Contini, Dalit, Maria Laura Di Tommaso, Caterina Muratori, Daniela Piazzalunga, and Lucia
Schiavon. 2021. The COVID-19 pandemic and school closure: learning loss in mathematics in
primary education. Technical report. Collegio Carlo Alberto.

Cooper, Harris, Barbara Nye, Kelly Charlton, James Lindsay, and Scott Greathouse. 1996. “The
effects of summer vacation on achievement test scores: A narrative and meta-analytic review”.
Review of educational research 66 (3): 227–268.

14
Dee, Thomas, Elizabeth Huffaker, Cheryl Phillips, and Eric Sagara. 2021. The revealed preferences
for school reopening: Evidence from public-school disenrollment. Technical report. National
Bureau of Economic Research.

Dorn, Emma, Bryan Hancock, Jimmy Sarakatsannis, and Ellen Viruleg. 2020. “COVID-19 and
learning loss—disparities grow and students need help”. McKinsey & Company, December 8.

Education, Colorado Department of. August 2021. News release - statewide assessment results show
decreased participation, lower CMAS scores than previous years. Technical report.

Education, Connecticut State Department of. August 2021. Statewide Summative Assessment
Report 2020-21. Technical report.

Education, Ohio Department of. September 2021. Data insights: Evidence of the pandemic’s impact
on students in 2020-2021. Technical report.

Education, Rhode Island Department of. 2021. English language arts and mathematics assessment
results 2021. Technical report.

Engzell, Per, Arun Frey, and Mark D Verhagen. 2021. “Learning loss due to school closures during
the COVID-19 pandemic”. Proceedings of the National Academy of Sciences 118 (17).

Ertem, Zeynep, Elissa M Schechter-Perkins, Emily Oster, Polly van den Berg, Isabella Epshtein,
Nathorn Chaiyakunapruk, Fernando A Wilson, Eli Perencevich, Warren BP Pettey, Westyn
Branch-Elliman, et al. 2021. “The impact of school opening model on SARS-CoV-2 community
incidence and mortality”. Nature Medicine, 1–7.

Fuchs-Schündeln, Nicola, Dirk Krueger, Andre Kurmann, Etienne Lale, Alexander Ludwig, and
Irina Popova. 2021. The Fiscal and Welfare Effects of Policy Responses to the Covid-19 School
Closures. Technical report. National Bureau of Economic Research.

Gewertz, Catherine. 2020. “Instruction during COVID-19: Less learning time drives fears of academic
erosion”. Education Week.

Goldhaber, Dan, Scott A Imberman, Katharine O Strunk, Bryant Hopkins, Nate Brown, Erica
Harbatkin, and Tara Kilbride. 2021. To what extent does in-person schooling contribute to the
spread of COVID-19? Evidence from Michigan and Washington. Technical report. National
Bureau of Economic Research.

15
Hammerstein, Svenja, Christoph König, Thomas Dreisörner, and Andreas Frey. 2021. “Effects of
COVID-19-Related School Closures on Student Achievement—A Systematic Review”.

Hanson, Melissa. September 2021. MCAS scores fell during COVID but decrease fairly even across
racial, ethnic groups. Technical report.

Harmey, Sinéad, and Gemma Moss. 2021. “Learning disruption or learning loss: using evidence
from unplanned closures to inform returning to school after COVID-19”. Educational Review,
1–20.

Harris, Douglas N, Engy Ziedan, and Susan Hassig. 2021. “The effects of school reopenings on
COVID-19 hospitalizations”. National Center for Research on Education Access and Choice.

Hilton, AnnMarie, and Rory Linnane. 2021. Fewer Wisconsin students took last year’s standardized
tests. those who did take them fared worse than usual. Technical report.

Hippel, Paul T von, Joseph Workman, and Douglas B Downey. 2018. “Inequality in reading and
math skills forms mainly before kindergarten: A replication, and partial correction, of “Are
schools the great equalizer?””. Sociology of Education 91 (4): 323–357.

Kaufman, Julia H, and Melissa Kay Diliberti. 2021. “Divergent and Inequitable Teaching and
Learning Pathways during (and Perhaps Beyond) the Pandemic: Key Findings from the American
Educator Panels Spring 2021 COVID-19 Surveys.” RAND Corporation.

Kerry, Trevor, and Brent Davies. 1998. “Summer learning loss: The evidence and a possible
solution”. Support for Learning 13 (3): 118–122.

Kuhfeld, Megan, James Soland, Beth Tarasawa, Angela Johnson, Erik Ruzek, and Jing Liu. 2020.
“Projecting the potential impact of COVID-19 school closures on academic achievement”.
Educational Researcher 49 (8): 549–565.

Kuhfield, Megan, and Beth Tarasawa. 2020. “The COVID-19 Slide: What Summer Learning Loss
Can Tell Us about the Potential Impact of School Closures on Student Academic Achievement.
Brief.” NWEA.

Lewis, K, M Kuhfeld, E Ruzek, and Andrew McEachin. 2021. Learning during COVID-19: Reading
and math achievement in the 2020-21 school year.

16
Littlehales, Alex. August 2021. COVID-19 pandemic affected scores, participation of standardized
testing in Virginia. here’s how. Technical report.

Maldonado, Joana Elisa, and Kristof De Witte. 2021. “The effect of school closures on standardised
student test outcomes”. British Educational Research Journal.

Marshall, David T, David M Shannon, and Savanna M Love. 2020. “How teachers experienced the
COVID-19 transition to remote instruction”. Phi Delta Kappan 102 (3): 46–50.

McCombs, Jennifer Sloan, Catherine H Augustine, and Heather L Schwartz. 2011. “Making summer
count: How summer programs can boost children’s learning”.

McLeod, Scott, and Shelley Dulsky. 2021. “Resilience, Reorientation, and Reinvention: School
Leadership During the Early Months of the COVID-19 Pandemic”. In Frontiers in Education,
6:70. Frontiers.

Musaddiq, Tareena, Kevin M Stange, Andrew Bacher-Hicks, and Joshua Goodman. 2021. The
Pandemic’s effect on demand for public schools, homeschooling, and private schools. Technical
report. National Bureau of Economic Research.

Schult, Johannes, and Marlit Annalena Lindner. 2021. “Did students learn less during the COVID-
19 pandemic? Reading and mathematics competencies before and after the first pandemic
wave”.

Storey, Nathan, and Qiyang Zhang. 2021. “A Meta-analysis of COVID Learning Loss”.

Tomasik, Martin J, Laura A Helbling, and Urs Moser. 2021. “Educational gains of in-person vs.
distance learning in primary and secondary schools: A natural experiment during the COVID-
19 pandemic school closures in Switzerland”. International Journal of Psychology 56 (4): 566–
576.

Verlenden, Jorge V, Sanjana Pampati, Catherine N Rasberry, Nicole Liddon, Marci Hertz, Greta
Kilmer, Melissa Heim Viox, Sarah Lee, Neha K Cramer, Lisa C Barrios, et al. 2021. “Association
of children’s mode of school instruction with child and parent experiences and well-being during
the COVID-19 pandemic—COVID experiences Survey, United States, October 8–November 13,
2020”. Morbidity and Mortality Weekly Report 70 (11): 369.

Werner, Katharina, and Ludger Woessmann. 2021. “The Legacy of Covid-19 in Education”.

17
Wyse, Adam E, Eric M Stickney, David Butz, Amanda Beckler, and Catherine N Close. 2020. “The
potential impact of COVID-19 on student learning and how schools can respond”. Educational
Measurement: Issues and Practice 39 (3): 60–64.

18
5 Tables and Figures

Figure 1: Pass Rate Changes by Subject and State

(a) Math Scores

Change in Percent Pass from Previous Year for Math
.1

-.1

-.2

-.3

-.4
VA MN MA NV WV CO OH WI CT RI WY FL

Pre-Pandemic Pandemic Year

(b) ELA Scores

Change in Percent Pass from Previous Year for ELA
.1

-.1

-.2

-.3

-.4
VA MN MA NV WV CO OH WI CT RI WY FL

Pre-Pandemic Pandemic Year

Notes: Figures show the change in percent pass from the previous year by state for pre-pandemic and
pandemic years separately. All test scores are for Grades 3-8. Details of the testing by state are in Appendix
B. States are in ascending order from left to right based on the share of the school year in-person.

19
Figure 2: In-Person Share and Pass Rate Changes

(a) Math Scores

Change in Percent Pass from Previous Year for Math
Pre-Pandemic Pandemic Year
.1

-.1

-.2

-.3

VA MN MA NV WV CO OH WI CT RI WY VA MN MA NV WV CO OH WI CT RI WY

Most In-person Least In-person
Intermediate In-person

(b) ELA Scores

Change in Percent Pass from Previous Year for ELA
Pre-Pandemic Pandemic Year
.1

-.1

-.2

-.3
VA MN MA NV WV CO OH WI CT RI WY VA MN MA NV WV CO OH WI CT RI WY

State
Most In-person Least In-person
Intermediate In-person

Notes: Figures show change in pass rates from the previous year by subject, state, and schooling modes for
pre-pandemic and pandemic years separately. In-person learning is defined as traditional in-person school; the
alternative categories are hybrid and distance learning. ”Most In-Person” includes districts that were in-person
at least two-thirds of the year; ”Intermediate In-Person” includes districts that were in person one-third to
two-thirds of the time; ”Least In-person” includes districts that were in-person less than a third of the time.
States are in ascending order from left to right based on the share of the school year in-person. Florida is
excluded due to limited variation in in-person shares.

20
Table 1: Summary Statistics by State

Districts Avg Years 2019 Pass Rate 2021 Pass Rate % In-Person % Hybrid % Virtual % Black & Hispanic % FRPL
CO 141 4.87 46.18 42.98 42.41 27.10 30.49 39.30 41.11
CT 177 4.94 55.84 48.31 57.41 31.80 10.78 39.55 41.43
FL 64 4.97 55.83 52.06 97.50 0.00 2.50 56.53 54.15
MA 352 3.99 52.06 45.84 27.49 52.44 20.07 31.09 0.00
MN 462 4.84 59.05 51.04 14.87 65.14 19.99 20.97 35.45
NV 19 5.00 47.25 41.85 31.64 48.34 20.02 38.64 53.39
OH 597 4.98 66.30 58.08 49.13 31.98 18.88 20.52 24.98
RI 49 2.98 47.05 40.28 58.80 35.75 5.45 26.49 35.53
VA 132 5.00 76.11 67.08 9.16 52.30 38.54 38.92 44.81
WI 414 4.95 41.16 33.74 56.93 21.16 21.90 21.80 40.30
WV 55 5.00 46.21 40.04 37.16 41.49 21.34 6.12 51.28
WY 48 3.00 56.84 54.65 93.31 5.80 0.89 14.86 34.71
21 Notes: This table shows summary statistics for the states included in the sample. The schooling mode shares represent the average % of the time the district
offered each schooling mode. Data on learning modes is drawn from the COVID-19 School Data Hub. Details on test score data for each state are in Appendix B.
Table 2: Determinants of Schooling Mode

(1) (2) (3) (4)
Share In-Person Share In-Person Share In-Person Share In-Person
Baseline Pass Rate 0.137 -0.0654 0.130 0.153
(0.036) (0.035) (0.036) (0.037)

Share Black -0.450 -0.613 -0.567 -0.564
(0.036) (0.035) (0.026) (0.026)

Share Hispanic 0.369 0.228 -0.212 -0.171
(0.0322) (0.032) (0.024) (0.024)

Share FRPL 0.433 0.302 0.123 0.136
(0.027) (0.027) (0.022) (0.022)

Avg. Case Rate 0.0039 0.0032
(0.0006) (0.0004)

Aug. 2020 Case Rate 0.011 0.00031
(0.0006) (0.0004)
Observations 4950 4950 4950 4950
State FE No No Yes Yes
Notes: This table shows regressions of the share of days in-person during the 2020-21 school year on district
characteristics. The share in-person measures the share of time during the 2020-21 school year in which the
district offered full time in-person instruction. COVID-19 case rates are from USA Facts. Regressions are
weighted by district enrollment.

22
Table 3: Schooling Mode and Test Score Changes

(1) (2) (3) (4) (5) (6)
Math Math Math ELA ELA ELA
Pass
% In-Person 0.101 0.143 0.117 0.0317 0.0425 0.0290
(0.00314) (0.00423) (0.00578) (0.00236) (0.00315) (0.00430)

% In-Person * -0.0386 0.0475
% Black-Hisp * 2021 (0.00916) (0.00682)

% In-Person * 0.0262 0.0735
% FRPL * 2021 (0.0122) (0.00910)
Observations 11772 11772 11772 11795 11795 11795
District FE YES YES YES YES YES YES
District Trends YES YES YES YES YES YES
Year FE. YES YES YES YES YES YES
Demog. X Year FE. YES YES YES YES YES YES
Race X In-Person Sh. NO YES NO NO YES NO
FRPL X In-Person Sh. NO NO YES NO NO YES
Notes: This table shows the relationship between district in-person share and test score changes. Districts
that were not in-person were either hybrid or virtual. Regressions are weighted by district enrollment.

23
Appendix A: Additional Tables and Figures

Appendix Figure 1: Pass Rate Changes, Varying Participation Assumptions

(a) Math Scores

Robustness: Participation (Math)
.1

-.1

-.2

-.3

-.4
MN MA NV WV CO OH WI CT RI WY FL

Baseline Calibrated
Negative Selection Positive Selection

(b) ELA Scores

Robustness: Participation (ELA)
.1

-.1

-.2

-.3

-.4
MN MA NV WV CO OH WI CT RI WY FL

Baseline Calibrated
Negative Selection Positive Selection

Notes: These figures show bounds on the pass rate changes under our baseline assumption and three
assumptions about test scores among non-participants. The “Positive Selection” bound assumes that test takers
are positively selected and the non-takers are drawn from a distribution with a mean one standard deviation
below the district mean. The “Negative Selection” bound assumes the test takers are negatively selected and
the non-takers are drawn from a distribution with a mean one standard deviation above the district mean.
The “Calibrated Selection” models the relationship between pass rates and participation, as detailed in the
discussion of robustness. Virginia is excluded due to missing participation data.

24
Appendix Table 1: Separating Hybrid and Virtual Learning

(1) (2)
Math ELA
Pass
% In-Person 0.127 0.0516
(0.00561) (0.00420)

% Hybrid 0.0408 0.0307
(0.00716) (0.00535)
Observations 11772 11795
District FE YES YES
District Trends YES YES
Year FE. YES YES
Demog. X Year FE. YES YES
Notes: This table shows the relationship between district in-person share, district hybrid share and test
score changes. The omitted category is virtual learning. Regressions are weighted by district enrollment. All
regressions include the same structure as those in Table 3.

Appendix Table 2: Participation Bounding Analysis

Math ELA
Baseline 0.0399 (0.00284) 0.0415 (0.00263)
Positive Selection 0.0693 (0.00280) 0.0623 (0.00260)
Negative Selection 0.00531 (0.00299) 0.0197 (0.00273)
Calibrated Selection 0.0634 (0.00280) 0.0552 (0.00259)

Notes: This table shows our primary results about in-person shares and pass rates under the baseline and three
assumptions about test scores among non-participants. The ”Positive Selection” bound assumes that test takers
are positively selected and the non-takers are drawn from a distribution with a mean one standard deviation
below the district mean. The ”Negative Selection” bound assumes the test takers are negatively selected and
the non-takers are drawn from a distribution with a mean one standard deviation above the district mean. The
”Calibrated Selection” models explicitly the relationship between pass rates and participation, as detailed in
the discussion of robustness. Virginia is excluded due to missing participation data. All regressions include the
same controls as in Table 3.

25
Online Appendix

Pandemic Learning Mode and Student Test Scores: Evidence from US States
Emily Oster, James Okun, Rebecca Jack, & Clare Halloran
Appendix B. Details for State Assessments Included in Analysis, Grades 3-8
Years 2021
Dates Tests Included In Participation
State Assessment Name Administered Analysis Rate Additional Information
In 2021, instead of all students testing in all
Colorado Measures of subjects as in prior years, Grades 3, 5, and 7 were
2016 – 2019,
Colorado Academic Success Apr 12 – May 14 81.8% tested in ELA, and Grades 4, 6, and 8 were tested
2021
(CMAS) in math. Parents could opt their children in to take
both tests.
Smarter Balanced
2016-2019,
Connecticut Assessment Consortium Mar 29 – Jun 4 92.4%
2021
(SBAC)

Florida Standards 2016-2019,
Florida Apr 5 – Jun 11 91.8%
Assessments (FSA) 2021

Massachusetts In 2017, the state first administered the
Comprehensive 2017-2019, next-generation MCAS to replace the state's
Massachusetts May 10 – Jun 11 95.5%
Assessment System 2021 former MCAS; years prior to 2017 were not
(MCAS) included in analyses.
Minnesota
2016-2019,
Minnesota Comprehensive Mar 8 - May 21 82.9%
2021
Assessments (MCA)
Smarter Balanced
2016-2019,
Nevada Assessment Consortium Feb 19 - May 19 90.0%
2021
(SBAC)
Mar 22 – May 7
Ohio Achievement (ELA); 2016-2019,
Ohio 97.0%
Assessment (OAA) Mar 29–May 21 2021
(math)
Rhode Island Mar 29 – May 14
In 2018, the state first administered the RICAS
Comprehensive (ELA); 2018-2019,
Rhode Island 91.5% assessment to replace the PARCC assessment;
Assessment System Apr 12 – May 28 2021
years prior to 2018 were not included in analyses.
(RICAS) (math)

Standards of Learning 2016-2019, Not Currently In 2021, the state updated its cut scores for
Virginia Mar 15 – Jun 25
(SOL) 2021 Available reading.

West Virginia General
2016-2019,
West Virginia Summative Assessment Mar 29 – Jun 4 85.9%
2021
(WVGSA)

2016-2019,
Wisconsin Forward Exam Mar 22 – May 14 95.0%
2021

Wyoming Test of In 2018, the state first administered the WY-TOPP
2018-2019,
Wyoming Proficiency and Progress April 13 – May 7 97.1% assessment to replace the state's use of the PAWS
2021
(WY-TOPP) assessment.

File and source

File: https-www-the74million-org-article-roza-without-clear-rules-theres-no-way-to-judge-how-school-relief-funds-are-being-spe.pdf
Size: 466,592 bytes
SHA-256: 02824c1714ae38861f58c7ab6f6626aaec6111622220a3a0965c4593c94fda5c
Our copy: https-www-the74million-org-article-roza-without-clear-rules-theres-no-way-to-judge-how-school-relief-funds-are-being-spe.pdf
Original: www.nber.org