The First Year of Pandemic Recovery: A District-Level Analysis
Summary
The First Year of Pandemic Recovery: A District-Level Analysis, a January 2024 report by Erin Fahle, Thomas J. Kane, Sean F. Reardon and Douglas O. Staiger, uses test score results from roughly 8,000 school districts in 30 states to measure pandemic losses and the recovery from Spring 2022 to Spring 2023. The report finds that students recovered approximately one-third of the original loss in math and one quarter of the loss in reading. It reports that Alabama is the only state where average achievement exceeds pre-pandemic levels in math and that, in most states, gaps between rich and poor districts are wider than before the pandemic. It says districts had $51 billion of the original $190 billion in federal relief funds remaining as of January 2024 and proposes ways to extend the recovery into summer and the next school year.
Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used
Cite this document
“The First Year of Pandemic Recovery: A District-Level Analysis.”
Archived at Pandemic Darlings, https://pandemicdarlings.com/source-documents/the-first-year-of-pandemic-recovery-a-district-level-analysis-educatio-383a96ac8a/
, SHA-256 d59111ebf22d544e89388c34626ff9be18bd40ad5761154699b2a77b66b1adea.Full text
January 2024
The First Year of
Pandemic Recovery:
A District-Level Analysis
Erin Fahle, Stanford University
Thomas J. Kane, Harvard University
Sean F. Reardon, Stanford University
Douglas O. Staiger, Dartmouth College
We thank Pete Claar at SchoolDigger, Sadie Richardson, Julia Paris, Demetra Kalogrides, Jie Min, and
Jiyeon Shim for their assistance in producing the SEDA data used in this paper. We also received
data on spending of federal relief dollars from Dennis and Julie Roche at Burbio and Marguerite
Roza at the Edunomics Lab at Georgetown University. Daniel Dewey, Victoria Carbonari and Dean
Kaplan at the Harvard Center for Education Policy Research provided research assistance. The
National Center for Education Statistics (NCES) and the National Assessment Governing Board
provided data on student achievement by state which we used to rescale state proficiency data. The
research was supported by grants from the Carnegie Corporation of New York, Bloomberg
Philanthropies, and Kenneth C. Griffin. The Bill & Melinda Gates Foundation has funded the Stanford
Education Data Archive. The opinions expressed here are ours and do not represent views of NCES,
the U.S. Department of Education, or any of the funders.
The First Year of Pandemic Recovery: A District-Level Analysis 1
Introduction
Beginning in March of 2020, the COVID pandemic disrupted learning nationwide, as schools in every
state abruptly shifted to remote instruction. Although some schools returned to in-person
instruction quickly the following fall, many schools did not return to fully in-person instruction until
the fall of 2021. Last year, we and others reported that by the time all states returned to regular
testing in the spring of 2022, the average student in grades 3 through 8 had lost the equivalent of
half a grade level in math achievement and a third of a grade level in reading achievement. [1] In
this new report, we describe the first full year of post-pandemic recovery, between the spring of
2022 and the spring of 2023.
To recover, school leaders needed to do more than turn the lights back on and return to “normal”
operations. Schools were expected to catch students up, which meant re-organizing school activities
to generate more learning per day than prior to the pandemic. For example, if students lost half a
year of learning during the pandemic, schools would need to find a way to produce 125 percent of
the typical learning for two consecutive years in a row to return to 2019 levels of achievement—a
challenging task given staffing shortages and a post-pandemic increase in student absenteeism
(Dee, 2023).
Schools had additional resources to pay for those supplemental activities in the form of $190 billion
in federal pandemic relief. For instance, forty percent of school districts planned to provide small
group tutoring (Jordan, DiMarco and Toch, 2022), sixty percent planned to expand summer and
after school programs (Jordan and DiMarco, 2022) and a third of districts planned to hire
psychologists, social workers and counselors to help with a burgeoning mental health crisis among
students (Jordan, Dwyer, DiMarco and Johnson-Green, 2022). Families and students also
contributed, working with educational software, private tutoring and attending summer learning.
Until now, the evidence about the net effect of all these efforts on catching up to pre-pandemic
levels of achievement has been inconclusive. For example, in July of 2023, NWEA—the provider of
student assessments used by thousands of school districts in the U.S.—reported that the growth in
student achievement in the districts using their assessment had been below pre-pandemic levels
during the 2022-23 school year (Lewis and Kuhfeld, 2023). Several months later, another vendor,
Curriculum Associates, also reported little improvement between the spring of 2022 and 2023 for
students using their curriculum and assessments (Curriculum Associates, 2023). However, Emily
Oster and her colleagues analyzed the change in proficiency rates between 2021 and 2023 in each
state and characterized 4 states as having fully recovered in English and 2 states in math. They also
concluded that 5 states had declined in English (as summarized in Barnum and Belsha, 2023).
[1] See, for example https://www.nationsreportcard.gov/highlights/mathematics/2022/;
https://www.nationsreportcard.gov/highlights/reading/2022/; https://educationrecoveryscorecard.org/wp-
content/uploads/2022/10/Education-Recovery-Scorecard_Key-Findings_102822.pdf; and Fahle, et al (2023).
The First Year of Pandemic Recovery: A District-Level Analysis 2
Unfortunately, percentage point changes in proficiency on state tests are unreliable measures of
improvement, especially for comparing rates of recovery between states, given that states define
proficiency differently (Ho, 2008). Although we use state test results as well, we have adjusted for
differences in state proficiency thresholds, using the National Assessment of Educational Progress
(NAEP).
Executive Summary of Findings
and Recommendations
Despite the lack of improvement during 2022-23 on assessments provided by NWEA and Curriculum
Associates, we find that student achievement did improve between Spring 2022 and Spring 2023: in
fact, students recovered approximately one-third of the original loss in math (0.17 grade levels out
of the 0.53 grade levels decline from 2019-2022) and one quarter of the loss in reading (0.08 grade
levels out of the 0.31 grade level decline from 2019-2022). Such improvements in grade levels in a
single school year mean that students learned 117 percent in math and 108 percent in reading of
what they would typically have learned in a pre-pandemic school year. These gains are large relative
to historical changes in math and reading achievement on the National Assessment of Educational
Progress.
Despite improvement, Alabama is the only state where average student achievement exceeds pre-
pandemic levels in math; average achievement in three states (Louisiana, Illinois, and Mississippi)
are above pre-pandemic levels in reading. However, even in these four states where average
achievement exceeds pre-pandemic levels, many of the poorest districts still lag behind 2019 levels
of achievement. For example, students in Montgomery, Alabama are achieving half a grade level
behind 2019 levels in math.
In some states, such as Alabama, Mississippi, and Georgia, the recovery last year was larger in
higher poverty districts. In other states, such as Massachusetts and Connecticut, the recovery was
led by wealthier districts and the higher poverty districts were left even further behind. However, in
most states, the recovery from 2022 to 2023 was insufficient to erase the sharp widening of
achievement gaps between high- and low-poverty districts which occurred during the pandemic. In
all but three states (Arkansas, Pennsylvania, and Virginia), poorer districts remain further behind
their 2019 levels of achievement than wealthier districts. In most states, achievement gaps between
rich and poor districts are even wider now than they were before the pandemic.
In the states which reported usable data on subgroups of students within districts, we also see that
the recovery was slower for poor students than for their non-poor classmates in the same districts.
The net result of larger initial losses in poor districts and slower recovery of poor students within
the average district is that achievement gaps have widened since 2019. The two states with the
largest widening of achievement gaps between poor and non-poor students are Massachusetts and
The First Year of Pandemic Recovery: A District-Level Analysis 3
Michigan, where there were large increases in gaps in both math and reading.
As of the January 2024, school districts had $51 billion of the original $190 billion in federal relief
funds remaining. Federal law requires that districts “obligate” such funds the end of September
2024—or return unused dollars to the federal treasury. Hopefully, districts are deploying those
dollars to continue the recovery this school year. However, even if school districts manage to
maintain the same pace of recovery that they achieved last year, average student achievement will
not have returned to 2019 levels by that deadline. Having recovered one-third of the loss in reading
and one-quarter of the loss in math means that districts would need at least another year of
recovery in math and two more years in reading.
To complete the catch up and to prevent the widened achievement gaps from becoming
permanent, the remaining federal funds should be used to replace what students lost during the
pandemic—which was time engaging with teachers. In the final section of this report, we propose
ways that states and districts could extend the recovery into the summer and next school year. For
example, we encourage districts to inform parents whose children remain behind grade level so that
they can sign up for summer learning this summer and to expand summer learning programs to
accommodate all students who sign up. Although districts cannot spend the federal dollars on their
own employee salaries after September, federal regulations will allow districts to make payments
on contracts beyond the September deadline, as long as the contracts are signed before September.
In fact, the federal Department of Education has signaled its openness to district extension requests
for contracts which extend academic recovery, for activities such as for tutoring, summer learning
and after-school programs. [2] Accordingly, we encourage districts to set aside federal dollars to pay
for tutoring and after-school programs to extend the recovery into next school year. In many (if not
most) districts, students will need additional time to catch up.
Once the federal dollars are depleted, however, state and local leaders may need to complete the
final leg of the recovery with their own resources. Rather than continue to provide unrestricted
dollars to districts, states might consider targeted incentives to districts to add instructional time by
extending the school year or expanding summer learning, as Texas has done. Other possibilities
would be to provide dollars for specific interventions—such as tutoring or summer learning—for
students who remain behind grade level on state tests or to districts/schools furthest behind 2019
levels of achievement. Until now, governors and state legislators have primarily been observers, as
90 percent of the federal relief dollars were provided directly to local districts, giving state
authorities little say in how those dollars were used. State leaders will need to be more actively
involved if the final stage of recovery is to be successful.
[2] https://oese.ed.gov/files/2024/01/ARP-Liquidation-Extension-Letter-1.9.24-final-for-signature-v3.pdf.
The First Year of Pandemic Recovery: A District-Level Analysis 4
Our Analysis
In this report, we use test score results from roughly 8,000 school districts in 30 states to measure
the extent to which test scores changed from Spring 2019 to Spring 2022 and from Spring 2022 to
Spring 2023. We use methods developed by Reardon and colleagues to put the test scores from
each state’s tests onto a common scale and to convert proficiency rates to “grade levels” of
achievement (Reardon, Kalogrides, & Ho, 2021; Reardon, Shear, Castellano, & Ho, 2017). This allows
us to measure changes in test scores between 2019 and 2023 on a common, interpretable scale for
all school districts in these 30 states, despite the fact that they use different tests and proficiency
thresholds.
Using these detailed data, we report 1) the extent of academic recovery for each of the 30 states; 2)
the extent to which the 2019-2022 losses and 2022-2023 improvements varied across school
districts; and 3) how these patterns varied by district poverty rates and by student racial and
economic background.
DATA
We use the results of state-administered standardized tests in math and reading to measure
pandemic impacts and recovery. [3] States are required to test all grade 3-8 students annually in
math and reading, but that requirement was waived in 2020 and 2021. [4] Given the lack of testing
in 2020 and the incomplete testing in 2021, we measure pandemic losses by comparing
achievement in spring 2019 and spring 2022; we measure the recovery by comparing achievement
in spring 2022 and spring 2023 (although many schools started their recovery efforts in the fall of
2021).
We use test score results aggregated by school district, year, subject, and grade, and, where
available, by racial and economic subgroup. We focus on traditional school districts that have
geographic boundaries. Charter schools that are administered by a local school district are often
included in their district’s average. We do include data from independent charter schools whenever
reporting state averages but do not report their results separately.
Each state’s department of education chooses the tests they use. In addition, each state sets its own
proficiency thresholds used to categorize students’ scores (typically into 3-5 categories) in each
grade and subject. As a result, the definition and measurement of proficiency varies by state, grade,
subject, and—when states change their tests or proficiency thresholds—by year. Comparisons of
[3] We use test score data from the Spring of 2016-2019 and from Spring 2022 and Spring 2023. Data for the years
2016-2019 is available for all states from EDFacts. We collected 2022 and 2023 test score results from state department
of education websites.
[4] In Spring 2021, states were required to administer tests, but students were explicitly allowed to opt-out. As a result,
many students were not tested in 2021, rendering the 2021 data not comparable to other years.
The First Year of Pandemic Recovery: A District-Level Analysis 5
proficiency rates across states, grades, subjects, and years are therefore invalid. Moreover, for
reasons described by Ho (2008), it can also be misleading to compare changes in proficiency rates
across grades and subjects in the same state.
In order to make test scores comparable across states, years, and grades, we link the state test
scores to the National Assessment of Educational Progress (NAEP) test scale, using methods
described in Reardon, Kalogrides, and Ho (2021). NAEP administered the NAEP math and reading
assessment to random samples of 4th and 8th graders in each state in 2015, 2017, 2019, and 2022.
By assuming that the state test results are reflective of the same population as the NAEP and that
the distribution of achievement is normally distributed (“bell-shaped”), we can use a state’s results
on the NAEP to place the state proficiency thresholds on the NAEP scale. We then use the
percentage of students in each proficiency category to estimate a district’s mean achievement in
NAEP points (see Reardon, Kalogrides, & Ho, 2021; Reardon, Shear, Castellano, & Ho, 2017).
In the years when the NAEP results are not available (e.g. 2016 and 2018), we interpolate state
average scores (Reardon, Kalogrides, & Ho, 2021). Because the NAEP test was not administered in
2023 (and the 2024 results are obviously not yet available), we do not have a NAEP estimate of each
state’s average test score in 2023. Instead, we identify the subset of states whose tests and
proficiency thresholds did not change between 2022 and 2023. We then use our estimates of the
2022 proficiency thresholds in these states to estimate each district’s average test scores on the
same scale used in 2022.
In other words, prior to 2023, we rely on NAEP to compare average test scores across states and
over time. This comparison is valid regardless of whether a state changed its test or proficiency
thresholds. But our estimates of changes in test scores from 2022 to 2023 rely on the stability of the
state tests and proficiency definitions. Because the psychometric process of equating test scores
from one year to the next is imperfect, there may be cases where the magnitudes of recovery from
2022 to 2023 are modestly over- or under-stated by our method. When the 2024 NAEP results
become available in early 2025, we will be able to update our 2023 estimates, by resuming our
practice of interpolating between the 2022 and 2024 estimates.
Because we must assume that the NAEP tests and the state tests apply to the same population, we
exclude state-year-grades where fewer than 95% of students took state assessments (this affects
New York and Colorado in particular, where test participation rates have been low since 2015). We
also exclude states that did not post sufficiently detailed data (i.e. counts of students in three or
more proficiency categories) on their websites in 2022 or 2023. As noted above, we also exclude
states whose tests changed between 2022 and 2023. Our analysis last year focused on 42 states
that had the necessary data for 2019 and 2022. We use a subset of 30 of these states—those whose
tests did not change between 2022 and 2023—to study the recovery from 2022 to 2023. In a few
states, we have data for only math or reading, but most states have data in both subjects. Not all
states report publicly district-level data broken down by race/ethnicity and/or by economic status.
The First Year of Pandemic Recovery: A District-Level Analysis 6
Data are available by race/ethnicity for 20 states in 2019, 2022, and 2023; and by economic status in
15 states.
The data we use for this report are available at https://edopportunity.org/recovery, where users
can visualize trends and patterns of academic achievement from 2019-2023, search for and
download data and reports for individual school districts and states, and obtain the detailed
technical documentation of the procedures we use to construct the data (Fahle et al, 2024).
RESULTS
Between 2019 and 2022, the average student in the 30 states lost 0.53 grade levels in math
achievement and 0.31 grade levels in reading. In other words, during the pandemic, students missed
out on half of a year’s typical learning in math and a third of a year in reading. Between 2022 and
2023, students recovered approximately one-third of the original loss in math (0.17 grade levels)
and one quarter of the loss in reading (0.08 grade levels). Thus, although students have yet to return
to pre-pandemic levels of achievement, progress is being made.
National Trends
FIGURE 1: TREND IN AVERAGE TEST SCORES, 2009-2023
Moreover, the gains from 2022 to 2023 are relatively large. One way to assess the relative
magnitude of the changes is to compare them to historical changes in average achievement.
Appendix Figure A1 shows the national trend in NAEP math and reading scores from 1990-2022 and
the change in state test scores in the 30 states in our data from 2009-2023. In math the 2022-2023
change of 0.17 grade levels is twice as large as the long-run annual rate of change from 1990-2019
(and 50% larger than the 0.11 annual rate of change from 1990-2013 when math scores grew most
The First Year of Pandemic Recovery: A District-Level Analysis 7
rapidly). [5] In reading, the 2022-2023 change of 0.08 grade levels in one year is much larger than in
most years, twice as large as the 0.04 average annual improvement rate during the period from
1994-2013 when reading scores grew most rapidly, and five times larger than the average annual
rate of change from 1992-2019.
A second way to assess the relative magnitude of the changes is to compare them to what prior
research indicates regarding the expected impact of the supplemental funding. As of September
2023, the Department of Education reported that districts had spent roughly $122 billion of the
total $190 billion in ESSER funds. [6] Even if we assume that districts spent half of this on academic
instruction and recovery efforts (more than the minimum 20 percent required), a back-of-the
envelope calculation suggests that the maximum improvement we might have expected from half
of $122 billion would be 0.07 grade levels. [7] That is less than half as large as the actual
improvement in math, and slightly smaller than the average improvement in reading. Given that
districts likely spent less than half of the dollars on academic catchup, the recovery was larger than
we would have expected based on spending alone. Other factors, such as supplemental efforts from
families and students, are likely to have played a role as well.
In sum, the achievement gains from 2022 to 2023 were large, both relative to historical trends and
relative to what prior research would lead us to expect based on the ESSER funding.
Patterns of Recovery by Race and Ethnicity
The trend in test scores during and after the pandemic differed by race and economic status. Figure
2 shows the patterns of loss and recovery by race in the 20 states for which we have data broken
down by race. In these states, test scores of Black and Hispanic students declined more than those
of White students during the pandemic, but Black students’ scores improved more from 2022-2023
than those White and Hispanic students; Hispanic students scores showed very little or no average
recovery from 2022-2023. Despite the fact the Black students experienced more recovery from
2022-2023; the White-Black gap was still slightly larger in 2023 than it was in 2019, particularly in
[5] See Figure A1. Nationally, average math scores increased by 2.6 grade levels from 1990-2019, an average increase of
0.08 grade levels/year; in the period with the most rapid sustained growth of math scores (1990-2013), scores increased
by an average of 0.11 grade levels/year. In reading, scores increased by 0.4 grade levels from 1992-2019, a rate of 0.015
grade levels/year; in the period of most rapid sustained growth in reading (1994-2013), scores increased at a rate of
0.04 grade levels/year.
[6] See https://covid-relief-data.ed.gov/ and https://oese.ed.gov/offices/education-stabilization-fund/elementary-
secondary-school-emergency-relief-fund/.
[7] Jackson and Mackevicius (2024) find that rigorous evaluations of supplemental school funding find an average 1-
year effect of 0.0078 SDs/$1000 (which translates to 0.023 grade levels/$1000), though there is variation in the effects
of funding. They find that the maximum effect is about 0.055 grade levels/$1000 of annual spending. The average ESSER
spending as of September 2023 was roughly $2,400/student ($122B divided among roughly 50 million students). If half
of that was spent on instruction, the literature would suggest that test scores would increase by 0.028 grade
levels/student if the ESSER funding were as effective as the average in the literature, and by 0.067 grade levels if the
ESSER funding were maximally effective.
The First Year of Pandemic Recovery: A District-Level Analysis 8
math. Likewise the White-Hispanic gap was larger in 2023 than in 2019 (trends in gaps are shown in
Appendix Figure A2).
FIGURE 2: TRENDS IN AVERAGE TEST SCORES, BY RACE
Similarly, as shown in Figure 3, both poor and non-poor students’ test scores increased from 2022-
2023 in the 15 states for which we have test results broken down by economic status. The
improvement, however, was larger for nonpoor students than for poor students. As a result, the
nonpoor-poor gap in achievement grew during the recovery.
FIGURE 3: TRENDS IN AVERAGE TEST SCORES, BY ECONOMIC
STATUS
The First Year of Pandemic Recovery: A District-Level Analysis 9
Variation Among States
Both the magnitude of the pandemic losses and the pace of the recovery last year varied by state. In
Figure 4 below, we portray the losses in average math and reading achievement over the course of
the pandemic by state. The five states with the smallest losses in math achievement (less than 40
percent of a grade level) were Alabama, Illinois, Wisconsin, Louisiana, South Dakota, and Utah. The
eight states with the largest losses in math achievement (Virginia, Oklahoma, West Virginia, New
Jersey, Connecticut, Mississippi, Oregon, and Pennsylvania) lost more than 70 percent of a grade
level during the pandemic.
FIGURE 4: AVERAGE TEST SCORE CHANGE, 2019-2022, BY STATE
AND SUBJECT
Figure 5 portrays the magnitude of each state’s recovery between Spring 2022 and Spring 2023. All
but one state (Oregon) improved in math achievement, with some states such as Tennessee,
Pennsylvania, and Mississippi improving by more than a third of a grade level in a single year. Any
improvement of such a magnitude was a remarkable achievement, since it implies that the average
student learned 133 percent or more of the typical learning over the academic year last year. In
reading, students in two states (Mississippi and Illinois) made up more than a third of a grade level,
while students in 10 states actually lost additional ground in reading (Nevada, Virginia, Oregon,
California, South Dakota, Wyoming, Indiana, Oklahoma, Connecticut, and Washington.)
The First Year of Pandemic Recovery: A District-Level Analysis 10
FIGURE 5: AVERAGE TEST SCORE CHANGE, 2022-2023, BY STATE
AND SUBJECT
Figure 6 portrays the net loss in math and reading achievement since 2019. Students in only one
state (Alabama) exceeded their pre-pandemic levels of achievement in Spring 2023 testing, while
students in three states (Illinois, Mississippi, and Louisiana) had returned to pre-pandemic levels in
reading.
FIGURE 6: AVERAGE TEST SCORE CHANGE, 2019-2023, BY STATE
AND SUBJECT
The First Year of Pandemic Recovery: A District-Level Analysis 11
The most important take-away from our report last year was not simply that achievement declined
—although it did; it was that gaps between high- and low-poverty districts widened sharply:
achievement declined much more for students in the highest poverty districts than those in the
lowest poverty ones (Fahle, et al. 2023). In Figure 7 below, we report the mean loss in math and
reading achievement by the percentage of students with family incomes qualifying for the federal
subsidized lunch program. Students in the highest income districts lost 30 percent of a grade level in
math and 10 percent of a grade level in reading, while students in the highest poverty school
districts lost more than twice as much in math achievement (70 percent of a grade level) and four
times as much in reading achievement (42 percent of a grade level).
FIGURE 7: TEST SCORE DECLINE AND RECOVERY, 2019-2023, BY
DISTRICT POVERTY RATE
Figure 7 also shows that the recovery so far has done little to close the gaps in achievement which
opened during the pandemic. Across the thirty states, the extent of recovery was generally similar in
high and low poverty districts (with the exception of the highest poverty category which showed
larger than average improvement in both math and reading).
However, Figure 7 masks the fact that the pattern of recovery varied by state. For example, Figure 8
reports the losses and recovery in math achievement for individual school districts in Alabama. The
panel on the left reports the declines in math achievement by district poverty status during the
pandemic. As was true nationally, high poverty districts in Alabama, such as Montgomery, Mobile
and Birmingham lost more than half a grade level, while higher income districts, such as Hoover or
Shelby Counties, lost little ground or improved in achievement. The second panel reports the
improvements between 2022 and 2023 for the same districts. As one would have hoped, the higher
The First Year of Pandemic Recovery: A District-Level Analysis 12
poverty districts in Alabama—those that had lost the most ground during the pandemic—led the
recovery, for instance with Birmingham making up nearly half a grade level in math in one year. The
third panel reports the net change from 2019 to 2023. Although average math achievement in
Alabama may be above 2019 levels, there are many districts, such as Montgomery County, which
are still lagging behind 2019 levels. The relationship between achievement loss and district poverty
is somewhat flatter in 2019-23 than in 2019-22 because poorer districts made greater gains from
2022 to 2023. Nevertheless, the gaps remain wider than before the pandemic.
FIGURE 8: MATH SCORE CHANGES IN ALABAMA DISTRICTS, 2019-
2023, BY PERIOD
Source: Education Recovery Scorecard, by Harvard CEPR and Stanford SEDA
Notes: All estimates are based on published state assessment results, which have been rescaled to grade
equivalents using state scores on the National Assessment of Educational Progress.
For historical comparability, the proportion of students receiving free or reduced price lunch reflects the
estimated number of students in households with incomes below 185% of the federal poverty level in Census
data.
Some districts may have higher rates of federally subsidized lunch recipients due to the community eligibility
provision. The sample of districts shown have been limited to districts with reliable estimates of test score
changes.
Blue points represent districts with at least 800 tested students per grade. The regression line displays the
overall trend within the state. For details on methodology, see https://edopportunity.org/methods.
Figure 9 reports a very different pattern of recovery in the state of Massachusetts. In
Massachusetts, the recovery between 2022 and 2023 was led by the wealthier districts, such as
Lexington, Brookline, Newton and Wachusett. The higher poverty districts in the state, such as Lynn,
Fall River and Revere lost more ground—their students declined in math achievement—between
The First Year of Pandemic Recovery: A District-Level Analysis 13
2022 and 2023. As a result, the difference in achievement loss for high and low poverty districts was
larger over the 2019 to 2023 period than it was between 2019 and 2022. In other words, even
though there has been some recovery, the gaps between high and low poverty districts were even
wider in 2023 than in 2022, immediately after the pandemic.
FIGURE 9: MATH SCORE CHANGES IN MASSACHUSETTS DISTRICTS,
2019-2023, BY PERIOD
Source: Education Recovery Scorecard, by Harvard CEPR and Stanford SEDA
Notes: All estimates are based on published state assessment results, which have been rescaled to grade
equivalents using state scores on the National Assessment of Educational Progress.
For historical comparability, the proportion of students receiving free or reduced price lunch reflects the
estimated number of students in households with incomes below 185% of the federal poverty level in Census
data.
Some districts may have higher rates of federally subsidized lunch recipients due to the community eligibility
provision. The sample of districts shown have been limited to districts with reliable estimates of test score
changes.
Blue points represent districts with at least 600 tested students per grade. The regression line displays the
overall trend within the state. For details on methodology, see https://edopportunity.org/methods.
To get a sense of how the change in the gaps between high-and low-poverty districts differs among
states, we report in Figure 10 the average change between 2019 and 2023 in high- and low-poverty
districts in each state. [8] In eight states—Massachusetts, Ohio, Kentucky, Oregon, Rhode Island,
Tennessee, Connecticut and Indiana—the 2023 gap in math scores between high- and low-income
[8] Four states—Louisiana, Mississippi, New Hampshire and Nevada—are not shown because they either have no
districts with fewer than 25 percent or have no districts with more than 75 percent free/reduced-price lunch eligible
students in 2019.
The First Year of Pandemic Recovery: A District-Level Analysis 14
districts is more than half a grade level larger than in 2019. In 6 states—Connecticut, Indiana,
Massachusetts, Virginia, Ohio and Wyoming—the gap in reading scores between high- and low-
income districts grew by more than half a grade level from 2019 to 2023. Four states—
Massachusetts, Ohio, Connecticut and Indiana—were on both lists with the largest widening of gaps
in math and reading. Appendix Figures A3 and A4 show the changes in these gaps from 2019-2022
and 2022-2023, respectively.
FIGURE 10: CHANGE IN ACHIEVEMENT GAP BETWEEN HIGH- AND
LOW-POVERTY DISTRICTS, 2019-2023, BY STATE
However, the widening gaps between rich and poor districts tell only part of the story. In some
states, achievement gaps also widened between poor and non-poor students within the same
districts; in other states they narrowed. So the between-district change in gaps does not necessarily
capture the change in the test score gap between poor and non-poor students, which could be
occurring between districts as well as within districts. Fourteen states provided math achievement
data for subgroups of students in each district (fifteen in reading). For these states, Figure 11
reports the statewide change in the test score gap between poor and non-poor students from 2019-
2023. The states with the greatest widening in math achievement gaps between poor and non-poor
students were Massachusetts, Connecticut, and Michigan. [9]
[9] Some states, such as a Massachusetts, changed their criteria for categorizing a student as low-income or
economically disadvantaged between 2019 and 2023. This has several implications for trends in test scores and test
score gaps. In general, if the income threshold defining “low-income” is raised (that is, allowing students with slightly
higher incomes to qualify), this will add more middle-income students to the “low-income” category and remove them
The First Year of Pandemic Recovery: A District-Level Analysis 15
In three states, the poor/non-poor gap in math achievement was slightly smaller in 2023 than in
2019: South Dakota, Louisiana, and Kentucky. Massachusetts and Michigan were also among the
top three states with the greatest widening in reading achievement gaps.
FIGURE 11: CHANGE IN NONPOOR-POOR TEST SCORE GAP, 2019-
2023, BY STATE
from the “not-low-income” category. Because average test scores are higher, on average, for higher-income students,
adding middle income students to the low-income category will tend to raise the reported average scores for “low-
income” students (because the added students will tend to have higher scores than the students originally in the low-
income category). It will also tend to raise the reported average scores for “high-income” students (because the
removed students will tend to have lower scores than the remaining students). In other words, changing the income
threshold used to define the “low-income” will move both the low- and the high-income category scores in the same
direction. If average scores are a linear function of students’ income percentile rank within a state (which is generally a
good rough approximation), then the two group’s average scores will move in the same direction and by the same
amount, leaving the measured difference in scores between the high- and low-poverty groups unchanged. In other
words, changing the definition of “low-income” will affect interpretations of each group’s trend in scores, but will leave
the gap between them generally unchanged. So while one should interpret changes in achievement and achievement
gaps with some caution when the definition of low-income changes (as in Massachusetts), the large reported change in
the non-poor achievement gap shown for Massachusetts in Figure 11 likely reflects a real and substantial widening of
the gap in that state. In Appendix Figures A5 and A6, we show the change in the poor/non-poor gaps separately for
2019-22 and for 2022-23. Massachusetts implemented the new definition of “low income” students in the 2021-22
school year. If the change in the definition did have any effect it would have been in the 2019-22 period when the
measured gap expanded by 0.2 grade levels in math and 0.31 grade levels in reading. However, between 2022 and
2023, when the subgroup definition remained the same, the Massachusetts gap grew by slightly more (0.23 grade levels
in math and 0.31 in reading). In both years, Massachusetts was among the states with the largest increases. The
between-district comparisons in Figure 10, where Massachusetts also showed large widening, are unaffected by the
subgroup redefinitions.
The First Year of Pandemic Recovery: A District-Level Analysis 16
Within-District Gaps
In the spring of 2023, we wrote a report exploring the school district and community factors
associated with larger achievement losses (Fahle et al, 2023). We identified several district and
community-level factors which seemed to play a role. For instance, higher poverty districts stayed
closed longer during the 2020-21 school year and, for each week of closure, their students lost more
ground than in wealthier communities. Communities with less access to broadband internet, with
greater disruptions to social and economic activity (as measured by Google’s data on declines in
time spent at retail, recreation establishments and grocery stores, cell phone activity in restaurants,
respondents to a Facebook survey reporting going to market/pharmacy/grocery, visiting a
bar/restaurant/café in the last 24 hours, seeing anyone outside one’s house or attending large
events), with weaker trust in community institutions (as reflected by voter participation), and with
higher COVID mortality rates all suffered larger losses in achievement.
One striking finding was that, within school districts, White and non-poor students lost about the
same amount of ground as Black, Hispanic, and economically disadvantaged students in the same
districts. In other words, when a district declined in achievement, subgroups all lost about the same
amount of ground. Poor students lost more ground because they were concentrated in districts
which lost ground, not because they fell behind their richer peers in the same school district. We
interpreted this as implying that the cause of the losses were community-level and district-level
factors such as those discussed above, rather than differences in household resources, which would
have varied between rich and poor students attending the same school district.
During the recovery, though, just the opposite occurred. Within the average district in the 15 states
for which we have data by economic status, non-poor students showed faster recovery – roughly
twice faster – than poor students. And poor and non-poor districts in the same state experienced
roughly similar average recovery rates. The net result was that the nonpoor-poor achievement gap
has widened in most states since 2019, but that widening was due to between-district differences in
achievement declines from 2019 to 2022 and to within-district differences in achievement increases
from 2022 to 2023. Overall, roughly 60-70% of the widening from 2019-2023 is due to between-
district differences in test score declines. We do not know the reason for this, but it is troubling.
Even as student achievement has improved rapidly since 2022, those gains have not been equally
shared, even within the same school district.
ESSER Spending
Over the course of the pandemic, the federal government provided three packages of supplemental
funding for K-12 education, totaling $190 billion. The dollars were distributed not on the basis of
learning loss—which was still unknown in March 2021 when the American Rescue Plan legislation
passed—but using the federal Title I formula, which is targeted on the basis of district poverty. The
highest poverty districts (those with federal subsidized lunch participation between 90 and 100
percent) received approximately $8000 per student in pandemic relief aid, while the lowest poverty
The First Year of Pandemic Recovery: A District-Level Analysis 17
districts received much less, roughly $500 per student.
Figure 12 reports the average amount of ESSER funding allocated to districts and the average
amount spent by Fall 2023, by district poverty rates. Roughly a third of the ESSER funds remained
unspent as of Fall 2023. While high-income districts had roughly 20 percent of their funds
remaining, the lowest income districts had nearly 40 percent of the total pandemic relief funding
still available in the fall of this school year.
FIGURE 12: TOTAL ESSER FUNDS PER STUDENT ALLOCATED AND
SPENT, BY DISTRICT POVERTY RATE
One of the missed opportunities over the last three years was the lack of uniform guidance on the
way districts were expected to track academic recovery efforts. We wish we could say more about
how districts spent the funds and the relative efficacy of different types of spending. Unfortunately,
states report district spending with different levels of detail, in different categories, and on different
time frames.
We also cannot say yet what the impact of total ESSER funding was on academic recovery. The
answer to that question requires a careful analysis that disentangles the effects of ESSER spending
from the effects of other factors related to allocation of ESSER dollars, such as district poverty rates.
We are currently working on such an analysis and will report our findings in our next report.
The First Year of Pandemic Recovery: A District-Level Analysis 18
Completing the Recovery
The federal funds must be obligated by the end of September— roughly 8 months from now [10].
Given the magnitude of the achievement gaps remaining, it is especially important that districts use
that funding wisely to help the neediest students catch up.
We have four recommendations for actions that state and local agencies to undertake over the next
8 months:
1. This spring, schools should be required to inform parents if their child is below grade level
in math or English to give parents time to enroll in summer learning before the federal relief
expires.
Surveys have consistently shown (Pew Research Center, 2022, Peterson, et al., 2022, Policy
Analysis for California Education/USC Poll, 2021) that parents underestimate the impact of
the pandemic on their children’s achievement. Parents cannot advocate effectively for their
children’s future if they are misinformed.
2. Schools should expand summer learning seats this summer. States should require districts to
set aside sufficient funds to accept all students who sign up.
Research has shown that six weeks of summer learning produces the equivalent of a quarter
of year of learning, especially in math (Augustine et. al., 2016). Programs which combine
academic instruction with enriching activities, such as art, sports or outdoor activities will be
most effective in drawing students to summer programming and ensuring regular
attendance. However, district staff may be hard-pressed to plan both enrichment activities
and academic instruction. That is why programs such as the Boston After-School and Beyond
program [11] allows organizations such as summer camps and zoos and science museums
plan the enrichment activities, while the district plans the academic content. The school
[10] Under the American Rescue Plan legislation, the federal relief dollars must be “obligated” by September 30, 2024,
or returned to the federal Treasury. Changing the obligation deadline would require an act of Congress, which is unlikely
given the current stalemate in Washington. The U.S. Department of Education, recognizing districts’ desire (and
students’ need) to extend recovery efforts into next year has laid out a process (https://oese.ed.gov/files/2024/01/ARP-
Liquidation-Extension-Letter-1.9.24-final-for-signature-v3.pdf) by which districts can request an extension of the
deadline for liquidating funds through March 2026 (https://oese.ed.gov/files/2024/01/Updated-Technical-FAQs-for-
Liquidation-Extensions-1.9.24-v-2-for-posting.pdf)—16 months after the obligation deadline. But there is an important
catch: under federal regulation (34 CFR 76.707) (https://www.ecfr.gov/current/title-34/part-76/section-76.707),
payments to an external contractor are obligated on the date on which a binding agreement is signed, while payments
to school district employees are only obligated on the date they are paid. Therefore, school districts can use the federal
relief dollars to pay for academic recovery through next school year, Summer 2025 and even the fall semester of 2025,
but only if they sign a contract before September 2024 to do so (and request the extension by December 2024.) The
federal guidance explicitly mentions contracts for tutoring, after school and summer learning as potentially eligible for
the extension. But districts cannot expect to use the federal dollars to pay the salaries of teachers or other school
employees after September 2024.
[11] See https://bostonbeyond.org/.
The First Year of Pandemic Recovery: A District-Level Analysis 19
district offers financial incentives for such programs to make room in their schedules for
academic programs. The school district provides the academic component (curriculum and even
the instructors). It is a promising model that other districts seeking to expand summer learning
opportunities should consider.
3. Districts can extend the recovery efforts into the next school year by contracting for tutoring
and after-school programs before September.
Even if districts maintain the same pace of academic recovery this year as they did last year, the
recovery will not be complete by September. Although the federal relief dollars cannot be used
to pay school employee salaries after September, they can be used to make payments on
contracts that are signed before the deadline. Thus, the only way to use the federal dollars to
extend the recovery into next year will be by contracting with external providers such as tutors,
summer school and after-school programs. The federal guidance explicitly invites districts to
apply for extensions to use the relief funds for such programs. [12]
4. Local government, employers and community leaders can help schools reduce student
absenteeism, which has doubled since the pandemic.
Elected officials, employers, and community leaders should launch public awareness campaigns
and other initiatives to lower student absenteeism. Absent students not only miss out on
learning time while they are out, but they also make it hard for teachers to keep the whole class
moving when they return.
Conclusion
Although U.S. students made historically large gains in achievement last year, average achievement
in most states remains behind 2019 levels. Moreover, the recovery so far has not closed the
achievement gaps which widened during the pandemic. In fact, in many states, the recovery during
2022-23 was being led by wealthier districts, meaning that the achievement gaps are getting even
wider. Thus, even in the few states where average achievement has returned to pre-pandemic
levels, students in lower income districts remained further behind in 2023 than they were in 2019.
Few would be content to know that poor children paid a higher price for the pandemic than others
—but that is exactly the path many states are on. In the midst of the pandemic, districts had many
needs to address with the pandemic relief dollars—masking supplies, computers, mobile hot spots
and HVAC systems capable of reducing the spread of the virus. With the spending deadline
approaching, their options for extending the academic recovery beyond this school year have
[12] The U.S. Department of Education’s recent guidance on seeking an extension can be found at
https://oese.ed.gov/offices/office-state-grantee-relations-evidence-based-practices/state-and-grantee-
relations/deadlines-and-announcements/. For ideas on how to tie contractor payments to student outcomes, see the
Outcomes-Based Contracting project at the Southern Education Foundation at https://obc.southerneducation.org/.
The First Year of Pandemic Recovery: A District-Level Analysis 20
narrowed. They can notify parents of children who are behind grade level and expand programming
this summer and they can contract for tutoring and after school services next year (and even
contract for summer learning in Summer 2025.) Unfortunately, leaders will have to make these
decisions without the benefit of the spring 2024 assessment data, which, in many states, will not be
available until after the federal spending deadline. By providing these data, we hope to allow for a
more realistic assessment of the lost ground remaining at the end of 2023 and the likelihood that
students will have caught up by this spring. In most districts, that seems unlikely. We hope that
district leaders will use these data to inform their plans to spend the remaining federal relief dollars
to best support their students’ academic needs.
Last year, students made up one-third of the pandemic loss in math and one-quarter of the loss in
reading. Although good news, it also means that even if schools maintain the same pace this year,
students, especially in lower-income districts, are unlikely to have returned to 2019 levels of
achievement when the federal dollars are gone. Governors and state legislators, who have largely
been observers as local districts have decided how to spend the federal relief dollars, may need to
take a more active role if the recovery is to be completed successfully.
The First Year of Pandemic Recovery: A District-Level Analysis 21
References
Augustine, Catherine H., Jennifer Sloan McCombs, John F. Pane, Heather L. Schwartz, Jonathan
Schweig, Andrew McEachin, and Kyle Siler-Evans (2016). Learning from Summer: Effects of
Voluntary Summer Learning Programs on Low-Income Urban Youth. RAND Corporation.
https://www.rand.org/pubs/research_reports/RR1557.html.
Barnum, Matt and Kalyn Belsha (2023, October 2). “Blizzard of state test scores shows some
progress in math, divergence in reading” Chalkbeat.
https://www.chalkbeat.org/2023/10/2/23896045/state-test-scores-data-math-reading-pandemic-
era-learning-loss/
Curriculum Associates (2023). “State of Student Learning in 2023”
https://cdn.bfldr.com/LS6J0F7/at/x8v8wp2c6j4s4wttsw2nwphb/ca-state-of-student-learning-
technical-report-2023.pdf
Dee, Thomas. S. (2023, August 10). “Higher Chronic Absenteeism Threatens Academic Recovery
from the COVID-19 Pandemic.” OSF Preprints. August 10. https://doi.org/10.31219/osf.io/bfg3p
Fahle, Erin M., Thomas J. Kane, Tyler Patterson, Sean F. Reardon, Douglas O. Staiger, and Elizabeth
A. Stuart (2023). “School District and Community Factors Associated with Learning Loss During the
COVID-19 Pandemic” Available at
https://cepr.harvard.edu/sites/hwpi.harvard.edu/files/cepr/files/explaining_covid_losses_5.23.pdf.
Fahle, Erin M., Sean F. Reardon, Benjamin R. Shear, Andrew D. Ho, Jie Min, and Demetra Kalogrides
(2024). Stanford Education Data Archive Technical Documentation: SEDA 2023. Available at
https://edopportunity.org/methods/.
Ho, Andrew D. (2008). “The problem with ‘proficiency’: Limitations of statistics and policy under No
Child Left Behind.” Educational Researcher, 37, 351-360.
https://doi.org/10.3102/0013189X08323842.
Jackson, C. Kirabo, and Claire L. Mackevicius (2024). "What Impacts Can We Expect from School
Spending Policy? Evidence from Evaluations in the United States." American Economic Journal:
Applied Economics, 16 (1): 412-46. https://doi.org/10.1257/app.20220279.
Jordan, Phyllis and Bella DiMarco (May 2022). “School Districts’ Post-Covid Strategies for Summer
Learning” https://www.future-ed.org/school-districts-post-covid-summer-learning-strategies/
The First Year of Pandemic Recovery: A District-Level Analysis 22
Jordan, Phyllis, Bella DiMarco and Thomas Toch (April 2022). “An Analysis of Local School Districts’
Ambitious Post-Covid Tutoring Plans” https://www.future-ed.org/an-analysis-of-local-school-
districts-ambitious-post-covid-tutoring-plans/.
Jordan, Phyllis, Anne Dwyer, Bella DiMarco, and Margaux Johnson-Green (June 2022). “How
Medicaid Can Help Schools Sustain Support for Students’ Mental Health” https://www.future-
ed.org/how-medicaid-can-help-schools-sustain-support-for-students-mental-health/
Lewis, Karen and Megan Kuhfeld (2023). “Education’s long COVID: 2022–23 achievement data
reveal stalled progress toward pandemic recovery.”
https://www.nwea.org/research/publication/educations-long-covid-2022-23-achievement-data-
reveal-stalled-progress-toward-pandemic-recovery/.
Peterson, Paul, David M. Houston, and Martin R. West (2022). “Parental Anxieties over Student
Learning Dissipate as Schools Relax Anti-Covid Measures” Education Next, 23(1):20.
https://www.educationnext.org/parental-anxieties-over-student-learning-dissipate-as-schools-
relax-anti-covid-measures-2022-education-next-survey-public-opinion/
Pew Research Center American Trends Panel (September 2022). Survey of American Parents, Wave
115. Available at https://www.pewresearch.org/wp-content/uploads/2022/10/W115-
topline_COVID-impact-on-K-12-new.pdf.
Policy Analysis for California Education/USC Poll (2021). Available at
https://edpolicyinca.org/sites/default/files/2021-07/pace_usc_poll_ca_statewide_235-
l_fq_parents_final.pdf.
Reardon, Sean F., Demetra Kalogrides, and Andrew D. Ho (2021). Validation methods for aggregate-
level test scale linking: A case study mapping school district test score distributions to a common
scale. Journal of Educational and Behavioral Statistics 46(2):135-137.
https://doi.org/10.3102/1076998619874089.
Reardon, Sean F., Benjamin R. Shear, Katharine E. Castellano, and Andrew D. Ho (2017). “Using
Heteroskedastic Ordered Probit Models to Recover Moments of Continuous Test Score Distributions
from Coarsened Data.” Journal of Educational and Behavioral Statistics 42(1):3-45.
https://doi.org/10.3102/1076998616666279.
The First Year of Pandemic Recovery: A District-Level Analysis 23
Appendix
FIGURE A1: NATIONAL TRENDS IN MATH AND READING
ACHIEVEMENT, 1990-2023
FIGURE A2: NATIONAL TRENDS IN RACIAL AND ECONOMIC
ACHIEVEMENT GAPS, 2009-2023
The First Year of Pandemic Recovery: A District-Level Analysis 24
Appendix (cont.)
FIGURE A3: CHANGE IN ACHIEVEMENT GAP BETWEEN HIGH-AND
LOW-POVERTY DISTRICTS, 2019-2022, BY STATE
FIGURE A4: CHANGE IN ACHIEVEMENT GAP BETWEEN HIGH-AND
LOW-POVERTY DISTRICTS, 2022-2023, BY STATE
The First Year of Pandemic Recovery: A District-Level Analysis 25
Appendix (cont.)
FIGURE A5: CHANGE IN NONPOOR-POOR TEST SCORE GAP, 2019-
2022, BY STATE AND SUBJECT
FIGURE A6: CHANGE IN NONPOOR-POOR TEST SCORE GAP, 2022-
2023, BY STATE AND SUBJECT
The First Year of Pandemic Recovery: A District-Level Analysis 26
5
File and source
- File
- the-first-year-of-pandemic-recovery-a-district-level-analysis-educatio-d59111eb.pdf
- Size
- 5,693,489 bytes
- SHA-256
- d59111ebf22d544e89388c34626ff9be18bd40ad5761154699b2a77b66b1adea
- Original
- No public link identified.