COVID-19 Modeling (GAO-20-582SP)

Summary

A GAO Science & Tech Spotlight on COVID-19 modeling, GAO-20-582SP, dated June 2020, from GAO's Science, Technology Assessment, and Analytics team. It explains that infectious disease models are mathematical representations that help researchers analyze the dynamics of a disease and have been used in the COVID-19 response to project new infections, deaths and the need for health care resources. It contrasts models built on disease biology or behavior with statistical models that use only data, such as the model developed by the Institute for Health Metrics and Evaluation (IHME). It cautions that limitations in data inputs and assumptions can lead to considerable uncertainty in model estimates.

Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used

Full text

Science, Technology Assessment,
JUNE 2020
and Analytics
WHY THIS MATTERS
Infectious disease models can help guide major
policy decisions, such as how to allocate health care
SCIENCE & TECH SPOTLIGHT: resources in the COVID-19 response. However,

COVID-19 MODELING
limitations in data inputs and assumptions can lead
to considerable uncertainty in model estimates.
Interpreting these estimates requires understanding
their purpose, limitations, and assumptions.

/// THE SCIENCE Statistical models, by contrast, use only data and not disease biology or
behavior. For example, they might use data on reported deaths to forecast
What is it? Infectious disease models are mathematical representations future deaths or hospital needs.
that help researchers analyze the dynamics of a disease. They have
played a prominent role in the COVID-19 response in the United States The Institute for Health Metrics and Evaluation (IHME) developed a
and abroad, for example in projecting new infections, deaths, and the statistical model that forecasts COVID-19 infections, deaths, hospital
potential need for health care resources. needs (e.g., beds or ventilators), and when states may be able to relax
social distancing. To do so, researchers use data on, among other things,
How does it work? The equations used in a model are based on deaths, how the disease spread in other locations, and when states
biological knowledge, data on the disease, or both. Models can be put into implemented social distancing.
two broad categories: mechanistic and statistical (fig. 1).
Models may also combine statistical and mechanistic elements. For
example, in May 2020, researchers added an SEIR component to the
IHME model to quantify how people move through the phases of disease.

Each model has a distinct purpose that must be considered when
interpreting its results. For example, statistical models may be designed to
project the need for hospital beds or ventilators, based on the assumption
that past trends will continue. But if human behavior changes—for
instance, if social distancing is relaxed—then the forecasts are likely to be
less accurate. Similarly, mechanistic models may be designed to provide
insight on hypothetical scenarios, in which case their results are useful for
comparing scenarios, but not as predictions.

How mature is it? The field of infectious disease modeling is well
established, but the maturity of a specific model depends upon the quality,
completeness, and accuracy of the data. Early in an outbreak, data and
knowledge of the disease may be scarce (as is the case with COVID-19),
and model predictions are unlikely to be accurate. However, as data and
knowledge of the disease become more available, the model matures as
its predictions can become more accurate and precise (fig. 2). Models can
Figure 1. There are generally two broad categories of infectious disease models: still be useful early in an outbreak because they can provide quantitative
mechanistic models, which use scientific understanding of disease dynamics and human estimates to help decision makers respond to a range of possible
behavior, and statistical models, which rely only on patterns in the data. scenarios. This has been the case for the COVID-19 pandemic.
Mechanistic models use equations to represent the mechanics of how
Models can also be useful even when they do not provide an accurate
a disease progresses, based on scientific understanding of disease
estimate of the final results. For example, the Imperial College model
dynamics and human behavior. Researchers can estimate the effect
explored several scenarios, including one without intervention, which
of proposed interventions, such as social (i.e., physical) distancing, by
projected over 2 million U.S. deaths by October 2020. This estimate is no
adjusting the model and rerunning it.
longer useful as a prediction because that scenario did not occur, but it did
One widely used mechanistic model is the susceptible-exposed-infectious- prove helpful in showing the potential benefit of early intervention.
recovered (SEIR) model. It describes how a population moves through the
stages of a disease. For example, movement from susceptible to exposed
can be based on the rate of infections. For COVID-19, Imperial College
London built an SEIR-based model that refines this rate of movement
using population size, age distribution, and social contact patterns. The
model projections based on this rate and other inputs can help estimate
disease impacts.

GAO-20-582SP COVID-19 Modeling
Science, Technology Assessment,
and Analytics

■ Communication. Models can be complex and thus difficult to
explain to lay audiences. As a result, their methods and outputs may
be oversimplified and thus misunderstood. For example, the public
may assume model outputs are intended as predictions, when in
some cases they illustrate what-if scenarios that may not occur.

/// POLICY CONTEXT AND QUESTIONS
The British statistician George Box once said, “All models are wrong, but
some are useful.” In other words, all outbreak models simplify reality but
can still support decisions and improve our understanding of a disease.
This context raises many policy questions, such as:
Figure 2. During the outbreak of a new disease, models can be most helpful early in the
response, but are most limited by a lack of data. Later in the outbreak, more data becomes ■ How can modelers and policymakers collaborate to identify the most
available, but there is less time to implement an optimal response for ending the outbreak. appropriate and reliable models to consider for decision-making?

/// OPPORTUNITIES ■ What steps could improve data quality, data sharing, and
transparency for models used to inform decisions?
When used and interpreted correctly, models can provide: ■ How can model results be better communicated to the public?
■ Support for early response. Models can inform decision-making ■ How can agencies further coordinate with each other and with
early in an outbreak, when data and knowledge of the disease are academic and other modelers?
limited. For example, they can help prepare for worst-case hospital
bed needs and show which steps might best slow the outbreak. /// SELECTED GAO WORK
■ Scenario comparison. Models can provide a range of scenarios In addition to two forthcoming reports on modeling and ongoing work
based on the potential behavior of the disease, helping policymakers related to COVID-19, GAO has previously issued the following products:
prepare for multiple outcomes or identify the most likely.
■ Science & Tech Spotlight: Coronaviruses, GAO-20-472SP.
■ Evaluation of options. Models can estimate the possible impact
of different policy decisions, such as relaxing social distancing or ■ Science & Tech Spotlight: Social Distancing During Pandemics,
accelerating the development of a vaccine. GAO-20-545SP.
■ Science & Tech Spotlight: COVID-19 Vaccine Development,
/// CHALLENGES
GAO-20-583SP.
Models are only as good as the data and assumptions that go into them. ■ Science & Tech Spotlight: COVID-19 Testing, GAO-20-584SP.
The following issues may make results less precise and harder to explain:
/// SELECTED REFERENCES
■ Data. Models are inherently limited by the underlying data, which
can be scarce and inconsistent, especially early in a novel disease Dembek Z.F., T. Chekol, and A. Wu. 2018. “Best practice assessment of disease modelling
outbreak, lessening the precision and accuracy of model estimates. for infectious disease outbreaks.” Epidemiology and Infection, vol. 146, no. 10 (2018): pp.
1207–1215.
■ Uncertainty. Especially in the early stages of a novel infectious
disease outbreak, there are multiple sources of uncertainty, such Department of Health and Human Services. Centers for Disease Control and Prevention.
as overreliance on a few data points and lack of information on “COVID-19 Forecasts.” In Coronavirus Disease 2019 (COVID-19). Atlanta, Ga.: Center for
behavioral changes that might occur among the population. Disease Control and Prevention, 2020. Accessed May 22, 2020.
https://www.cdc.gov/coronavirus/2019-ncov/covid-data/forecasting-us.html.

GAO SUPPORT:
This document is not an audit product and is subject to revision based on continued
GAO meets congressional information needs in several ways, including by
advances in science and technology. It contains information prepared by GAO
providing oversight, insight, and foresight on science and technology issues.
to provide technical insight to legislative bodies or other external organizations.
GAO staff are available to brief on completed bodies of work or specific reports
This document has been reviewed by the Chief Scientist of the U.S. Government
and answer follow-up questions. GAO also provides targeted assistance on
Accountability Office.
specific science and technology topics to support congressional oversight activities
and provide advice on legislative proposals. This is a work of the U.S. government and is not subject to copyright protection in
Timothy M. Persons, PhD, Chief Scientist, personst@gao.gov the United States. The published product may be reproduced and distributed in its
entirety without further permission from GAO. However, because this work may
Staff Acknowledgments: Karen Howard, PhD (Director); Sushil Sharma, PhD,
contain copyrighted images or other material, permission from the copyright holder
DrPH (Assistant Director); Laura Holliday (Assistant Director); Chi Mai, PhD
may be necessary if you wish to reproduce this material separately.
(Analyst-in-Charge); Anika McMillon; Jim Rice, PhD; Ben Shouse; and
Amber Sinclair, PhD.

GAO-20-582SP COVID-19 Modeling

File and source

File: gao-20-582sp.pdf
Size: 353,478 bytes
SHA-256: 3903c6bbe9a60a1f9aab8533efc643cdab0ea77c68a1e40a4697af06478190de
Our copy: gao-20-582sp.pdf
Original: www.gao.gov