Opportunities to Improve Coordination and Ensure Reproducibility (GAO-20-372)
Summary
A GAO report to congressional requesters, GAO-20-372, dated May 2020, on infectious disease modeling at the Department of Health and Human Services (HHS). It reviews how the Centers for Disease Control and Prevention (CDC) and the Office of the Assistant Secretary for Preparedness and Response (ASPR) used models to inform decision-making during and after outbreaks of Ebola, Zika and pandemic influenza. GAO reports that four CDC centers perform modeling. It finds that HHS agencies use multiple mechanisms to coordinate modeling but do not routinely monitor, evaluate or report on the extent and success of that coordination.
Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used
Full text
United States Government Accountability Office
Report to Congressional Requesters
INFECTIOUS
May 2020
DISEASE MODELING
Opportunities to
Improve Coordination
and Ensure
Reproducibility
GAO-20-372
May 2020
INFECTIOUS DISEASE MODELING
Opportunities to Improve Coordination and Ensure
Reproducibility
Highlights of GAO-20-372, a report to
congressional requesters
Why GAO Did This Study What GAO Found
Outbreaks of infectious diseases—such Within the Department of Health and Human Services (HHS), the Centers for
as Ebola, Zika, and pandemic Disease Control and Prevention (CDC) and the Office of the Assistant Secretary
influenza—have raised concerns from for Preparedness and Response (ASPR) used models to inform decision-making
Congress about how federal agencies during and after outbreaks of Ebola, Zika, and pandemic influenza. These
use modeling to, among other things, agencies’ modeling efforts informed public health planning, outbreak response,
predict disease distribution and potential and, to a limited extent, resource allocation. Four CDC centers perform modeling.
impacts. In general, a model is a
representation of reality expressed HHS agencies reported using multiple mechanisms to coordinate modeling
through mathematical or logical efforts across agencies, but they do not routinely monitor, evaluate, or report on
relationships. Models of infectious the extent and success of coordination. Consequently, they risk missing
diseases can help decision makers set opportunities to identify and address modeling challenges—such as
policies for disease control and may communicating clearly, and obtaining adequate data and resources—before and
help to allocate resources. during an outbreak. As a result, agencies may be limiting their ability to identify
GAO was asked to review federal
improvements in those and other areas. Further, there is potential for overlap and
modeling for selected infectious duplication of cross-agency modeling efforts, which could lead to inefficiencies.
diseases. This report examines (1) the Office of the Assistant Secretary for Preparedness and Response’s Visualization Hub, which
extent to which HHS used models to Can Be Used for Infectious Disease Planning and Response
inform policy, planning, and resource
allocation for public health decisions; (2)
the extent to which HHS coordinated
modeling efforts; (3) steps HHS
generally takes to assess model
development and performance; and (4)
the extent to which HHS has addressed
challenges related to modeling. GAO
reviewed documents and interviewed
HHS officials, state officials, and subject
matter experts. GAO identified practices
commonly used to assess infectious
disease model performance and
reviewed 10 selected modeling efforts to
see if they followed these practices.
What GAO Recommends
GAO recommends that HHS (1) develop
a way to routinely monitor, evaluate, and
report on modeling coordination efforts
across multiple agencies and (2) direct CDC and ASPR generally developed and assessed their models in accordance
CDC to establish guidelines to ensure with four steps GAO identified as commonly-recognized modeling practices: (1)
full reproducibility of its models. HHS communication between modeler and decision maker, (2) model description, (3)
agreed with GAO’s recommendations. verification, and (4) validation. However, for four of the 10 models reviewed, CDC
did not provide all details needed to reproduce model results, a key step that lets
other scientists confirm those results. GAO found that CDC’s guidelines and
policy do not address reproducibility of models or their code. This is inconsistent
with HHS guidelines and may jeopardize the reliability of CDC’s research.
View GAO-20-372. For more information, This report also identifies several modeling-related challenges, along with steps
contact Timothy M. Persons, PhD, Chief agencies have taken to address them.
Scientist at (202) 512-6888 or
personst@gao.gov.
______________________________________ United States Government Accountability Office
Contents
Letter 1
Background 6
HHS Has Used Infectious Disease Models to Help Inform Policy
and Planning 16
Agencies Coordinate Infectious Disease Modeling Efforts but Do
Not Fully Monitor, Evaluate, and Report on Coordination 25
CDC and ASPR Generally Followed Identified Practices for
Infectious Disease Modeling, but CDC Has Not Fully Ensured
Model Reproducibility 37
Modelers Faced Several Challenges and Have Worked to
Address Them 45
Conclusions 51
Recommendations for Executive Action 52
Agency Comments and Our Evaluation 53
Appendix I Objectives, Scope, and Methodology 55
Appendix II Bibliography of Selected Model Publications Reviewed 62
Appendix III Ten Selected Infectious Disease Models and
Questions from Data Collection Instrument 67
Appendix IV Comments from the Department of Health
and Human Services 71
Appendix V GAO Contacts and Staff Acknowledgments 73
Tables
Table 1: Examples of National Plans Identifying Modeling 16
Table 2. Selected Examples of Working Groups for Infectious
Disease Modeling 28
Page i GAO-20-372 Infectious Disease Modeling
Table 3. Selected Examples of Memoranda of Understanding for
Coordinating on Infectious Disease Modeling 29
Table 4: Elements of Model Development 39
Table 5: Documents Describing Models Evaluated in our Data
Collection Instrument by Agency and Disease 67
Figures
Figure 1: Timeline of Ebola outbreaks since 2014 10
Figure 2: Office of the Assistant Secretary for Preparedness and
Response’s (ASPR) Visualization Hub 18
Figure 3: Timeline of Data Availability for Models Compared to
Usefulness of Modeling During an Outbreak. 22
Figure 4: Outline of Process to Develop Models and Assess Their
Performance. 38
Abbreviations
ASPR Office of the Assistant Secretary for Preparedness and
Response
CDC Centers for Disease Control and Prevention
DRC Democratic Republic of the Congo
Ebola Ebola virus disease
EOC Emergency Operations Center
FDA Food and Drug Administration
HHS Department of Health and Human Services
NIH National Institutes of Health
NSTC National Science and Technology Council
PHEMCE Public Health Emergency Medical Countermeasures
Enterprise
Zika Zika virus disease
This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.
Page ii GAO-20-372 Infectious Disease Modeling
Letter
441 G St. N.W.
Washington, DC 20548
May 13, 2020
The Honorable Frank Pallone Jr.
Chairman
The Honorable Greg Walden
Republican Leader
Committee on Energy and Commerce
House of Representatives
The Honorable Diana DeGette
Chairwoman
Subcommittee on Oversight and Investigations
Committee on Energy and Commerce
House of Representatives
Today’s globalized economy and transportation systems allow infectious
diseases to spread more rapidly than ever. Notable outbreaks include
novel coronavirus beginning in 2019, Zika virus disease (Zika) in 2015,
Ebola virus disease (Ebola) in 2014, and H1N1 pandemic influenza in
2009. Disease outbreaks can cause catastrophic harm to the United
States, disrupt economic and social systems, and kill, sicken, and
traumatize people on a massive scale. For example, approximately 1
billion people worldwide get sick annually from zoonotic pathogens—
pathogens that can spread from animals to humans—of which,
approximately 15 million people die. Such outbreaks are on the rise. The
latest example is the novel coronavirus disease which had, as of May 6,
2020, caused approximately 250,000 deaths worldwide and sickened
approximately 3,600,000 people. 1 In the United States, the virus had
caused approximately 63,000 deaths, and sickened approximately
1,200,000 people. The situation has heightened U.S. attention to potential
future infectious disease threats and raised questions about the nation’s
preparedness and response capabilities. It has also raised concerns
1These numbers represent case counts as provided to the World Health Organization on
May 6, 2020. According to the World Health Organization, they are likely an
underestimation of the true number of cases because of differences in testing strategies,
definitions, reporting practices, and other factors between countries, territories, and areas.
According to the Centers for Disease Control and Prevention (CDC), the exact number of
novel coronavirus disease illnesses, hospitalizations, and deaths is unknown for a variety
of reasons: it can cause mild illness, symptoms might not appear immediately, there are
delays in reporting and testing, not everyone who is infected gets tested or seeks medical
care, and there may be differences in how states and territories confirm numbers in their
jurisdictions. The novel coronavirus pandemic began in the United States near the end of
our review period; as such, we do not discuss it in this report.
Page 1 GAO-20-372 Infectious Disease Modeling
among some members of Congress about how federal agencies predict
the spread of emerging infectious diseases, in particular through the use
of modeling.
A model is a simplified representation of reality expressed through
mathematical or logical relationships. Modeling is widely used in fields as
diverse as engineering, finance, meteorology, and wildlife management.
In public health, infectious disease modeling can help decision makers by
predicting the social and economic effects of an intervention and
informing spending for preparedness and response, among other things.
It can answer public health questions that other methods cannot, whether
for practical, ethical, or financial reasons. 2 However, because models
simplify reality, they may give misleading answers if the underlying data
or assumptions are flawed or not fully understood by decision makers.
Further, some real-world systems can be difficult to model because of
their inherent complexity, scale, or randomness. For these and other
reasons, researchers must carefully design, interpret, and communicate
the results of models that may be used to support public health decisions.
Understanding where and when infectious disease outbreaks may occur
can provide information—in near real time—to decision makers who help
set disease control policies and allocate resources.
You asked us to examine how federal agencies have used models to
inform decision-making in recent infectious disease outbreaks, and the
limitations and challenges in developing and using models. This report
examines (1) the extent to which the Department of Health and Human
Services (HHS) has developed or used models to inform public health
planning, policy, and resource allocation for Ebola, Zika, and pandemic
influenza; (2) the extent to which HHS coordinated its modeling efforts for
selected infectious diseases; and (3) steps HHS took to develop and
assess the performance of its models for the selected infectious diseases
and steps it applied to a selection of infectious disease models. It also (4)
describes the extent to which HHS has addressed challenges related to
modeling for selected infectious diseases.
In our review, we focused on HHS because of its leadership in scientific
and technical issues related to infectious disease modeling, role in
2Modeling can help to address ethical concerns. For example, modeling can help to
design a vaccine trial that can help minimize the number of cases of disease while also
fulfilling ethical requirements. Additionally, it can help with financial issues, such as
reducing the costs of conducting a large-scale vaccine trial.
Page 2 GAO-20-372 Infectious Disease Modeling
infectious disease outbreak preparedness and response activities, and
use of infectious disease modeling for policy and regulatory activities.
Within HHS, we identified four agencies—the Centers for Disease Control
and Prevention (CDC), Office of the Assistant Secretary for Preparedness
and Response (ASPR), 3 National Institutes of Health (NIH), and Food and
Drug Administration (FDA)—that may develop or use infectious disease
models. It is important that these agencies coordinate with one another
and with other relevant external entities to avoid the overlap and
duplication of modeling efforts across agencies and to share new ideas
and advances in modeling that might lead to new insights. We focused on
three infectious diseases in our review: Ebola, Zika, and pandemic
influenza. We selected these diseases based on their inclusion on the
National Institute of Allergy and Infectious Diseases’ Emerging Infectious
Diseases/Pathogens list and consulted with agency officials and five
infectious disease modeling experts for input on the selection of diseases
in our review. 4 We selected the experts based on our background
research and input from agency officials (additional details on expert
selection methodology can be found in appendix I).
To examine the extent to which HHS has conducted modeling to inform
public health planning, policy, and resource allocations for selected
infectious diseases:
• We interviewed agency personnel, including agency officials and staff
who develop and use models, referred to here as “modelers,” and
reviewed agency documents and reports to determine how or why the
agencies develop or fund models; determine the types of models used
and the questions they are addressing; or obtain a general description
3Subject to the authority of the Secretary, the Assistant Secretary for Preparedness and
Response has duties related to public health emergency preparedness and response,
biodefense, medical countermeasures, and other relevant topics. 42 U.S.C. § 300hh–10.
4The National Institute of Allergy and Infectious Diseases is one of NIH’s 27 Institutes and
Centers that conducts and supports basic and applied research to better understand,
treat, and ultimately prevent infectious, immunologic, and allergic diseases. The National
Institute of Allergy and Infectious Diseases’ pathogen priority list is periodically reviewed
and is subject to revision in conjunction with federal partners, including the U.S.
Department of Homeland Security, which determines threat assessments, and CDC,
which is responsible for responding to emerging pathogen threats in the United States.
Page 3 GAO-20-372 Infectious Disease Modeling
and specific examples of how these agencies use models to inform
planning, policy, and resource allocation. 5
• We interviewed NIH officials about funding for research related to
modeling for the selected diseases.
• We interviewed officials from five state health departments—selected
based on a review of a CDC draft report on model usage, on the level
of influenza activity that states experienced, and geographic variation
by U.S. region—about their experiences using CDC-developed
modeling tools for influenza response.
• For context on and examples of the types of modeling that CDC and
ASPR conducted, we reviewed documents CDC and ASPR officials
provided to us or cited in our interviews. (For a bibliography of models
reviewed, see appendix II.) We did not include FDA and NIH in this
portion of the review, because FDA has a limited role in modeling, and
NIH funds, rather than conducts, modeling. 6
To examine the extent to which HHS agencies coordinated their modeling
efforts for the selected infectious diseases:
• We interviewed agency officials and reviewed documents related to
coordination and collaboration, including memoranda of
understanding between agencies, to identify the nature and extent of
coordination and collaboration across HHS agencies that conduct or
fund modeling. We compared these actions to six of the eight leading
collaboration practices we identified in our prior work based on their
relevance to the coordination efforts we reviewed (see appendix I). 7 In
this report, and in our past work, we define coordination broadly as
any joint activity that is intended to produce more public value than
could be produced when organizations act alone.
5We asked agency officials questions about how models inform budget decisions. In
response to these questions, agency officials said that models are not used to inform
budgets, but that they may inform resource allocation decisions, which may or may not
inform budgets.
6FDA had limited modeling activity related specifically to the blood donor bank for Zika
virus, while NIH funds modeling outside the agency.
7We excluded two leading practices from our review related to reinforcing agency
accountability and reinforcing individual accountability for collaborative efforts. GAO,
Results-Oriented Government: Practices That Can Help Enhance and Sustain
Collaboration among Federal Agencies, GAO-06-15 (Washington, D.C.: Oct. 21, 2005).
Page 4 GAO-20-372 Infectious Disease Modeling
To examine steps HHS took to develop and assess the performance of
models for selected diseases and the steps it applied to a selection of
infectious disease models:
• We identified steps that infectious disease modelers generally
consider when developing and assessing the performance of models
from a synthesis of information gathered from interviews with agency
officials, interviews with additional relevant experts, and reviews of
documents. From these sources, we also gathered information on
how these assessments may impact the use of models for public
health decision-making.
• We reviewed information regarding steps taken to develop and
assess the performance of models, for a non-probability sample of
models in published papers or memos, including seven models
prepared by CDC (two each for Ebola and Zika, and three for
pandemic influenza); and three prepared by ASPR (one for each
disease). 8 We compared the steps taken in the development and
assessment of the performance of these models to the commonly-
considered steps we identified as described above and followed up
with agencies to confirm our determinations and gather information on
why some steps were not taken.
To describe the extent to which HHS has addressed challenges related to
modeling for selected infectious diseases, we took the following steps:
• We interviewed selected experts regarding modeling-related
challenges. We also interviewed agency officials, including modelers,
and selected experts, regarding challenges and limitations related to
modeling; steps they’ve taken to address the challenges; and whether
these challenges can be addressed or are ongoing.
• We reviewed documents and reports from agencies and other
sources such as the National Science and Technology Council report,
“Towards Epidemic Prediction: Federal Efforts and Opportunities in
Outbreak Modeling” to identify challenges related to modeling and
steps taken or recommended, if any, to alleviate these challenges. 9
8For Ebola and Zika, we focused on review of selected papers or memos produced since
2014 in order to capture the period following the 2014-2016 Ebola and 2015-2016 Zika
outbreaks. For pandemic influenza, we focused on papers and memos produced since
2009, when the H1N1 pandemic occurred in the United States.
9Pandemic Prediction and Forecasting Science and Technology Working Group of the
National Science and Technology Council, Towards Epidemic Prediction: Federal Efforts
and Opportunities in Outbreak Modeling (Washington, D.C.: December 2016).
Page 5 GAO-20-372 Infectious Disease Modeling
We conducted this performance audit from May 2018 to May 2020 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Background
Public Health Agency In the United States, HHS is the lead federal agency responsible for
Roles in Infectious public health. Its responsibilities include preparing for, mitigating,
responding to, and recovering from public health emergencies. 10 Within
Disease Outbreaks and
HHS, ASPR and CDC prepare for and respond to infectious disease
Response outbreaks.
ASPR leads and coordinates national preparedness and response to
outbreaks in the United States. It also coordinates and supports
advanced research and development, manufacturing, and procurement
and deployment of medical countermeasures, such as vaccines, drugs,
therapies, and diagnostic tools that can be used in the event of a potential
public health emergency to protect the public from harm.
CDC monitors and responds to outbreaks by, among other things,
studying the link between infection and health; monitoring and reporting
cases of infection; and providing guidance to the public, travelers, and
health care providers. During public health emergencies, CDC may
operate an Emergency Operations Center (EOC) for monitoring and
10The U.S. Department of Agriculture has a lead role for incident management during an
animal disease incident affecting domestic livestock or poultry. It conducts both infectious
disease and economic modeling. Other agencies are also involved in planning or
responding to infectious disease threats. For example, the Department of Transportation
has a responsibility to guide preparedness for the U.S. aviation system to respond to such
diseases. See GAO, Air Travel and Communicable Diseases: Comprehensive Federal
Plan Needed for U.S. Aviation System’s Preparedness, GAO-16-127 (Washington, D.C.:
Dec. 16, 2015). The Department of Homeland Security is responsible for domestic
incident management and serves as the lead for interagency coordination and planning for
emergency response. It assists in providing information to emergency management
officials and provides enforcement of international quarantines through the Department of
Homeland Security/U.S. Coast Guard, U.S. Customs and Border Protection, and U.S.
Immigration and Customs Enforcement. See GAO, Defense Civil Support: DOD, HHS,
and DHS Should Use Existing Coordination Mechanisms to Improve Their Pandemic
Preparedness, GAO-17-150 (Washington, D.C.: Feb. 10, 2017).
Page 6 GAO-20-372 Infectious Disease Modeling
coordinating its response to emergencies—including infectious disease
outbreaks of Ebola, Zika, and pandemic influenza—in the United States
and abroad. The EOC staff helps with directing specific incident
operations; acquiring, coordinating, and delivering resources to incident
sites; and sharing incident information with the public.
Other agencies perform additional work related to infectious diseases. For
example, FDA monitors and protects the blood supply, and NIH makes
grant awards that support research related to diseases and modeling. 11
ASPR, CDC, and FDA have different approaches to modeling. 12 In the
cases of Zika, Ebola, and pandemic influenza, CDC and ASPR are two
key agencies that conduct federal infectious disease modeling efforts. As
of February 2020, ASPR had a centralized modeling unit staffed by about
nine people, who are a mix of federal and contract employees, according
to ASPR officials. At CDC, however, modeling is decentralized and
integrated into the individual centers that make up the agency. 13 Some
staff work full time on modeling, while others spend part of their time on
other tasks. In addition, some of CDC’s modeling efforts are conducted
externally. According to CDC, approximately 70 staff members
participated in modeling studies, as of October 2018. 14 Of those staff,
CDC’s Health Economics and Modeling Unit employed about 10
modelers who have worked on Ebola and other diseases. For Zika, CDC
officials responding to Zika said most modeling work was done by one
modeler in CDC’s Division of Vector-Borne Diseases, a part of the
11FDA establishes standards and is responsible for identifying and responding to potential
threats to blood safety or supply. FDA promulgates and enforces standards for blood
collection and the manufacturing of blood products. Of our selected diseases, FDA
officials said the agency has only modeled for Zika. NIH funds and carries out basic and
applied research to better understand diseases and related technology.
12According to NIH officials, NIH supports extramural modeling through its partners,
universities, and other organizations. One institute within NIH, the Fogarty International
Center, models internally on a case-by-case basis, and modelers generate the questions
and topics themselves. NIH officials explained that their modeling is for basic research
and should not be used for making policy or other decisions.
13“Centers” refers collectively to CDC’s centers, institute, and offices.
14Modeling is conducted within four CDC Centers and is organized differently in each
Center. According to CDC, the number of modelers should be regarded as an estimate,
because many of the people who perform CDC modeling activities self-identify as
epidemiologists, statisticians, or economists. Further, some may model part-time in order
to address questions in their areas of expertise.
Page 7 GAO-20-372 Infectious Disease Modeling
National Center for Emerging and Zoonotic Infectious Diseases. 15 CDC
influenza officials said influenza modeling is conducted by six or seven
members of CDC’s Influenza Division. 16 Agency infectious disease
modeling activities are not limited to Ebola, Zika, or pandemic influenza.
Agency efforts to protect the nation from disasters and emergencies can
be organized into two elements: preparedness and response. Infectious
disease modeling is one tool used to inform a wide range of decisions
related to outbreak preparedness and in response to an outbreak. In the
context of infectious disease outbreaks, ASPR and CDC perform work on
preparedness and response. For example, ASPR leads the Public Health
Emergency Medical Countermeasures Enterprise (PHEMCE), an
interagency group that helps develop medical countermeasures—FDA-
regulated products including drugs, or devices that may be used in the
event of a potential public health emergency to protect the public from
harm. 17 CDC may activate its EOC to assist with the response during an
outbreak. 18 For example, during the 2014-2016 West Africa Ebola
outbreak, CDC activated its EOC in July 2014 to help coordinate
activities. CDC personnel were deployed to West Africa to assist with
response efforts, including surveillance, data management, and
laboratory testing.
Infectious Disease Since the 1980’s, emerging infectious diseases have resulted in more
Outbreaks recurrent disease outbreaks, causing an increasing number of human
infections. Emerging infectious diseases have at least one of the following
15CDC officials responding to Zika said this Zika modeler was supported by one to three
short-term interns or graduate fellows. During the response, according to these officials,
additional modelers from around CDC were brought in to help model. CDC is made up of
centers, institutes, and offices, which we collectively refer to as centers. Such centers can
be made up of units known as divisions, and divisions may contain branches.
16While this audit focuses on pandemic influenza, CDC modelers and officials said that
they use seasonal influenza modeling to prepare for a pandemic and that seasonal
models can be used in a pandemic. Therefore, we are discussing seasonal modeling
efforts in our review of pandemic influenza modeling.
17The PHEMCE was established by HHS in 2006. It is responsible for providing
recommendations to the Secretary of HHS on medical countermeasure priorities, and
development and procurement activities. HHS, Office of Public Health Emergency
Preparedness; Statement of Organization, Functions, and Delegations of Authority, 71
Fed. Reg. 38403 (July 6, 2006).
18An EOC is a physical location where responders, including federal and state/territorial
responders, and nongovernmental responders, can meet to coordinate information and
resources to support incident management during a response.
Page 8 GAO-20-372 Infectious Disease Modeling
characteristics: they are newly recognized, have emerged in new areas,
are newly affecting many more individuals, or have developed new
attributes. Some of these diseases—including Ebola and Zika—are
zoonotic pathogens, meaning they spread from animals to humans. 19
Zoonotic pathogens can be carried from an animal to a human by another
animal, such as a mosquito, chicken, or bat, which is known as a vector.
Such pathogens sicken approximately 1 billion people annually.
Ebola According to the World Health Organization, Ebola causes an acute,
serious illness, which is often fatal if untreated. Ebola is introduced into
human populations through close contact with the blood and other bodily
fluids of infected animals. Humans spread Ebola through direct contact
with the bodily fluids of infected individuals or objects contaminated with
these fluids. Ebola symptoms include fever, muscle pain, vomiting,
diarrhea, impaired kidney and liver functioning, and, in some cases,
internal and external bleeding. There have been five Ebola outbreaks
since 2014, including the 2014-2016 West Africa outbreak which caused
more than 28,600 cases and 11,325 deaths. 20 Since 2018, there has
been an ongoing outbreak in the Democratic Republic of the Congo.
Figure 1 provides a timeline of Ebola outbreaks since 2014.
19Strains of zoonotic influenza that have caused human infections or that have the
potential to cause such infections are described as zoonotic influenza viruses of public
health concern. They are considered to be “of concern” because if the virus gains the
ability to spread efficiently among humans, it could cause a global pandemic. The primary
sources for influenza A viruses are birds and swine, but they can infect other animal
species, including cats, dogs, and ferrets. Influenza B viruses only circulate widely among
humans. The 2014-2016 Ebola, 2015-2016 Zika, and 2009 H1N1 pandemic influenza
outbreaks are examples of emerging infectious disease outbreaks. For the purposes of
this report, we will refer to these outbreaks as infectious disease outbreaks, and these
diseases as infectious diseases.
20During the 2014-2016 Ebola epidemic, 11 people were treated for the virus in the United
States. The majority of the cases were people who were exposed to the virus and became
ill while in West Africa. During this outbreak, CDC helped to coordinate technical
assistance and disease control activities with partners and deployed personnel to help
with response efforts and provided support with logistics, staffing, and communication,
among other things.
Page 9 GAO-20-372 Infectious Disease Modeling
Figure 1: Timeline of Ebola outbreaks since 2014
a
According to WHO, a Public Health Emergency of International Concern is an event which
constitutes a public health risk to other countries through the international spread of disease and
potentially requires a coordinated international response.
Page 10 GAO-20-372 Infectious Disease Modeling
b
This patient later died. Two health care workers who cared for the patient tested positive for Ebola,
both recovered.
c
Seven other people were cared for in the U.S. after being exposed to the virus and becoming ill in
West Africa. Six recovered, one died.
d
There are 66 cases and 49 deaths. Ebola was first identified in the DRC in 1976.
CDC estimates this outbreak resulted in over 28,600 total cases of Ebola and 11,325 deaths. This
e
was the largest Ebola outbreak recorded.
f
There are eight suspected cases, including four deaths.
g
There are 54 total cases and 33 deaths.
h
Since June 12, 2019, no additional cases have been reported in Uganda.
This is the DRC’s largest Ebola outbreak, and the second largest Ebola outbreak recorded.
i
Zika Zika is a virus that is primarily transmitted through mosquito bites. It can
cause symptoms such as fever, rash, conjunctivitis (red eyes), and joint
and muscle pain. It can also be transmitted from mother to child during
pregnancy, or around the time of birth, or from person to person through
sexual contact or blood transfusion. 21 Many infected people do not have
symptoms or will only experience mild symptoms. The Zika outbreak that
began in 2015 affected individuals infected with the virus in ways that had
not been seen with previous outbreaks of the disease. Specifically, during
the 2015-2016 outbreak, Zika infection in pregnant women was linked to
microcephaly and other severe brain defects, according to CDC. 22 CDC
officials said this was the first time in more than 50 years that an
infectious pathogen has been identified as the cause of birth defects. Zika
was also linked to other problems, such as miscarriage, stillbirth, and
Guillain-Barré syndrome, an uncommon disorder affecting the nervous
system. 23 In the Western Hemisphere, the first cases of locally-
21GAO, Emerging Infectious Diseases: Actions Needed to Address the Challenges of
Responding to Zika Virus Disease Outbreaks, GAO-17-445 (Washington, D.C.: May 23,
2017).
22Microcephaly is a rare nervous system disorder that causes a baby’s head to be smaller
than expected and not fully developed, which can lead to impaired thought processes,
delayed motor function, and other adverse outcomes. A study published in December
2017 found that 19 children ages 19 to 24 months born with microcephaly and with
laboratory evidence of Zika infection experienced problems, including an inability to sit
independently, difficulties with sleeping and feeding, seizures, and hearing and vision
problems. Ashley Satterfield-Nash et al., “Health and Development at Age 19-24 Months
of 19 Children Who Were Born with Microcephaly and Laboratory Evidence of Congenital
Zika Virus Infection During the 2015 Zika Virus Outbreak—Brazil, 2017,” Morbidity and
Mortality Weekly Report, vol. 66, no. 49 (Atlanta, Ga.: Centers for Disease Control and
Prevention, Dec. 15, 2017): 1347-1351.
23Guillain-Barré syndrome is a rare disorder in which the body’s immune system attacks
the nervous system outside the brain and spinal cord, causing muscle weakness and, in
some cases paralysis, although most people recover.
Page 11 GAO-20-372 Infectious Disease Modeling
transmitted Zika were confirmed in Brazil in May 2015. In December
2015, locally-transmitted Zika was reported in Puerto Rico. On January
22, 2016, CDC activated its Emergency Operations Center to respond to
outbreaks of Zika occurring in the Americas and to increased reports of
birth defects and Guillain-Barré syndrome in areas affected by Zika.
Within the continental United States, the first locally-transmitted cases
were confirmed in Florida in June 2016. The World Health Organization
declared Zika a Public Health Emergency of International Concern from
February to November 2016. 24
Pandemic Influenza In the spring of 2009, a novel influenza virus emerged, known as
influenza A (H1N1)pdm09. 25 According to CDC, it was detected first in the
United States and quickly spread across the world, causing a pandemic
or global outbreak of a new influenza A virus. This new virus contained a
combination of influenza genes not previously identified in animals or
people. The virus was very different from other H1N1 viruses circulating
at the time, so seasonal influenza vaccines offered little cross-protection
against infection with the new H1N1 virus, according to CDC. A vaccine
against the new virus was produced, but it was not available in large
quantities until late November—after the peak of illnesses during the
second wave in the United States. 26 CDC activated its EOC on April 22,
2009, to manage the H1N1 response. From April 12, 2009, to April 10,
2010, CDC estimated there were about 60.8 million cases, 274,304
hospitalizations, and 12,469 deaths in the United States due to the new
H1N1 virus. According to CDC, few young people had any existing
immunity—as detected by antibody response—to the virus, but nearly
24According to the World Health Organization, a Public Health Emergency of International
Concern is an event which constitutes a public health risk to other countries through the
international spread of disease and potentially requires a coordinated international
response.
25Periodically, new influenza A viruses emerge from animals such as birds and pigs that
have sustained transmission and spread efficiently among humans. This situation is called
an influenza pandemic. Pandemic influenza is different from seasonal influenza.
Pandemic influenza may cause moderate to high rates of medical visits, complications,
hospitalization, and death. It may cause major impacts on the general public including
travel restrictions and school or business closings and has the potential for having a
severe impact on domestic and world economies. In the 2009 pandemic, this virus was
designated as influenza A (H1N1) pdm09 virus.
26We previously reported on key issues raised by the federal government’s response to
the 2009 H1N1 influenza pandemic. See GAO, Influenza Pandemic: Lessons from the
H1N1 Pandemic Should Be Incorporated into Future Planning, GAO-11-632 (Washington,
D.C.: June 27, 2011).
Page 12 GAO-20-372 Infectious Disease Modeling
one-third of people over 60 years old had antibodies against it, likely from
exposure to an older H1N1 virus.
Multiple strains of influenza can infect humans, including strains that
originate in animals. According to CDC, human infections with an Asian
lineage avian influenza A (H7N9) virus were first reported in China in
March 2013. During an epidemic that lasted from October 1, 2016,
through September 30, 2017, the World Health Organization reported 766
human infections with H7N9 virus, making it the largest H7N9 epidemic.
From 2013 to December 7, 2017, there were 1,565 humans infected with
Asian lineage H7N9 reported by the World Health Organization.
According to CDC, while the risk posed by H7N9 virus to the public’s
health was low, the agency was concerned about its pandemic potential.
Infectious Disease Models Agencies use infectious disease models to answer a variety of public
health questions, including those related to outbreak preparedness and
response. 27 A model is a physical, mathematical, or logical representation
of a system, phenomenon, or process that allows a researcher to
investigate that system, phenomenon, or process in a controlled way. For
example, the classic Susceptible-Infected-Recovered or “SIR” model
divides a population into three categories: 1) susceptible to the disease,
S; 2) infected and infectious, I; and 3) recovered or removed from the
infected or susceptible population, R. This model uses equations to
determine how many people move between these three categories. The
equations contain parameters—numerical descriptors of the disease
based, for example, on experiment, expert opinion, or statistics of an
ongoing or past outbreak. The equations allow the researcher to estimate
how many people are or could be affected by the disease. For example,
for past Ebola outbreaks, models estimated that after 40 days, about 44
percent of the population in close contact with infected individuals was
susceptible to infection, 31 percent was infected, and 22 percent was
recovered. 28 Based on these parameters, equations for transfer between
27For the purposes of this report, the terms “models” and “modeling” refer specifically to
models focused on infectious disease, rather than other types of models or modeling of
human health issues. The types of modeling discussed herein may be helpful for
emerging infectious diseases such as the novel coronavirus disease; however, this
disease is not included in the scope of our review because of the timing of the novel
coronavirus event.
28Astacio, Jaime, D. Briere, M. Guillen, J. Martinez, F. Rodriguez, N. Valenzuela-Campos,
Mathematical Models to Study the Outbreaks of Ebola, Biometrics Unit Technical Reports;
Number BU-1365-M (1996).
Page 13 GAO-20-372 Infectious Disease Modeling
categories, and underlying demographics of the community, an
epidemiologist could use the model to estimate how many people within a
given town could be susceptible, infected, or removed from the categories
of susceptible or infected (due to death or recovery and immunity). 29
Based on model estimates and if a vaccine was available, CDC officials
said the decision maker could plan for a specific number of vaccine kits
and additional medical staff and supplies to treat infected patients.
Models can also help agency officials anticipate future outbreaks, forecast
the spread or severity of a disease, and predict the effects and costs of
different intervention options. After an outbreak, models can help sort out
what happened, what drove the outbreak, and how it compared to past
outbreaks. Other tools are available to accomplish some of these tasks,
but models are particularly useful when existing data are not sufficient to
answer a given question, or when agencies need to integrate data from
disparate sources.
Infectious disease models can be put into two broad categories:
• Statistical models. This type of model identifies relationships or
patterns that can be used to describe what is occurring or predicts
what may occur in the future based on what has occurred in the past.
Statistical models tend to use a large amount of data, such as past
observed events, to forecast future events, such as disease
occurrence, but do not require a fundamental understanding of
biological processes or human behavior. 30 They can predict outcomes
when causes are not known or understood and when scientific
understanding of a disease is limited. They tend to use large amounts
of data on past events to forecast future events. Statistical models do
not provide full explanations about an infectious disease but may be
used when epidemiologists have all or most of the data needed to test
a hypothesis. Several benefits can be derived from statistical
modeling, including the ability to control for multiple factors that might
impact the outcome reviewed, and the ability to isolate the potential
effect of infectious disease factors on a particular outcome.
29To accurately estimate the numbers of people in any of the Susceptible-Infected-
Recovered categories of the model, the latest demographic information, such as
population size and density, would be needed.
30Statistical models can be further broken down by individual techniques, such as
regression, which utilizes relationships between predictor variables and response
variables, or Bayesian techniques, which use new data to continually update probabilities
regarding prior assumptions.
Page 14 GAO-20-372 Infectious Disease Modeling
• Mechanistic models. Mechanistic models rely heavily on scientific
evidence and theory related to infectious diseases, and the
understanding of disease dynamics or human behavior from prior
knowledge—such as biological processes or interactions between
people—to represent known processes. They use basic infectious
disease science to inform public health guidance and provide insights
into outbreak emergence, spread, and control. For example,
population-based models can simulate the course of an epidemic by
dividing the population into different categories, such as susceptible,
infected, and recovered. Mechanistic models can project the likely
course of disease transmission, calculate and predict the effect of
proposed interventions, and take into account variable conditions,
such as human behavior. 31
Both statistical and mechanistic models can range from simpler to more
complex. A simpler model may, for example, have fewer parameters
(inputs) or equations than a more complex model. According to CDC
modelers and an expert, a simpler model may be run with a variety of
software, ranging from spreadsheet software to more sophisticated
software, whereas more complex models are usually run using
sophisticated statistical or mathematical programming languages. As a
model becomes more complex, it can become harder to describe,
recreate, and understand its internal functioning.
Modeling is identified as a beneficial tool in various national plans for
disease response and biodefense. These plans do not define the extent
to which modeling should occur or how models should be developed for
policy, resource allocation, or planning purposes. See table 1 for
examples of relevant national plans.
31Mechanistic models can be further broken down by individual techniques, such as
Susceptible-Infected-Recovered models, which move populations through the stages of a
disease epidemic from susceptible to infected to recovered, or agent-based models, which
incorporate data on an individual’s behaviors into simulations that assess the
effectiveness of public health actions, programs, or policies.
Page 15 GAO-20-372 Infectious Disease Modeling
Table 1: Examples of National Plans Identifying Modeling
Title Year issued Description of modeling
National Biodefense Strategy 2018 Maintains the goal of incorporating forecasting and modeling into
intelligence products and processes, as appropriate, and improving
the ability to model and forecast the likelihood and impact of
bioincidents. It calls for using modeling capabilities to maintain
situational awareness and support decision making during a
response effort.
2017-2018 Public Health Emergency 2017 Discusses using models for purposes such as setting requirements
Medical Countermeasures Enterprise for medical countermeasures and highlights modeling’s use in the
Strategy and Implementation Plan Zika response. As a part of this plan, ASPR established an
Innovation Modeling Hub designed to provide analytic decision
support and access to real-time modeling capabilities to senior
decision makers.
HHS Pandemic Influenza Plan 2005 with 2017 States that forecasting, modeling, and planning tools facilitate
update dynamic estimates of pandemic influenza spread, burden, and
impact. Key actions, as described in the 2017 plan, are using
innovative data sources and models to better forecast disease
emergence and patterns, and developing and using modeling tools
in emergencies to inform policy, clinical guidance, and response
strategies involving medical countermeasures.
HHS Ebola Response 2016 Discusses using modeling to inform the design of clinical vaccine
Improvement Plan trials.
National Strategy for Pandemic 2006 Expands infectious disease modeling capabilities and works to
Influenza Implementation Plan ensure that mechanisms are in place to share model results with
state and local authorities, and the private sector.
Legend:
ASPR: Office of the Assistant Secretary for Preparedness and Response
HHS: Department of Health and Human Services
Source: GAO analysis of White House and HHS documents. | GAO-20-372
HHS Has Used
Infectious Disease
Models to Help Inform
Policy and Planning
Use of Models to Inform CDC and ASPR use models primarily to answer questions from decision
Planning and Policy makers. CDC and ASPR officials told us, and documents show, that
modeling is one source of information that may inform such decisions,
Decisions
along with sources such as expert opinion, surveillance, other prior work
on the disease, and an official’s own knowledge.
Page 16 GAO-20-372 Infectious Disease Modeling
CDC modelers and officials said there is no “rule” as to when to use
models, and in some situations, it may not be considered useful. For
example, CDC did not use modeling when issuing a travel notice for an
Ebola outbreak in specific provinces in the Democratic Republic of the
Congo, officials said. Instead, CDC based the travel order on an analysis
that considered disease incidence and prevalence, public health
infrastructure, and the availability of therapeutics, among other things.
Similarly, CDC officials responding to Ebola said modeling may be
undesirable when it would take too long to engage the necessary external
subject matter experts or when modeling would detract from responding
to a disease. 32
CDC and ASPR modelers use models for a variety of purposes. CDC
officials said modeling is done differently for each disease, and the
amount and type of modeling varies across CDC centers, in part because
some centers have less capacity to conduct modeling than others.
According to a CDC internal report, the most frequent uses of infectious
disease modeling at CDC are:
• guiding preparedness and response efforts;
• conducting economic analyses to evaluate the benefits of public
health actions, thereby reducing illness and deaths from infectious
diseases;
• understanding pathogen biology, disease transmission, and
estimating disease burden; and
• assessing the effect of interventions and prevention strategies. 33
ASPR modelers and officials said models have provided information
about topics such as:
• resources, including protective equipment, needed to help respond to
an Ebola outbreak;
32During an infectious disease outbreak, modeling results may need to be provided to
decision makers in time frames potentially as short as one day.
33Results are based on CDC interviews with 38 CDC infectious disease modelers and 15
division directors from multiple divisions. Infectious disease modelers were interviewed in
groups, as were division directors from each center, along with a representative of the
Center’s division. According to these interviews, other uses of modeling include guiding
the development of public health policies and guidelines.
Page 17 GAO-20-372 Infectious Disease Modeling
• the number of therapeutics and vaccine doses needed to respond to
Ebola, both in Africa and domestically;
• expected U.S. demand for Zika diagnostics; and
• the number of vaccine doses needed to mitigate the spread of
pandemic influenza.
ASPR modelers and officials said modelers tend to serve in a broad role
that can include modeling, data analysis, or other tasks. For example,
officials said a modeler could provide a team with day-to-day analytic
support and not necessarily spend time developing models or use them.
Additionally, ASPR maintains a Visualization Hub that can be used for
outbreak planning and response, including outbreaks of pandemic
influenza and other emerging infectious diseases (see fig. 2).
Figure 2: Office of the Assistant Secretary for Preparedness and Response’s (ASPR) Visualization Hub
CDC and ASPR modelers and officials said they generally initiate
modeling in response to questions from decision makers. The modelers
then work closely with epidemiologists and other subject matter experts to
answer the questions. Modeling, according to CDC officials, may be used
by individuals or groups within centers, such as division directors,
Page 18 GAO-20-372 Infectious Disease Modeling
branches, or teams to influence decisions. 34 Who answers a particular
question depends, according to ASPR modelers and officials, on the
decision maker. Sometimes questions asked will not be within their
mission—modelers may suggest such questions be sent to a more
relevant agency or part of HHS. CDC and ASPR have modeled to answer
a variety of public health questions relevant to Ebola, Zika, and pandemic
influenza, and, at times, the results helped inform policy and planning
decisions. Modelers and officials provided the following examples:
• Planning: ASPR modelers and officials said the bulk of the agency’s
modeling is related to the planning, development, and deployment of
medical countermeasures. For example, these modelers and officials
said many clinical trials for vaccines and therapeutics were planned
during the 2014-2016 Ebola outbreak response. As a part of these
planning activities, ASPR modelers said modelers developed
forecasts of future trajectories of disease incidence under a variety of
conditions. These forecasts indicated a significant likelihood the
disease incidence in Sierra Leone could decrease to a level that
would significantly reduce the success of the trials, according to
modelers. Additionally, at the beginning of the 2014-2016 Ebola
outbreak response, CDC modelers received modeling questions
related to the resources needed to effectively limit the spread of the
disease, according to CDC documentation. CDC used models to
predict the number of Ebola cases that could be expected over time
with and without disease interventions such as Ebola treatment units,
community care centers, and safe burials. On the basis of this
information and other factors, including a United Nations document on
Ebola needs, CDC leadership and other U.S. government officials
recommended a rapid increase in Ebola response aid, according to
CDC documentation. According to CDC documentation, later
analyses demonstrated that this increase helped to greatly reduce the
actual number of cases, compared to the likely number if prompt
action had not been taken. 35 Additionally, in response to the H7N9
influenza outbreak in 2017, ASPR modeled to determine when doses
34CDC is made up of centers, offices, and an institute, which we collectively refer to as
centers. Such centers can be made up of units known as divisions, and divisions may
contain branches.
35Meltzer, Martin I, S. Santibanez, L. S. Fischer, T. L. Merlin, B. B. Adhikari, C. Y. Atkins,
C. Campbell, I. C. Fung, M. Gambhir, T. Gift, B. Greening, W. Gu, E. U. Jacobson, E. B.
Kahn, C. Carias, L. Nerlander, G. Rainisch, M. Shankar, K. Wong, M. L. Washington,,
Modeling in Real Time During the Ebola Response, Morbidity and Mortality Weekly
Report, vol. 65, no. 3 Supplement (Atlanta, Ga.: Centers for Disease Control and
Prevention, July 8, 2016).
Page 19 GAO-20-372 Infectious Disease Modeling
of influenza vaccine should be delivered and how many doses should
be administered in order to mitigate a domestic outbreak. This model
found that having a vaccine stockpile could be helpful in preventing
disease and that a slow effort to administer an H7N9 vaccine could
reduce the vaccine’s usefulness.
• Policy: During the Zika outbreak, CDC modelers and officials said
they modeled to determine the potential effectiveness of using
pesticides to remove insects from aircraft, trains, or ships. According
to modelers and agency officials, the issue arose as concern about
Zika virus grew, including from other countries and U.S. agencies, like
the Department of Transportation and Department of Defense. 36 The
model indicated that humans are more likely than insects to transport
Zika on airplanes, and officials therefore concluded that the use of
pesticides on airplanes would not be an effective intervention.
According to CDC modelers and officials, this modeling resulted in an
additional sentence being added to World Health Organization policy,
which stated that pesticide use was not expected to be effective. 37
The extent of modeling conducted for Ebola, Zika, and pandemic
influenza varied according to the question being asked, along with other
factors as follows: 38
• Type of question: CDC and ASPR have used models to answer
such questions as who should be prioritized for vaccination or
treatment, how transmissible a disease is, and how effective certain
interventions are likely to be, according to modelers and agency
officials. For example, ASPR modelers and officials said they modeled
36Within the United States, the Department of Transportation has a responsibility to guide
preparedness for the U.S. aviation system to respond to communicable diseases.
GAO-16-127.
37In the World Health Organization’s report on aircraft disinsection for controlling the
international spread of vector borne diseases, disinsection was described as considered
to be of low effectiveness for preventing the importation of pathogens as the risk of
pathogen importation by mosquitos when compared to the risk posed by infected travelers
is low. World Health Organization, Report of the WHO Ad-hoc Advisory Group on aircraft
disinsection for controlling the international spread of vector-borne diseases, (Geneva,
Switzerland: World Health Organization, 2016), 6.
38Experts and CDC officials said modelers should have the flexibility to select models
based on the question that needs to be answered. Neither the HHS agencies we
interviewed nor a variety of national strategies and plans we examined have guidance or
policies on how agencies should conduct modeling. While this audit focuses on pandemic
influenza, CDC modelers and officials said that they use seasonal influenza modeling to
prepare for a pandemic and that seasonal models can be used in a pandemic. Therefore,
we are including seasonal models in our review of pandemic influenza modeling.
Page 20 GAO-20-372 Infectious Disease Modeling
to help estimate the resources needed to respond to an Ebola
outbreak; the number of therapeutics and vaccine doses needed to
respond to Ebola, both in Africa and the U.S; and the expected U.S.
demand for Zika diagnostics. One ASPR official said that, during the
2009 pandemic influenza outbreak, modeling questions were used to
provide decision makers with information on what might happen in a
given situation. For example, models were used to provide information
related to decisions on early vaccine distribution and how this
intervention could affect the potential mortality rate.
• Time to model: How soon decision makers needed information also
influenced the extent to which CDC and ASPR modeled. For example,
if decision makers needed an answer in a week, modelers would
inform the decision makers about how much of the answer they could
provide within that time frame, ASPR modelers said. Similarly, CDC
modelers and officials said that, in one instance, modelers had only
12 hours to provide decision makers with information. Even estimating
the time needed to develop and conduct modeling could represent an
additional challenge, according to CDC modelers responding to Zika.
According to a CDC article on modeling to inform responses to novel
influenza viruses, the amount of time required to develop and execute
a model can vary from less than a week to more than a month. 39
Agency officials concurred with these time frames.
• Personnel and data availability: The availability of qualified
personnel was also a factor that affected how much modeling
agencies conducted for the selected diseases. For example, CDC
modelers and officials said the agency’s Division of Vector-Borne
Diseases has focused its resources in other areas, such as building
the capacity of states to address vector-borne diseases, and therefore
had not invested in individuals with the right skill sets to conduct
modeling for the Zika outbreak response. 40 As a result, the division
had to call on the three or four CDC modelers from outside of the
division who were available to assist with the Zika outbreak response,
which limited the amount of modeling that could be performed. Data
39M. Gambhir, C. Bozio, J. J. O’Hagan, A. Uzicanin, L. E. Johnson, M. Biggerstaff and D.
L. Swerdlow, “Infectious Disease Modeling Methods as Tools for Informing Response to
Novel Influenza Viruses of Unknown Pandemic Potential,” Clinical Infectious Diseases,
vol. 60, Supplement 1 (2015).
40CDC’s Division of Vector-Borne Diseases has a mission of reducing illness and death
from vector-borne diseases such as Zika. Its goals are to (1) identify and detect disease-
causing vector-borne pathogens; (2) understand when, where, how often, and how people
are exposed to them; (3) prevent exposure and mitigate the consequences of infection;
and (4) implement disease diagnostic, surveillance, control, and prevention programs.
Page 21 GAO-20-372 Infectious Disease Modeling
challenges can also limit the types of modeling conducted. For
example, when modeling for Zika, ASPR modelers said they used
available information, but data quality and availability limited their
ability to model. More data typically become available as an outbreak
progresses, but models may be most helpful at the beginning of an
outbreak when critical decisions need to be made (see fig. 3).
Figure 3: Timeline of Data Availability for Models Compared to Usefulness of Modeling During an Outbreak.
CDC and ASPR do not keep a list of all modeling conducted, and we
therefore cannot quantify the extent of their efforts in terms of a number of
models. ASPR modelers and officials said modeling is typically one small
aspect of the way the agency carries out its mission. One ASPR official
said models are never the sole source of information for decision-making.
According to NIH officials, NIH does not conduct or fund internal modeling
for decision-making purposes. NIH’s Fogarty International Center has
conducted self-initiated, internal modeling to answer questions generated
from research, and from ideas from Center-held workshops. Two NIH
institutes—the National Institute of General Medical Sciences and the
National Institute of Allergy and Infectious Diseases—along with NIH’s
Fogarty International Center have awarded grants for external modeling
research for our selected diseases. However, NIH officials said these
Page 22 GAO-20-372 Infectious Disease Modeling
efforts were intended to advance science, not for policy or outbreak
response. 41
Use of Models to Inform CDC and ASPR modelers and officials said they considered modeling
Resource Allocation results to a limited extent when making decisions about resource
allocation. 42 While modeling can help determine the amount of particular
Decisions
resources needed during an infectious disease outbreak, CDC modelers
and officials said it is not central to their resource allocation planning. For
example, CDC modelers and officials noted that while a model could
inform a decision maker about how many diagnostic testing supplies
would be needed based on the range of predicted cases, this would be
one input among many into the decision. Decision makers would also
consider whether there are other diagnostic test supplies for similar
diseases that could be used, the extent of laboratory testing capacity, or
the longevity of those supplies.
Models can be used to help plan for the cost of interventions by
determining the numbers or types of interventions that can be used during
a response to an infectious disease outbreak, according to CDC modelers
and officials. It can also help decision makers recognize gaps in their
ability to implement resource allocation decisions, according to CDC
officials. For example, CDC leadership described how modeling input
requirements spurred analysis of the factors limiting hospitals’ use of
ventilators during a pandemic influenza outbreak. 43 This work, according
to CDC officials, helped determine the number of ventilators that should
be included in the national stockpile. While modeling results are important
to consider during a public health event, ASPR officials and modelers
41NIH is made up of 27 institutes and centers. Of these, 24 institutes and centers award
more than 80 percent of the NIH budget each year to support investigators at more than
2,500 universities, medical schools, and other research organizations around the world.
NIH institute or center directors decide which grants to fund, considering staff input,
results of scientific peer review of grant applications, public health need, scientific
opportunity, and the need to balance their scientific portfolios.
42In this review, we asked agency officials about how models inform budget decisions. In
response to our questions, modelers and officials said that models are not used to inform
budgets but discussed how models informed resource allocation decisions, which may or
may not inform budgets. In the context of infectious disease outbreaks, resource allocation
refers to planning or using resources—such as drugs or trained personnel—when a
disease occurs simultaneously in different but interconnected regions.
43According to the World Health Organization, analysis is a detailed examination of
anything complex in order to understand its nature or to determine essential features. A
model is a physical, mathematical, or logical representation of a system, phenomenon, or
process that allows a researcher to investigate that system, phenomenon, or process in a
controlled way.
Page 23 GAO-20-372 Infectious Disease Modeling
said it is also important to consider concrete financial estimates based on
prior experience and whether recommended medical interventions or
countermeasures are available or effective.
For example, ASPR modelers and officials have occasionally been asked
to analyze costs for medical countermeasures, but modelers and officials
said that few medical countermeasures typically meet the requirements of
decision makers, and existing medical countermeasures are typically
unavailable for use in a response. ASPR modelers and officials noted that
the usefulness of modeling to the decision maker in these instances is
limited. In the event that they were asked to model for such questions,
ASPR modelers and officials said time would also be a limiting factor in
their analysis.
CDC has also developed models to inform decision-making at the state
level, specifically to assist state and local public health agencies in
developing outbreak response plans. A professional organization of
epidemiologists we contacted expressed some concerns with limitations
of CDC models, specifically noting that state and local officials viewed
CDC models as lacking the level of refinement needed for their state- and
local-level planning needs. To follow up, we interviewed officials from a
non-generalizable selection of five states based on their reported use of
CDC models, the level of selected disease activity in the state, and
geographic variation. Two of the five state health departments we
contacted reported using one of CDC’s models for Ebola, Zika, or
pandemic influenza. These two states confirmed that the usefulness of
the CDC FluSurge pandemic influenza model was limited by unrealistic
assumptions or a lack of predictive capability, but added that the models
were useful to them when considering how to allocate resources or
otherwise prepare for a severe pandemic. 44 Officials from one state health
department told us they had similar concerns with the CDC Ebola model
regarding an unrealistic overestimate of the potential cases, but added
that it was useful for informing staff allocation planning as part of their
overall response.
Officials from another state health department told us they used CDC’s
Zika modeling results that indicated how many emergency room visits
they could expect and what symptoms it would take to confirm a Zika
infection. At the time, state officials said, commercial testing for Zika was
44According to CDC, FluSurge is a spreadsheet-based model which provides hospital
administrators and public health officials with estimates of the surge in demand for
hospital-based services during an influenza pandemic.
Page 24 GAO-20-372 Infectious Disease Modeling
not available, so this modeling was very helpful to health officials looking
to recommend who hospitals should test based on the presence of Zika
symptoms. State health department officials added that many other
factors are considered when deciding on resource allocation, such as
local leadership and willingness to embrace the public health response.
The four HHS agencies that work on infectious disease modeling reported
Agencies Coordinate using multiple mechanisms to coordinate their efforts. 45 However, they do
Infectious Disease not routinely monitor these efforts, evaluate their effectiveness, or report
on them to identify areas for improvement.
Modeling Efforts but
Do Not Fully Monitor,
Evaluate, and Report
on Coordination
HHS Agencies Coordinate The four HHS agencies that work on infectious disease modeling—ASPR,
Infectious Disease CDC, FDA, and NIH—reported using multiple mechanisms to varying
extents to coordinate such efforts. For example:
Modeling Efforts in
Multiple Ways • Emergency Operations Center (EOC). During the response to an
outbreak, CDC activates its EOC—a temporary, formal organizational
structure for coordinating expertise within CDC and among agencies.
The four HHS agencies—ASPR, CDC, FDA, and NIH—used EOCs to
coordinate modeling efforts during responses to Ebola, Zika, and
pandemic influenza outbreaks. 46 For example, during the 2015-2016
Zika outbreak, CDC’s EOC served as the command center for
monitoring and coordinating the response by bringing together CDC
scientists with expertise in areas such as arboviruses (the category
that includes Zika), reproductive health, birth defects, and
45In this report, we define coordination broadly as any joint activity that is intended to
produce more public value than could be produced when organizations act alone.
46According to ASPR officials, ASPR uses the HHS Secretary’s operations center, which
coordinates with CDC’s EOC. Although included in the EOC, NIH officials stated that
NIH’s primary focus was on funding research rather than the use of models during
outbreaks. ASPR officials said they have not tested and deployed NIH-sponsored models
directly for use during responses, but that their models are informed by the structural and
parametric assumptions used by many academic modelers, including those funded by
NIH.
Page 25 GAO-20-372 Infectious Disease Modeling
developmental disabilities. 47 CDC modelers and officials told us that
they had weekly strategy meetings and briefings with response
leadership within the EOC where they discussed which modeling
questions to prioritize. 48 In general, CDC modelers in the EOC were
expected to coordinate with modelers from other agencies within and
outside of HHS—such as ASPR, FDA, NIH, and the Department of
Homeland Security—to produce timely estimates of cases,
hospitalizations, and deaths. These estimates can inform response
leadership and enable them to assess the speed and impact of the
geographic spread of the pandemic. Modelers in the EOC also
provide support to decision makers as they examine the potential
effects of various response options. These options include when and
how to deploy Strategic National Stockpile assets, such as influenza
antiviral drugs and mechanical ventilators. 49 We found the use of
EOCs to be consistent with leading collaboration practices we have
previously identified, such as defining and articulating a common
outcome. 50
• Public Health Emergency Medical Countermeasures Enterprise
(PHEMCE). The four HHS agencies also participated in PHEMCE, a
federal interagency body formed by HHS in 2006 that coordinates the
development, acquisition, stockpiling, and recommendations for use
of medical products that are needed to effectively respond to a variety
47Arboviral disease is a general term used to describe infections caused by a group of
viruses spread to people by the bite of infected arthropods (insects) such as mosquitoes
and ticks. Infections usually occur during warm weather months, when mosquitoes and
ticks are active. Examples include California encephalitis, Chikungunya, dengue, Eastern
equine encephalitis, Powassan, St. Louis encephalitis, West Nile, Yellow Fever, and Zika.
48Within CDC’s EOC, the Incident Manager provides leadership and direction for the
response and is responsible for ensuring response activities are coordinated across the
agency and with external entities. Other leadership positions, such as the Deputy Incident
Manager, Chief Science Officer, and Chief of Staff, provide assistance to the Incident
Manager.
49The Strategic National Stockpile is the nation’s largest supply of potentially life-saving
pharmaceuticals and medical supplies for use in a public health emergency severe
enough to cause local supplies to run out. In October 2018, oversight of the Strategic
National Stockpile was transferred from CDC to ASPR. Following this transfer, CDC
modelers have continued to support decision-making regarding the acquisition of Strategic
National Stockpile assets as well as their release during a response, according to CDC
officials. For example, CDC officials said modelers and subject matter experts are working
with ASPR modelers to determine stockpiling needs for the recently-approved Ebola
vaccine.
50GAO-06-15.
Page 26 GAO-20-372 Infectious Disease Modeling
of high-consequence public health emergencies. 51 PHEMCE is led by
ASPR and also includes partners at the Departments of Defense,
Veterans Affairs, Homeland Security, and Agriculture. PHEMCE’s
2017-2018 strategy and implementation plan, its most recent,
identified Ebola, pandemic influenza, and emerging infectious
diseases more broadly as high-priority threats. PHEMCE leadership
could ask modelers to address questions related to these infectious
diseases, according to ASPR modelers and officials. According to
ASPR officials, such questions tend to support larger response-
related efforts, and modeling results are often incorporated into final
reports and products. According to ASPR officials, as of February
2020, the PHEMCE structure has been updated and it is unclear how
modeling fits into the new structure. We found that coordination
through PHEMCE is consistent with leading collaboration practices
such as establishing mutually-reinforcing or joint strategies.
• Working groups. Modelers with the four HHS agencies have
participated in working groups related to infectious disease modeling
(see table 2). The use of working groups and similar bodies is
consistent with leading collaboration practices that we have previously
reported as useful for enhancing and sustaining interagency
collaboration, such as identifying and addressing needs by leveraging
resources. 52 For example, CDC and ASPR modelers participated in
the National Science and Technology Council’s Pandemic Prediction
Forecasting Science and Technology Working Group, which facilitates
coordination among numerous federal agencies. In 2016, this group
produced a report that identified challenges in outbreak prediction and
modeling for federal agencies and offered recommendations for
federal actions to advance the development and effective application
of outbreak prediction capabilities. 53
51HHS, Office of Public Health Emergency Preparedness; Statement of Organization,
Functions, and Delegations of Authority, 71 Fed. Reg. 38403 (July 6, 2006).
52GAO-06-15.
53National Science and Technology Council, Towards Epidemic Prediction. From 2015
until July 2018, the National Security Council maintained a permanent Pandemic
Preparedness and Response Directorate. This directorate was to oversee ongoing work to
prepare for future infectious disease threats, and to coordinate a response when such
threats arise.
Page 27 GAO-20-372 Infectious Disease Modeling
Table 2. Selected Examples of Working Groups for Infectious Disease Modeling
Working group Description
Pandemic Prediction Forecasting Science and This interagency working group, directed by the National Science and Technology
Technology Working Group Council, is responsible for analyzing the state of infectious disease modeling and
prediction, and facilitating coordination among numerous federal agencies.
According to CDC modelers and officials, as of October 2018, the charter for this
group is no longer active, and it meets on a voluntary, ad hoc basis.
Centers for Disease Control and Prevention’s According to CDC officials, this group connects modelers by holding seminars,
(CDC) infectious disease modeling community managing an email list, and arranging for members to peer review one another’s
of practice models. This group had over 160 participants from various centers across CDC,
as of June 2019.
Modeling coordination groups During the 2014-2016 Ebola and 2015-2016 Zika outbreaks, the Department of
Health and Human Services’ (HHS) Office of the Assistant Secretary for
Preparedness and Response (ASPR) established temporary modeling
coordination groups that brought together government agencies and academics to
share early modeling results and discuss pressing questions that could be
answered through modeling, according to ASPR modelers and officials. A wide
range of entities participated in these groups, including the four HHS agencies,
other federal agencies such as the Departments of Defense and Homeland
Security, universities, and foreign entities, such as the World Health Organization
and the United Kingdom. According to ASPR modelers and officials, there are no
plans to convene modeling coordination groups unless there is an ongoing
infectious disease outbreak.
Source: GAO analysis of ASPR and CDC information. | GAO-20-372
• Joint model development. ASPR and CDC modelers jointly
developed some modeling products during outbreak responses. For
example, during the 2014-2016 Ebola response, ASPR and CDC
developed a model to estimate future numbers of Ebola patients
needing treatment at any one time in the United States. According to
a publication describing the model, policymakers have used it to
evaluate responses to the risk for arrival of Ebola-infected travelers,
and it can be used in future infectious disease outbreaks of
international origin to plan for persons requiring treatment within the
United States. 54 Building these positive working relationships can help
bridge organizational cultures by building trust and fostering
communication, which facilitates collaboration and is vital in
responding to emergencies. For example, in our 2011 report, we
found that, through interagency planning efforts, federal officials built
relationships that helped facilitate the federal response to the H1N1
influenza pandemic. 55 Similarly, HHS officials said that federal
54Rainisch G. et al. “Estimating Ebola Treatment Needs, United States”. Emerging
Infectious Diseases (2015).
55GAO-11-632.
Page 28 GAO-20-372 Infectious Disease Modeling
coordination during the H1N1 pandemic was much easier because of
these formal networks and informal relationships built during
pandemic planning activities and exercises.
• Memoranda of understanding. The four HHS agencies have
entered into various agreements through memoranda of
understanding in order to define their relationships for coordinating
infectious disease modeling (see table 3). Generally these
memoranda were between individual agencies rather than
department-wide. We found that the use of memoranda of
understanding was consistent with leading collaboration practices,
such as agreeing on roles and responsibilities. Our prior work found
that agencies that articulate their agreements in formal documents
can strengthen their commitment to working collaboratively. 56
Similarly, CDC modelers and officials said that written agreements
can reduce the possibility of misunderstandings or disagreements and
help ensure that participants have a mutual understanding of
collaboration goals. For example, in the absence of such written
agreements, the potential for duplication is increased because
agencies could be working on similar types of models without one
another’s knowledge.
Table 3. Selected Examples of Memoranda of Understanding for Coordinating on Infectious Disease Modeling
Collaborating agencies Description
The Office of the Assistant From 2013 to 2018, CDC and ASPR had a memorandum of understanding to promote collaboration,
Secretary for Preparedness provide expertise, and facilitate data and information exchange related to infectious disease modeling.
and Response (ASPR) and This agreement expired in 2018. ASPR modelers and officials told us that, as of August 2019, it had not
Centers for Disease Control been updated, and there were no plans to do so. Despite this, according to CDC modelers and officials,
and Prevention (CDC) the substance of the agreement is still being followed. CDC modelers and officials told us they continue
to collaborate with ASPR modelers on the development of models that address questions of mutual
interest. For example, for the ongoing Ebola response, CDC modelers and officials said they have kept
ASPR informed on modeling efforts, and ASPR shares data on vaccine production that is included in
one of the models.
ASPR and the Food and ASPR and FDA have a memorandum of understanding to promote collaboration and enhance
Drug Administration (FDA) knowledge and efficiency by providing for the sharing of information and expertise. This memorandum
was in place from 2012 to 2017, and was then renewed in 2019. It remains valid unless modified by
consent of both parties or terminated by either party immediately upon written notice in the event that a
federal statute is enacted or a regulation is issued by a federal partner that materially affects the
memorandum. According to FDA modelers and officials, the agreement facilitates collaboration related
to FDA’s Medical Countermeasure Initiative and FDA’s role in supporting the HHS-led Public Health
Emergency Medical Countermeasures Enterprise (PHEMCE).a FDA modelers and officials told us that
the agreement supports the frequent, ongoing collaborations between FDA and ASPR, including
collaboration related to preparedness for emerging infectious diseases. However, FDA modelers and
officials said, while no specific steps have been taken with regards to collaborating on infectious
disease modeling under the agreement, modeling assistance could be provided in the future, if needed.
56GAO-06-15.
Page 29 GAO-20-372 Infectious Disease Modeling
Collaborating agencies Description
ASPR and the National From 2013-2018, ASPR had a memorandum of understanding with NIH’s Models of Infectious Disease
Institutes of Health’s (NIH) Agent Study program to (1) enable Models of Infectious Disease Agent Study program researchers to
Models of Infectious work with ASPR as part of public health preparedness and response activities, (2) share data and
Disease Agent Study information, and (3) support model development and use in the HHS modeling hub. This agreement
programb has expired. ASPR modelers and officials told us that, as of August 2019, it has not been updated, and
there were no plans to do so.
CDC and NIH’s Models of Since 2015, CDC has had a memorandum of understanding with NIH’s Models of Infectious Disease
Infectious Disease Agent Agent Study program, to promote collaboration and facilitate the exchange of data, tools (models),
Study programb methods, and information. It was set to expire in February 2020.
ASPR and other From 2013 to 2018, ASPR had separate memoranda of understanding with the Departments of
federal agencies Defense and Homeland Security to promote collaboration, provide expertise, and facilitate data and
information exchange. The goals of the collaboration in both agreements were to explore ways to,
among other things: share analytical approaches and efforts, such as modeling and simulation tools, in
support of public health preparedness and response activities; provide personnel as needed to facilitate
analytical efforts; and share data and information. These goals were similar to those laid out in the
agreement between CDC and ASPR. These agreements expired in 2018. ASPR modelers and officials
told us that, as of October 2019, they have not been updated, and there were no plans to do so.
Source: GAO analysis of ASPR and CDC information. | GAO-20-372
a
Launched in 2010, FDA’s Medical Countermeasures Initiative is intended to coordinate medical
countermeasure development, preparedness and response within FDA.
b
NIH’s Models of Infectious Disease Agent Study program is a collaboration of research and
informatics groups to develop computational models describing the interactions between infectious
agents and their hosts, disease spread, prediction systems, and response strategies.
• Forecasting competitions. CDC and NIH have sponsored formal
forecasting competitions to improve modeling for Ebola, Zika, and
seasonal influenza. 57 According to a report from the National Science
and Technology Council, controlled, multi‐center modeling contests
and projects generate valuable insights. For example, they often show
that simpler models perform as well as more complex models and that
ensemble models, which combine the results of multiple models to
57CDC modelers told us that models are developed to model for influenza in general, not
specifically for seasonal or pandemic influenza. These officials said they model specifically
for pandemic influenza when they identify a novel influenza strain. There are different
ways a model can be modified for pandemic influenza to represent the nuances of how
the virus is shared and the disease’s dynamics. When new models are developed for
pandemic influenza, the basics of the disease’s transmission and growth are the same as
those used for seasonal influenza. However, for pandemic influenza, they are more
extreme.
Page 30 GAO-20-372 Infectious Disease Modeling
predict an outcome, perform better than an individual model. 58 Such
competitions are consistent with a leading collaboration practice we
previously reported: identifying and addressing needs by leveraging
resources. 59 In this case, such leveraging allowed CDC and NIH to
obtain additional benefits and insights on models that may not
otherwise be available. These modeling competitions can therefore
help the HHS agencies better prepare for future outbreaks through
coordination with participants. The following are examples of
forecasting competitions sponsored by CDC or NIH:
• Ebola competition. NIH’s Fogarty International Center held an
Ebola forecasting competition from August to December 2015,
related to the 2014-2016 West African Ebola outbreak, to compare
the accuracy of predictions from different Ebola models, among
other things. According to NIH modelers and officials, lessons
learned from the challenge were that (1) with regard to short-term
incidence predictions, ensemble estimates were more consistently
accurate than predictions by any individual participating model; (2)
as expected, more accurate and granular epidemiological data
improved forecasting accuracy; (3) the availability of contextual
information, including patient-level data and situational reports, is
important for accurate predictions; (4) the accuracy of forecasting
was not positively associated with more complex models; and (5)
coordination of modeling teams and comparison of different
models is important to ensure robustness of predictions.
According to NIH officials, based on these lessons and in
response to the most recent Ebola outbreak, NIH has established
a coordination group to share information about modeling and
data sharing for this particular outbreak and a formal model
comparison is underway under World Health Organization
leadership.
• Aedes (Zika) competition. In 2019, CDC hosted a forecasting
competition related to using models to predict the presence of
58National Science and Technology Council, Towards Epidemic Prediction. Simpler
models are ones that use few parameters (inputs) or equations and interactions between
those parameters or equations, while complex models are ones that use many more
parameters or equations and have more interactions between them. Ensemble modeling
is a process where multiple models that show different possible outcomes are combined
to predict an outcome. The objective of ensemble modeling is to improve the accuracy of
the model through averaging the result of multiple models. In weather forecasting,
ensemble models are used by forecasters to measure the likelihood of a forecast.
59GAO-06-15.
Page 31 GAO-20-372 Infectious Disease Modeling
Aedes mosquitoes, which is a vector for the Zika virus. Evaluating
these models can, according to CDC, help clarify model accuracy
and utility, the seasonal and geographical dynamics of these
mosquitoes, and key directions for future research. According to
CDC documentation, these advances can contribute to improved
preparedness for arboviral invasion in the United States and in
other regions where Aedes suitability may be limited and
changing. 60 CDC plans to evaluate forecasts for this competition in
early 2020, as soon as final surveillance data for 2019 are
available.
• FluSight (seasonal influenza) competition. CDC holds an
annual seasonal influenza forecasting competition—known as
FluSight—to facilitate efforts to engage external researchers to
improve the science and usability of seasonal influenza forecasts.
The results of the competition are evaluated by the CDC Influenza
Division, which works with state and local partners to determine
whether the results are useful to them and if there are other
metrics, milestones, or targets that would be more helpful in
making public health decisions. According to CDC officials in
February 2020, the results from the FluSight competition are not
directly incorporated into pandemic influenza forecasting because
the most accurate seasonal influenza forecasts would not
necessarily be the most accurate pandemic influenza forecasts.
According to these officials, the overall lessons learned from the
FluSight competition relate to how to quantify, visualize, and
communicate model results and model accuracy, as well as the
value of forecast ensembles to summarize multiple models. CDC
officials said these lessons are incorporated into pandemic
influenza forecasting plans.
• Coordination with academic and other modelers. CDC
coordinated infectious disease modeling efforts with academic and
other modelers through various means, including the following:
• Intergovernmental Personnel Act agreements. CDC has used
agreements under the Intergovernmental Personnel Act of 1970 to
60Arboviral disease is a general term used to describe infections caused by a group of
viruses spread to people by the bite of infected arthropods (insects) such as mosquitoes
and ticks. Infections usually occur during warm weather months, when mosquitoes and
ticks are active. Examples include California encephalitis, Chikungunya, dengue, Eastern
equine encephalitis, Powassan, St. Louis encephalitis, West Nile, Yellow Fever, and Zika.
Page 32 GAO-20-372 Infectious Disease Modeling
collaborate with external experts on modeling efforts. 61 For
example CDC’s Division of Vector-Borne Diseases had an
agreement from 2014 to 2017 to assign a CDC official to the
Harvard T.H. Chan School of Public Health. The agreement was
to help CDC integrate with a larger modeling community and
provide the Harvard School of Public Health with expertise in
arboviral diseases and applied public health.
• Vector-Borne Disease Centers of Excellence. CDC has funded
the Vector-Borne Disease Centers of Excellence, which are
engaged in modeling-specific projects. In 2017, CDC established
five universities as regional centers of excellence to help prevent
and rapidly respond to emerging vector-borne diseases across the
United States. According to CDC, the goals of the centers are to
build effective collaboration between academic communities and
public health organizations at federal, state, and local levels for
surveillance, prevention, and response, among other things.
• Support for other governmental entities. CDC has coordinated
with other entities—such as state and local officials—to provide
modeling tools, estimates of case counts, or effects of
interventions during the Ebola, Zika, and pandemic influenza
outbreaks. For example, CDC developed pandemic influenza
models for state and local health departments to use in influenza
pandemic planning activities. The tools are available on the CDC
pandemic influenza website and from ASPR’s emergency
preparedness information portal. As previously discussed, officials
from two of the states we spoke with said they generally were
unaware of the availability of the models. According to CDC
modelers and officials, these models were developed in the mid-
2000s for pandemic influenza planning and remain useful but had
not been a priority to update because they have not received a
request to do so.
• Informal collaboration. CDC has engaged in a range of informal
collaborations related to infectious disease modeling. According to
CDC modelers and officials, modelers often develop relationships
through conferences or other contacts. For example, CDC
modelers and officials said they informally collaborated on Ebola
61The Intergovernmental Personnel Act of 1970, Pub. L. No. 91-648, 84 Stat. 1909 (Jan. 5,
1971), as amended, codified at 5 U.S.C. §§ 3371-3375. The Intergovernmental Personnel
Act’s Mobility Program provides for the temporary assignment of personnel between the
federal government and state and local governments, colleges and universities, Indian
tribal governments, federally-funded research and development centers, and other eligible
organizations.
Page 33 GAO-20-372 Infectious Disease Modeling
modeling needs with academic institutions, as well as modelers
and analysts in the World Health Organization and other U.S.
government agencies, such as the Federal Emergency
Management Agency. For example, CDC modelers and officials
told us that model estimates produced under collaboration with
academics helped inform decisions about how many beds to be
ordered and delivered on the ground in West Africa during the
2014-2016 Ebola Outbreak. Similar to the forecasting
competitions described above, such informal coordination
mechanisms are consistent with the best practice of identifying
and addressing needs by leveraging resources, thus obtaining
additional benefits that may not be available if they were working
separately. 62 For example, we have previously reported that
informal collaboration mechanisms—such as building
relationships between key personnel and soliciting input for
research projects—can provide the opportunity to leverage
expertise. 63
HHS Agencies Do Not CDC and ASPR modelers and officials did not routinely monitor, evaluate,
Fully Monitor, Evaluate, and report on coordination efforts for infectious disease modeling. 64 While
and Report on CDC did conduct after-action reviews for Ebola and Zika, which included
a review of modeling efforts, such reviews are not routine outside of a
Coordination Efforts response and do not examine modeling coordination between agencies. 65
ASPR modelers and officials told us they saw no reason to monitor
coordination efforts under the memorandum of understanding with CDC
because such memoranda outline expectations rather than requirements.
However, we have found that agencies that create a means to monitor,
evaluate, and report the results of collaborative efforts can better identify
62GAO-06-15.
63GAO, Biodefense: Federal Efforts to Develop Biological threat Awareness, GAO-18-155
(Washington D.C.: Oct. 11, 2017).
64As previously discussed, FDA maintains a limited role in modeling for the three diseases
we focused on related solely to the blood donor bank for Zika, and NIH funds, rather than
conducts, modeling. As such, we have chosen to focus on ASPR and CDC—the agencies
which more routinely model for Ebola, Zika, and pandemic influenza—in this portion of our
discussion.
65CDC conducted an after-action review of the 2014-2016 Ebola outbreak to assess the
effectiveness of the response. Specifically, the report found that response teams and task
forces did not fully understand the role of modeling. The report also found that, when
modelers were embedded in their own task force, they had better access to response
knowledge and leadership, could more easily collaborate, and could establish their own
priorities based on all response needs.
Page 34 GAO-20-372 Infectious Disease Modeling
areas for improvement. 66 We have previously reported that progress
reviews or after action reviews can be useful mechanisms for monitoring,
evaluating, and reporting on collaborative efforts. For example, we
previously reported that, to monitor, evaluate, and report on the status of
achieving the Healthy People 2010 objectives, HHS held progress
reviews in which the federal agencies with lead responsibilities for a focus
area reported on the progress towards achieving the objectives. During
these reviews, the participating agencies discussed the data trends,
barriers to achieving the objectives, strategies undertaken to overcome
barriers, and alternative approaches to attain further progress. By holding
similar progress reviews in which CDC and ASPR evaluate and report on
coordination efforts for infectious disease modeling, these agencies could
be better positioned to identify and address challenges prior to infectious
disease outbreaks occurring, which could lead to improved responses.
Further, there is the potential for overlap and duplication of modeling
efforts across agencies, which may not be identified if coordination efforts
are not effectively being monitored, and which could lead to inefficiencies.
The memorandum of understanding between CDC and ASPR had
expired in 2018. Agency officials told us they had no plans to review or
update the agreement. According to ASPR modelers and officials, the
agreement has not been updated because it was not a priority and the
substance of the expired agreement is being followed. However, without
an active agreement in place that clearly defines the goals of the
collaborative effort and the roles and responsibilities of participants, a lack
of understanding and agreement becomes more likely, particularly as
agencies’ priorities evolve over time. Our prior work on leading
collaboration practices found that agencies that articulate their
agreements in formal documents can strengthen their commitments to
working collaboratively, and that such agreements are most effective
when they are regularly reviewed and updated. 67
Further, we found that the memorandum of understanding between ASPR
and CDC was not fully implemented when it was active. For example,
according to this agreement, CDC was to appoint a designee to
participate in a steering committee related to modeling within HHS.
However, ASPR modelers and officials told us that this steering
committee was never formed because of changing leadership and
66GAO-06-15.
67GAO-06-15.
Page 35 GAO-20-372 Infectious Disease Modeling
priorities. They told us that HHS does not have any intention to form such
a steering committee in the future. However, our past work shows
creating a steering committee or other similar coordination mechanism
could help facilitate monitoring of coordination efforts. 68
We similarly found that other memoranda of understanding related to
infectious disease modeling were not fully implemented. For example,
although ASPR had a 2013-2018 memorandum of understanding with
NIH’s Models of Infectious Disease Agency Study program, ASPR
modelers and officials said they rarely use models funded by NIH,
including those funded through the program. 69 In particular, ASPR
modelers and officials recalled only using one such model in recent years.
That model, known as “FluTE,” is an influenza model that was used as
part of a larger study on vaccine availability. However, ASPR modelers
faced challenges in using this model. Specifically, these ASPR modelers
and officials said the FluTE model initially was not compatible with
ASPR’s computer system, so software engineers had to modify the
source code to resolve the compatibility issue. The model did not have
documentation describing its parameters, according to ASPR modelers
and officials, so they had to read through the model’s source code to
understand them. Similarly, regarding a separate agreement between
ASPR and FDA, FDA modelers and officials said that, while there is
ongoing information sharing, no specific steps have been taken with
regard to collaborating on infectious disease modeling under the
agreement. However, these modelers and agency officials said that
modeling assistance could be provided in the future, if needed.
68GAO, Marine Debris: Interagency Committee Members Are Taking Action, but Additional
Steps Could Enhance the Federal Response, GAO-19-653 (Washington, D.C.: Sept. 25,
2019).
69NIH’s Models of Infectious Disease Agency Study Program is a collaborative network of
scientists who develop and use models to improve the understanding of infectious disease
dynamics.
Page 36 GAO-20-372 Infectious Disease Modeling
We identified four elements of practices for developing and assessing
CDC and ASPR models: (1) communication between decision maker and modeler, (2)
Generally Followed description of the model, (3) verification, and (4) validation. We
determined that CDC and ASPR generally followed these GAO-identified
Identified Practices practices for 10 models we reviewed. 70 However, for four of the 10
for Infectious Disease models, CDC modelers did not provide all of the details needed in the
verification steps to reproduce their model results, which is inconsistent
Modeling, but CDC with HHS guidelines on transparency and reproducibility. 71
Has Not Fully
Ensured Model
Reproducibility
CDC and ASPR Generally According to our interviews with agency modelers and experts, along with
Followed Identified our review of selected literature, there are no documented standards that
prescribe the steps agencies must or should follow when developing and
Modeling Practices but Did
assessing models. However, based on our interviews and review, we
Not Always Fully Assess identified four broad elements of the modeling process that modelers
Model Performance generally consider. They are:
1 communication between modelers and officials to refine questions to
be addressed by the model, such as geographic spread of the
disease and total cases of the disease;
2 description of the model, including detailed descriptions of
assumptions and data sources used;
3 verification; and
4 validation. 72
Figure 4 outlines the model development and assessment process.
70GAO shared with agency modelers and outside experts the practices we identified for
model development and assessment and confirmed with them that these practices were
generally used and appropriate for our analysis.
71U.S. Department of Health & Human Services, Guidelines for Ensuring and Maximizing
the Quality, Objectivity, Utility, and Integrity of Information Disseminated to the Public,
(Washington, D.C.: October 2002) (HHS Guidelines) Pt. I, D.2.c.2.
72Verification establishes that the model is running as intended by its developers.
Validation involves running the model and determining if the results are consistent with
data external to the model itself.
Page 37 GAO-20-372 Infectious Disease Modeling
Figure 4: Outline of Process to Develop Models and Assess Their Performance.
Based on our assessment of 10 selected models, we found that CDC and
ASPR generally took steps that corresponded to our four elements, and
agency modelers generally agreed with our assessment of each model.
See table 4 for more information on the elements. 73 See appendix III for a
list of models we reviewed and a complete list of the steps we identified
that make up each element.
73We reviewed seven infectious disease models from CDC and three from ASPR. Of
these models, three were for Ebola (two CDC, one ASPR), three for Zika (two CDC, one
ASPR), and four for pandemic influenza (three CDC, one ASPR).
Page 38 GAO-20-372 Infectious Disease Modeling
Table 4: Elements of Model Development
Element Description and selected steps
Communication between decision Establishes clear understanding of model questions, limitations, uncertainty, etc. This element
maker and modelera can include a discussion between a decision maker and a modeler. If done properly, it confirms
the model is designed to answer the decision maker’s questions.
Description of the model This step includes a clear description of model assumptions, limitations, inputs, outputs,
general type of model (stochastic, deterministic, etc.). It also includes a statement of the model
equations, algorithms, and software.b
Verification This element establishes that the model is running as intended by its developers. It consists of
an independent expert reviewing model programming, checking code accuracy, confirming
model code is publicly shared, and testing model assumptions and handling of input data.
Validation This element establishes that the model is providing results consistent with data external to the
model itself. It consists of comparing model output with real-world data (historical or current),
data from comparable models, and analysis of the sensitivity of model outputs to changes in
various model inputs (sensitivity analysis).
Source: GAO Analysis of documents from the Centers for Disease Control and Prevention, Office of the Assistant Secretary for Preparedness and Response, and other sources | GAO-20-372
a
For ease of reading and comprehension, Table 4 combines the elements of “Clarifying Objectives”
and “Communicating Results” in Appendix 3 into the overarching “Communication between decision
maker and modeler” element listed above.
b
The National Institute of Standards and Technology defines an algorithm as a computable set of
steps to achieve a desired result.
Communication between modeler and decision maker. In all 10
agency models we reviewed, we found that agencies took all the steps
we identified for communication between decision maker and modeler. In
some cases, these steps were formalized, while in others they were
informal. For example, CDC modelers responding to Ebola ensured
communication with decision makers by following a memo template they
developed, which has a section requiring modelers to communicate key
aspects of their model. These modelers noted, however, that they would
not follow all the steps in their memo template for models developed
during an outbreak because of time constraints. CDC modelers
responding to pandemic influenza noted they do not have formal best
practices for communication about key model aspects to decision makers,
and a CDC modeler responding to Zika highlighted the role of CDC’s
Emergency Operations Center (EOC) in communication between decision
makers and modelers, which is activated only during a response. ASPR
modelers noted that—as a best practice—they hold a discussion for all
new models, in which decision makers describe what they are looking for
and modelers describe what they can provide.
Description of the model. In nine of the 10 models we reviewed,
modelers took all steps we identified for describing their model type,
inputs, outputs, assumptions, and limitations. In one case, ASPR’s
Page 39 GAO-20-372 Infectious Disease Modeling
“flumodels” package, the agency did not carry out the step of describing
the model’s limitations. ASPR modelers told us they did not do so
because they expected the model’s intended users—primarily federal
public health modeling experts—would understand the limitations of their
model, an assumption we find reasonable.
Verification. In six of 10 models reviewed, we found agency modelers
followed most of the steps we identified for model verification. However,
in four of the seven CDC models reviewed, CDC did not publish the
model’s code, a part of model reproducibility and a model verification
step. 74 We examine CDC’s policy and efforts on reproducibility in more
detail below.
Validation. For four of the 10 models we reviewed, agencies performed
few validation steps. In all three CDC pandemic influenza models we
reviewed, and the ASPR Zika model, sensitivity analysis was the only
validation step performed. 75 CDC influenza modelers said they did not
perform other validation steps because of a lack of comparable external
models or applicable data which could be used for other types of model
validation. For example, they said they could not validate their models
using real-world data because they made projections for scenarios that
did not come to pass (e.g., an unmitigated pandemic influenza outbreak).
They said they have continued to look for comparable models that could
be used to cross-validate their model estimates. ASPR modelers
responding to the Zika outbreak also did not have access to comparable
external models or applicable data to confirm their model projections, but
have since attempted to validate their model. For the other six models we
reviewed, agencies carried out most but not all validation steps. For
example, CDC modelers responding to Zika also said they did not
perform cross-validation (comparison of different model results to each
other) for their Zika model because of a lack of comparable models.
However, these ASPR and CDC Zika modelers said they have attempted
to validate their model since its publication as new data emerges, and we
found this occurred.
74All three of the ASPR models we reviewed included published model code.
75We examined three Ebola models, three Zika models, and four pandemic influenza
models for our review. Sensitivity analysis is a model validation method that is used to
determine how much the model projections change in response to changes in input data.
Page 40 GAO-20-372 Infectious Disease Modeling
Assessing Model Validity CDC modelers and ASPR modelers responding to Zika followed identified
Assessing model validity means determining practices and validated their model projections for the Zika outbreak,
whether a model is sufficiently accurate for its although their efforts yielded mixed results for model performance. CDC
purpose. Several methods are available,
including the following:
modelers responding to Zika attempted to estimate whether there was an
• Modelers can compare the results of the enhanced risk of microcephaly in infants born to expectant mothers
model against real-world data the model infected with Zika. 76 Using data available during the initial stage of the
was designed to predict.
outbreak, they calculated the enhanced risk to be between 0.88 and 13.2
• If there are no such data, another method
is to determine how much the model
percent if the mother was infected in the first trimester. In two subsequent
projections change in response to studies using later data on the actual incidence of microcephaly as a
changes in input data. This is known as result of the outbreak, other researchers found the enhanced risk was
model sensitivity analysis.
within the bounds of CDC modelers’ earlier projections: a 10 percent
• Modelers can also withhold a part of the
available data in building the model and enhanced risk in one study and an 8.3 percent enhanced risk in the other.
then confirm the model can reproduce the In the second case, ASPR modelers attempted to estimate potential new
withheld data.
cases of Guillain-Barré syndrome, a rare disorder in which the body’s
• A method that does not rely directly on
real-world data is to run the model along immune system attacks part of its own nervous system, in places
with a separate, independent model using burdened by Zika infection. Their initial projections were that there would
the same input data, and comparing the
outputs.
be between 191 and 305 new cases in Puerto Rico, a three- to five-fold
• Model validity can also be assessed
increase above the number normally expected. ASPR modelers
through independent performance attempted to verify these results themselves and found that the incidence
evaluations. For example, agencies did increase, but only two-fold, to 123 new cases.
sometimes host modeling competitions, in
which independent modelers compare the
predictive performance of multiple models
under controlled conditions using
standardized data. The National Institutes
of Health hosted an Ebola forecasting
competition in 2015, and the Centers for
Disease Control and Prevention (CDC)
launched its FluSight competition in 2013.
During an infectious disease outbreak,
multiple validation approaches may be
necessary, since there may be insufficient
data or few comparable models. For these
reasons, experts said that model validation
should be frequent and is never complete.
Source: GAO analysis of published academic literature, US
Department of Energy documents, CDC documents and data.
| GAO-20-372
76Microcephaly is a birth defect where a baby’s head is smaller than expected when
compared to babies of the same sex and age. Babies with microcephaly often have
smaller brains that might not have developed properly.
Page 41 GAO-20-372 Infectious Disease Modeling
Agency Modelers Follow a We also found that CDC and ASPR modeling approaches varied
Variety of Approaches to somewhat, while generally remaining within the bounds of our identified
practices. For example, all the agency modeling groups reviewed their
Modeling
model assumptions, but they also varied in whether this review was
formal or informal and internal or external. CDC modelers responding to
The Challenge of Modeling During an
Outbreak.
Ebola use a formal internal peer review process during non-outbreak
Early in the 2014-2016 Ebola outbreak, periods, as well as a detailed checklist to ensure communication with
Centers for Disease Control and Prevention decision makers, full consideration of model inputs and outputs,
(CDC) officials faced the challenge of quantification of model uncertainty, and validation of the model. By
answering questions with limited data and
time. In order to estimate the potential number contrast, CDC modelers responding to Zika told us they do not have a
of future cases and to aid in planning for formal system for evaluating their models, and instead rely on their own
additional disease-control efforts, CDC
developed EbolaResponse, an Excel review of model assumptions. ASPR and CDC pandemic influenza
spreadsheet-based model that could forecast modelers told us their modeling approach also relied on peer review, but
how interventions would impact the outbreak.
the review was done by external experts; informally for ASPR and
Using EbolaResponse, CDC predicted in early
September 2014 that 1.4 million cases of formally for CDC pandemic influenza modelers.
Ebola could occur in Liberia and Sierra Leone
by January 2015, if the world health There are several reasons agency modeling approaches can vary.
community did not increase interventions.
These estimates included a correction factor According to agency modelers, agency modeling practices can be
intended to account for the underreporting of influenced by the availability of time, data, and comparable models. For
cases and that, according to officials, was to
represent model uncertainty. Partly because example, CDC pandemic influenza modelers and officials said they follow
of these estimates of rapidly increasing cases, a shortened process when facing time constraints by documenting model
CDC and others increased intervention by
sending more treatment units, personnel, and
development in a journal publication after the model has already been put
medical supplies in late 2014. to use. Similarly, CDC modelers responding to Ebola noted that, during a
EbolaResponse was created to model the response, a lack of time may mean models are not reviewed through
effects of intervention, and it later turned out CDC’s formal clearance process; instead, a more informal review of
to be unreliable for the 4-month forecast that
CDC used to support its request for increased model results may occur.
intervention. Independent analysis found that
the model could forecast cases up to a month
ahead well but could not provide any measure CDC and ASPR modelers also described variation in the complexity of
of uncertainty. Furthermore, the model was the models they use. They said they sometimes use both simple and
unable to make accurate forecasts much
beyond 3 months, a limitation that was
complex models for the same disease and during the same outbreak.
common among the models used during the CDC modelers and officials responding to Ebola said that they preferred
outbreak. CDC later reported that roughly models run in spreadsheet programs for their transparency and
8,500 cases, or 34 percent of the corrected
EbolaResponse prediction of 25,000 cases, communicability, whereas CDC influenza modelers mostly use dedicated
occurred in Liberia by the end of January statistical software programs to run models and spreadsheets for
2015.
communicating with state and local health departments. ASPR modelers
Source: GAO analysis of CDC documents and
data. | GAO-20-372 develop more complex prediction models so that they can be reused to
answer more than one question, as opposed to models run in
spreadsheet programs that are designed to answer one question.
Experts and agency modelers generally agreed that infectious disease
models should not be more complex than is necessary to answer the
questions they were developed to address. A simpler model may be run
on a variety of software programs, ranging from spreadsheet programs to
Page 42 GAO-20-372 Infectious Disease Modeling
specialized programming languages that can do statistical analysis. One
downside of models run in spreadsheet programs, according to CDC
influenza modelers, is that it is harder to conduct quality control
measures. Two experts we spoke to, along with CDC Zika modelers, also
expressed concerns with reliability and reproducibility of models run in
spreadsheet programs. 77
CDC Has Not Fully Since 2002, HHS agencies responsible for disseminating influential
Implemented a Policy to scientific, financial, or statistical information have been required to ensure
methods used to develop this information are “reproducible.” 78 A 2019
Ensure Model
report from the National Academies of Sciences, Engineering, and
Reproducibility Medicine noted that the scientific enterprise depends on the ability of the
scientific community to scrutinize scientific claims and to gain confidence
over time in results and inferences that have stood up to repeated testing.
As part of this process of scrutiny, a study’s data and code should be
made available so that the study is reproducible by others. 79 The National
Academies report defines reproducibility as obtaining consistent
computational results using the same input data, computational steps,
methods, code, and conditions of analysis. Reproducibility is specifically
77The American Statistical Association notes that, while spreadsheet tools such as
Microsoft Excel are useful for a variety of purposes, they do not consider it an ideal
environment for programming or reproducible analysis and they do not recommend using
it for the primary analysis of data.
78Section 515 of the Consolidated Appropriations Act, 2001, Pub. L. No. 106-554, App. C,
114 Stat. 2763A-125, 153-54 (Dec. 21. 2000), directed the Office of Management and
Budget to issue government-wide guidelines “that provide policy and procedural guidance
to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and
integrity of information (including statistical information) disseminated by Federal
agencies.” The Office of Management and Budget issued these guidelines in 2002. Office
of Management and Budget, Guidelines for Ensuring and Maximizing the Quality,
Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies;
Republication, 67 Fed. Reg. 8452 (Feb. 22, 2002). In accordance with the Office of
Management and Budget’s guidelines, HHS issued guidelines in 2002, (HHS Guidelines).
HHS is also subject to a number of statutory requirements that protect the sensitive
information it gathers and maintains including, but not limited to, the Health Insurance
Portability and Accountability Act of 1996, Privacy Act of 1974, and Federal Information
Security Management Act of 2014. In addition, the HHS guidelines state that agency
requirements for reproducibility and transparency in their scientific publications “does not
override other compelling interests such as privacy, trade secrets, intellectual property,
and other confidentiality protections” required by law. (HHS Guidelines, Pt. I, D.2.c.2.)
79National Academies of Sciences, Engineering, and Medicine, Reproducibility and
Replicability in Science (2019) (Washington, D.C.: National Academies Press, 2019).
Page 43 GAO-20-372 Infectious Disease Modeling
addressed earlier in this section in our discussion of model verification, a
step that requires making code available for independent review.
HHS requires its component agencies to either follow HHS department
guidelines on reproducibility or to ensure their own guidelines include a
high degree of transparency about the data and methods used to
generate scientific information. 80 HHS guidelines require that, in a
scientific context, agencies identify the supporting data and models for
their published scientific information and provide sufficient transparency
about data and methods that an independent reanalysis could be
undertaken by a qualified member of the public. When asked whether
CDC has specific policies related to reproducibility that would have
applied to provision of model code in their published scientific research,
CDC referred to its guidelines developed in response to the 2002 HHS
Guidelines. 81 However, CDC guidelines do not contain any reference to
reproducibility, models, or provision of model code. CDC guidelines for
review of scientific information provided to the public focus on
completeness, accuracy and timeliness, data management and analysis,
clarity and accuracy of presentation, and validity of interpretation of
findings. CDC’s policy on public health research and non-research data
management and access does not make any reference to reproducibility
or model code. This lack of reference to reproducibility in CDC’s
guidelines and policies is not in accordance with HHS guidelines.
Our review found four instances in which CDC modelers did not provide
model code when they published their models. CDC modelers said in
some instances, issues with publication formats made the code difficult to
share, they did not have time to produce a user-friendly version of the
code, or they would share the code upon request.
By contrast, ASPR modelers provided code for every model within our
review when they published their models. While neither agency cited a
specific HHS policy that required them to share model code, ASPR
modelers noted that their internal peer review process typically includes
sharing model source code with other modelers within PHEMCE. In our
review of HHS guidelines and agency-specific guidance for these HHS
guidelines, we found that, of three published agency guidance, two
require reproducibility, or transparency for the methods used in the
80HHS Guidelines, Pt. I, C, Pt. I, D.2.c.2.
81HHS Guidelines, Pt. I, D.2.j. Section D “Centers for Disease Control and Prevention and
Agency for Toxic Substances and Disease Registry, Subsection V, Part B.
Page 44 GAO-20-372 Infectious Disease Modeling
reports they issue to the public. Of these agencies, CDC was the only one
that did not explicitly require transparency or reproducibility.
The National Academies report noted that researchers have to be able to
understand others’ research in order to build on it. 82 This report also notes
that the ability of qualified third parties to reproduce a model using
published code is important because it can reveal mistakes in model
code, which can lead to serious errors in interpretation and reported
results. If researchers do not share an important aspect of their study,
such as their model code, it is difficult to confirm the results of their
research and ultimately produce new knowledge. One agency official
acknowledged the importance of releasing model code, noting that HHS
could benefit by ensuring policies across the agency are consistent
regarding reproducibility and transparency in modeling. By not specifically
addressing reproducibility in their policy on dissemination of scientific
information, CDC risks undermining the reliability of the scientific
information they disseminate to the public.
Based on our review of documents and reports from agencies, as well as
Modelers Faced expert and agency interviews, we identified three categories of challenges
Several Challenges that CDC modelers and officials and ASPR modelers faced when
modeling for Ebola, Zika, and pandemic influenza, along with steps they
and Have Worked to took to address the challenges. The categories are data, resources, and
Address Them communicating results. 83
Data Challenges According to a 2016 report from the National Science and Technology
Council (NSTC), obtaining timely and accurate data and information has
long been a major challenge to an effective response during an infectious
disease outbreak. 84 One expert described reliable data as a modeler’s
most limited resource. Until data of sufficient quality and quantity are
available and usable, the predictive value of models will be limited.
82National Academies of Sciences, Engineering, and Medicine, Reproducibility and
Replicability in Science.
83Within each category, officials said it may not be possible to address all challenges.
While the challenges we describe herein are drawn from modeling for Ebola, Zika, and
pandemic influenza, the categories of challenges we identified—data, resources, and
communicating results—are likely applicable to infectious disease modeling in general.
84National Science and Technology Council, Towards Epidemic Prediction.
Page 45 GAO-20-372 Infectious Disease Modeling
Agency modelers and officials provided examples of data-related
challenges, which we categorize as follows: 85
• Data Access. Public health data, according to one expert, often has
access restrictions. For example, ASPR modelers said their ability to
access data during the 2014-2016 Ebola outbreak was reduced by a
need to enter into agreements with data-owning countries in order to
obtain patient data. Modelers said there were agreements between
CDC and data owners, but further agreements would have been
required for ASPR to obtain data because the agreements did not
authorize CDC to share data with its partners. 86 In addition to the
example above, the lack of data sharing agreements during the 2014-
2016 Ebola outbreak response led to modeling projects being
delayed, according to a CDC publication. 87 ASPR modelers said their
inability to obtain data without a data-sharing agreement made it
challenging for them to developing a current, reliable estimate of
Ebola incidence before modelers could start creating future estimates
of disease incidence. They said that, as a result, they instead
developed a statistical model, which provided less reliable estimates
of future numbers of disease cases than they would have preferred.
Modelers said they worked to address this challenge by obtaining
data and indirect information through personal relationships with other
modelers. In addition to the example provided above, CDC modelers
and officials responding to Ebola described experiencing data access
challenges.
• Data availability. Without sufficient data, models may be unable to
identify an epidemic’s key drivers, which could result in misdirected
85These challenges tended to be specific to a disease or agency.
86CDC explained that it often attempts to negotiate broader terms with respect to data
sharing, but that in certain instances, the data-owning country will request that data
sharing be limited to the CDC. In those instances, other agencies wanting access to data
from that country would be free to pursue similar agreements. A World Health
Organization official explained that, when a country provides data on outbreaks, a
nuanced approach needs to be taken because countries expect their data to be
responsibly managed. This official further explained that data cannot be provided to other
countries without an arrangement with the home country.
87M. I. Meltzer, et al., Modeling in Real Time during the Ebola Response.
Page 46 GAO-20-372 Infectious Disease Modeling
intervention efforts. 88 For example, ASPR modelers noted that during
the 2015-2016 Zika outbreak response, there were substantial limits
on available data, and data that were available could be unreliable
and delayed. They said it was very difficult, and in many cases
effectively impossible, to determine the accuracy of forecasting
models for the evolving Zika outbreak. In addition, CDC officials and
modelers responding to Ebola, Zika, and influenza described
encountering limits on available data as an ongoing challenge. Steps
that modelers said they have taken to address data availability
challenges include designing models to use a minimum amount of
data, building trust and communication with stakeholders who might
be able to provide additional data, and updating data systems to
provide all available information. 89 According to CDC modelers, data
availability will likely continue to pose a challenge to public health
responses.
• Data collection. There is limited manpower during an infectious
disease outbreak response, which can limit the health care system’s
ability to collect data, according to CDC modelers and officials
responding to Ebola and ASPR modelers. 90 ASPR modelers said if a
provider has to fill out a time-consuming form, then they will be
delayed in treating the next patient. In order to address this challenge,
CDC modelers and officials and ASPR modelers said data requesters
should ask for the minimum amount of data needed. For example,
CDC modelers and officials said they focus on understanding what
88We have previously reported on challenges related to data integration at the Department
of Homeland Security’s National Biosurveillance Integration Center. These challenges are,
in part, based on a lack of data. GAO, Biodefense: The Nation Faces Long-Standing
Challenges Related to Defending Against Biological Threats, GAO-19-635T (Washington,
D.C.: June 26, 2019).
89Influenza officials explained that data may not be available at the start of a disease
outbreak and different entities may be reluctant to share data. Further, information may be
backfilled into the system as it becomes available, officials said. Data systems are
updated weekly, according to officials, and when officials find out about a missing data
source. At this time, data may be revised, but officials said, the unrevised data is also
preserved.
90CDC officials said, following a response, CDC maintains and stores data and other
information including after-action reports in response-specific files—data reported through
data collection systems would continue to be stored according to the data system’s
considerations. Officials noted they developed an open repository of data collected from
external sources in relation to Zika. In order to address data challenges, ASPR officials
told us they retain all data and source code from prior analyses, including those performed
for responses, which can, if applicable, be used in new analyses. However, officials said
they do not aim to be an official repository for such data.
Page 47 GAO-20-372 Infectious Disease Modeling
data are essential, how they are collected, and the policy implications
of reporting those data. A 2016 NSTC report recommended the
federal government address this challenge by identifying questions
likely to arise during an outbreak response, in order to help define and
prioritize data collection and modeling goals. 91
• Data quality. Experts said creating models with low-quality data can
result in inaccurate models that may not provide clear answers to
decision maker questions. For example, CDC modelers and officials
responding to the 2015-2016 Zika outbreak said the data quality
varied, based on many factors such as surveillance systems that were
doing different things and defining reporting Zika cases differently,
and the availability of diagnostic testing. 92 Because of data quality
concerns, there were questions about whether modeling could be
conducted, but through discussions modelers and agency officials
said they were able to address challenges. To address such
challenges, CDC modelers and officials responding to Zika said they
worked to improve public data sharing, sent an official to the Pan-
American Health Organization to help interpret data and understand
the outbreak from an international perspective, and used modeling
methods appropriate for data with high levels of uncertainty. In
addition to the example provided above, CDC modelers and officials
responding to Ebola, ASPR modelers, and experts described
experiencing data quality challenges.
• Data integration. CDC modelers and officials responding to Ebola
and Zika also faced the challenge of integrating multiple data sets,
which may not be standardized or in a readily usable form. For
example, CDC modelers and officials responding to Zika found it
challenging to integrate data as the definition of the disease was
refined over time. As the definition got more specific and monitoring
systems became available, it was hard to establish data trends, these
officials said. Further, there were variations in who would be tested,
with all people who exhibited symptoms being tested in some areas,
and only pregnant women in others, and also when data would be
placed into a combined form and reported to state, national, or
international officials, according to these officials. This integration
issue may have complicated efforts to conduct modeling such as
determining the risk of microcephaly in infants over time. In order to
91National Science and Technology Council, Towards Epidemic Prediction.
92We have previously reported on the challenges faced by manufacturers of Zika
diagnostic tests, including those related to research and development, testing, and
regulatory approval. GAO-17-445.
Page 48 GAO-20-372 Infectious Disease Modeling
address this challenge, Zika modelers said they set up an online data
repository to, among other things, standardize shared data.
Resource-Related CDC modelers and officials responding to Ebola and Zika, along with
Challenges experts, said finding staff with sufficient training to support modeling
during an infectious disease outbreak represented an ongoing
challenge. 93 For example, CDC modelers responding to Zika said it can
be difficult to find modelers with both an epidemiological background and
skills in coding and mathematics. 94 Modelers and agency officials said
those who had the correct skills were in high demand, and it was difficult
to fully engage them in the Zika outbreak response. They said they could
have conducted more modeling or completed modeling efforts more
rapidly if they had had access to more modelers with the right skills. To
address this challenge, modelers participate in trainings on how to
communicate what models can and cannot do, participate in working
groups that support modeling efforts, employ the Intergovernmental
Personnel Mobility Act Program, maintain collaborations with external
partners, and host students and researchers. 95 ASPR modelers said they
faced personnel challenges in their modeling efforts but that they were
wide-ranging and not specific to Ebola, Zika, or pandemic influenza. 96
According to a 2016 NSTC report, time constraints make it challenging for
researchers to keep up with scientific literature during an outbreak. 97 CDC
influenza modelers said they faced this challenge and that they conduct
93CDC Influenza modelers said they had previously experienced this challenge but had
since built up their modeling capacity and maintained relationships with modeling groups,
so it was not hard to identify additional modelers.
94According to CDC, many of the people who model at CDC consider themselves
epidemiologists, economists or statisticians, rather than modelers. Some studied modeling
as a part of their graduate work, while others acquired modeling skills from short courses
offered by universities and professional organizations, or from participating in modeling
projects.
95Intergovernmental Personnel Act agreements authorize the temporary assignment of
certain employees between the federal government and state, local and Indian tribal
governments, institutions of higher education and other eligible organizations, for up to 2
years, a period that may be extended.
96We have previously reported on strengthening the science and technology workforce as
a consideration for maintaining U.S. competitiveness through transformational
technological advances. GAO, Science and Technology: Considerations for Maintaining
U.S. Competitiveness in Quantum Computing, Synthetic Biology, and Other Potentially
Transformational Research Areas, GAO-18-656 (Washington, D.C.: Sept. 26, 2018).
97National Science and Technology Council, Towards Epidemic Prediction.
Page 49 GAO-20-372 Infectious Disease Modeling
weekly searches for new influenza publications, which normally identify
about 150 publications each week. To address this challenge, modelers
said they conduct literature searches, share the responsibility of reviewing
publications and informing others of their content, talk to experts, and
attend conferences. Modelers said this challenge was more easily
addressed than others.
Communication Communicating model results can be difficult and, as modelers and
Challenges agency officials pointed out, decision makers will not give credence to
results from a model they do not understand. Model results, according to
CDC influenza modelers, are often nuanced and complicated, and
officials have to think about what pieces of information are the most
important to convey to a decision maker, the public, or health officials.
Furthermore, as one expert noted, the complexities of modeling can get
lost in translation, especially with the media, which may focus on only a
worst-case scenario. When modeling for infectious diseases,
appropriately communicating complex information has been described as
a constant challenge, and CDC influenza modelers described it as their
biggest challenge. CDC influenza modelers particularly noted the
challenge of communicating uncertainty. 98 CDC influenza and ASPR
modelers said if decision makers did not understand the models, they
could misunderstand the results, which, according to ASPR modelers,
could lead to errors in decision making. CDC modelers and officials
responding to Ebola and Zika, CDC influenza modelers, ASPR modelers,
and experts described experiencing challenges communicating model
results to decision makers.
Clear communication may help prevent misunderstandings. For example,
one review article said officials may not understand what models can and
cannot do before an epidemic, and modelers may not be fully aware of a
decision maker’s needs. 99 An expert said there is a need to constrain the
use of models intended to inform decisions so that the model does not
over- or under-influence a decision maker. And, according to ASPR
modelers, decision makers sometimes want a model to make a decision
for them, although models can only inform the decision making process.
98Scientific information typically has some level of associated uncertainty. Uncertainty can
arise from many sources such as data and limits to scientific understanding. Through
quantifying uncertainty, scientists can compare their results, identify factors that contribute
to uncertainty and other information that may affect the results, and assess the level of
confidence in the results.
99B.Y. Lee, L. A. Haidari, and M.S. Lee, “Modelling during an emergency: the 2009 H1N1
influenza pandemic,” Clinical Microbiology and Infection, Vol 19, 2013.
Page 50 GAO-20-372 Infectious Disease Modeling
They said this is less of a problem during an outbreak response, when
decision makers know they have to act based on incomplete information.
Some steps officials described taking to address communication
challenges were similar across CDC and ASPR officials. For example,
CDC modelers and officials and ASPR modelers said they took steps to
improve communication, such as working to develop relationships outside
of an outbreak and to improve how data are visualized. For example,
ASPR modelers and officials said they provided decision makers with a
website that displays an interactive influenza model known as ShinyFlu.
The website lets users adjust a model to see how its results could change
based on its inputs used. However, modelers said this only works if the
decision maker is willing to engage with data.
Other steps to address communication challenges were not discussed by
all modelers we spoke to. For example, ASPR modelers said that, when
they use models with high uncertainty, they do additional research to
assess and communicate how a model could be misrepresenting a real-
world problem. Additionally, CDC modelers responding to Zika and CDC
influenza modelers said they sometimes use the language of weather
forecasting—which provides information on the risk of an event occurring
over a specified period of time—to help communicate model outcomes. 100
For all 10 of the models we reviewed, modelers communicated all the
information they had agreed to provide to decision makers, including
information about model uncertainty. Agency modelers and officials said
they provided this information through discussions with decision makers
and by showing decision makers the results of multiple modeling
situations to convey uncertainty.
Infectious disease modeling is one tool that can provide decision makers
Conclusions with valuable information to support outbreak preparedness and
response. In particular, modeling can help answer questions that are
difficult to address in other ways because of practical, ethical, or financial
reasons. Federal agencies have recognized the importance of modeling.
CDC and ASPR reported using it to inform policy and planning questions
and, to a more limited extent, to inform planning and the use of resources.
100CDC modelers responding to Zika described potential challenges with using weather
forecasting language for infectious disease modeling, noting that individuals respond
differently to weather events.
Page 51 GAO-20-372 Infectious Disease Modeling
HHS agencies that work on infectious disease modeling—ASPR, CDC,
FDA, and NIH—reported using multiple mechanisms to coordinate their
modeling efforts, including working groups, memoranda of understanding,
and coordination with academic and other external modelers. The use of
these mechanisms was consistent with many leading collaboration
practices, such as defining and articulating a common outcome and
addressing needs by leveraging resources. However, HHS does not
routinely monitor and evaluate its coordination efforts, as called for by
another leading collaboration practice, which limits the department’s
ability to identify areas for improvement. Further, there is the potential for
overlap and duplication of modeling efforts across agencies, which may
not be identified if coordination efforts are not effectively being monitored,
and could lead to inefficiencies. By holding progress reviews in which
CDC and ASPR evaluate and report on coordination efforts for infectious
disease modeling, these agencies could be better positioned to identify
and address challenges prior to infectious disease outbreaks, which could
lead to improved response efforts.
CDC and ASPR modelers generally followed GAO-identified modeling
practices, with the notable exception of model verification. Specifically,
CDC did not make model code available to others for four of the seven
CDC models we reviewed. HHS does not have a policy that requires its
agencies to share model code, but it does require its component agencies
to either follow its guidelines or ensure that their own guidelines include a
high degree of transparency to facilitate reproducibility by qualified third
parties. Without sharing code and other important information, CDC
cannot ensure that its models are reproducible, a key characteristic of
reliable, high-quality scientific research.
In order to facilitate HHS infectious disease modeling efforts, we are
Recommendations for making two recommendations.
Executive Action
• The Secretary of Health and Human Services should develop a
mechanism to routinely monitor, evaluate, and report on coordination
efforts for infectious disease modeling across multiple agencies.
(Recommendation 1)
• The Secretary of Health and Human Services should direct CDC to
establish guidelines that ensure full reproducibility of CDC’s research
by sharing with the public all permissible and appropriate information
needed to reproduce research results, including, but not limited to,
Page 52 GAO-20-372 Infectious Disease Modeling
model code.
(Recommendation 2)
We provided a draft of this report to the Department of Health and Human
Agency Comments Services (HHS) for review and comment. In its comments, reproduced in
and Our Evaluation appendix IV, HHS agreed with our recommendations and noted that it
was developing a process to coordinate its infectious disease modeling
efforts across its components.
With regard to our second recommendation—that HHS should direct CDC
to establish guidelines that ensure the full reproducibility of CDC’s
research by sharing all permissible and appropriate information needed to
reproduce research results, including, but not limited to, model code—
HHS’s comments indicated that CDC believes it has already completed
actions to implement this recommendation.
• For example, the HHS comments state that CDC has established
policies such as “Public Access to CDC Funded Publications” and
“Policy on Public Health Research and Nonresearch Data
Management and Access” that ensure that results are made
available to the public, as appropriate. However, as we state in
our report, these policies do not contain any reference to
reproducibility, models, or provision of model code and therefore
do not fully address our recommendation.
• CDC also said in the HHS comments that its methods—including
its practice of providing a copy of model code upon request—are
in line with standard practice in the scientific community and peer-
reviewed journals. However, in the four instances we identified
where CDC modelers did not share code, code being available
upon request was only one of the reasons cited. Further, this
practice is inconsistent with those of the other HHS agencies we
reviewed, and may limit the ability of external researchers to
confirm the results of CDC’s research and ultimately produce new
knowledge.
As noted in our report, by not specifically addressing reproducibility in its
policies on access to data and publications, CDC risks undermining the
reliability of scientific information disseminated to the public. Therefore,
we did not change our recommendation in response to HHS’s comments.
We did, however, revise our report to include information on other HHS
agency policies related to reproducibility.
HHS also provided technical comments, which we incorporated as
appropriate.
Page 53 GAO-20-372 Infectious Disease Modeling
As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies of this report to the
appropriate congressional committees, the Secretary of Health and
Human Services, and to other interested parties. In addition, this report
will be available at no charge on the GAO website at http://www.gao.gov.
If you are your staff have questions about this report, please contact
Timothy M. Persons, Chief Scientist, at (202) 512-6888 or
personst@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this report.
Key contributors to this report are listed in appendix V.
Carol A. Gotway Crawford, PhD
Chief Statistician
Timothy M. Persons, PhD
Chief Scientist and Managing Director,
Science, Technology Assessment,
and Analytics
Page 54 GAO-20-372 Infectious Disease Modeling
Appendix I: Objectives, Scope, and
Appendix I: Objectives, Scope, and
Methodology
Methodology
In conducting our review of infectious disease modeling by the
Department of Health and Human Services (HHS) agencies, our
objectives were to (1) examine the extent to which HHS has used various
types of models to inform policy, planning, and resource allocation for
public health decisions for selected infectious diseases, (2) examine the
extent to which HHS coordinated their modeling efforts for selected
infectious diseases, (3) examine the steps HHS generally took to develop
and assess the performance of its models for the selected diseases and
steps it applied to a selection of infectious disease models, and (4)
describe the extent to which HHS has addressed challenges related to
modeling for selected infectious diseases.
For purposes of this review, we focused on HHS because of its focus on
scientific and technical issues related to disease modeling, role in
infectious disease outbreak preparedness and response activities, and
use of modeling for policy and regulatory issues related to disease. Within
HHS, we identified four agencies—HHS’s Office of the Assistant
Secretary for Preparedness and Response (ASPR), the Centers for
Disease Control and Prevention (CDC), National Institutes of Health
(NIH), and Food and Drug Administration (FDA)—which may develop or
use infectious disease models.
To inform all four objectives, we selected three naturally-occurring
infectious diseases that have pandemic or epidemic potential—Ebola
virus disease (Ebola), Zika virus disease (Zika), and pandemic
influenza—to use as examples of broader infectious disease modeling
efforts. We selected these diseases based on document review, their
inclusion on NIH’s pathogen priority list, modeling being conducted by
HHS agencies, and interviews with experts that we selected based on
their experience with infectious disease. 1 Based on these steps, the team
selected diseases that fit into one of the three categories on NIH’s
pathogen priority list: the disease (1) can be transmitted easily from
person to person, resulted in a high mortality rate and had the potential
for major public health impact, might cause social disruption, and may
require special action for public health preparedness (Ebola), (2) was
moderately easy to disseminate, and required specific enhancements for
diagnostic capacity and enhanced disease surveillance (Zika), or (3) was
1NIH’s pathogen priority list is comprised of over 60 pathogens across the three
categories. We selected experts based on judgmental sampling, conversations with
individuals who work in the infectious disease or modeling field, their expertise in the area
of modeling related to infectious disease, a review of papers published by the experts, and
a snowball sampling to include additional experts.
Page 55 GAO-20-372 Infectious Disease Modeling
Appendix I: Objectives, Scope, and
Methodology
an emerging pathogen that could be engineered for mass dissemination
in the future because of availability, ease of production and
dissemination, and have the potential for high morbidity and mortality
rates and major health impacts (pandemic influenza).
HHS Use of Models to To examine the types of models developed by HHS agencies to inform
Inform Policy, Planning, policy, planning, and resource allocation decisions, we reviewed
documents from 2009—the year of the last pandemic influenza outbreak
and Resource Allocation
in the United States—to April 2019 to identify examples of models
Decisions developed by the agencies for the three selected diseases. For context
on and examples of the types of modeling that CDC and ASPR have
conducted, we reviewed published articles that CDC and ASPR officials
and experts provided to us or cited during the course of our review, such
as articles identified during interviews which we later obtained. We also
obtained selected internal memoranda, when available, that described
models used in the Ebola virus outbreak. We did not include FDA and
NIH in this review because FDA has a limited role in modeling, and NIH
generally funds, rather than conducts, modeling. 2 This review yielded
articles and memoranda describing about 60 CDC and ASPR models.
See appendix II for a bibliography of model publications reviewed. We
then categorized the models using categories derived from a federal
working group report to characterize the types of modeling conducted and
the purpose of the modeling, when that purpose was identified.
To analyze each study, one analyst initially coded each study, and each
classification was then independently reviewed to verify that it had been
correctly classified and to resolve any categorization discrepancies. We
used these categories to describe types of modeling efforts undertaken
by HHS agencies. Because we focused on studies published between
2009 and 2019, our findings are not generalizable to models that were
developed outside of that time period. Additionally, because we relied on
agency officials or reviews of relevant agency documents and
publications to identify studies, we may not have captured all studies
relevant to our scope. Further, because CDC and ASPR modelers and
officials said that they do not publish every model they conduct, our
review was not intended to develop an inventory of the modeling
conducted during the time period. Therefore, we were unable to
2FDA had limited modeling activity, specifically related to impacts to the pool of blood
donors during the Zika outbreak response. NIH funds modeling outside the agency.
Page 56 GAO-20-372 Infectious Disease Modeling
Appendix I: Objectives, Scope, and
Methodology
determine the extent to which the models we identified represented
agency modeling efforts as a whole.
To describe the extent of model use for public health decision making, we
interviewed officials from HHS agencies identified as decision makers for
conducting the response to these selected diseases—CDC, ASPR, and
FDA—and officials who conducted the modeling. 3 We also interviewed
two NIH institutes and one center about funding for research related to
modeling for the selected diseases. Additionally, we conducted semi-
structured interviews of officials from five states concerning their use of
models prepared by HHS agencies for decision making, among other
topics. We selected these states based on a review of a CDC draft report
on states’ use of CDC models, on the level of influenza activity
experienced by states, and consideration of geographic variation by U.S.
region. During our review, we sought to identify the common types of
decisions that could be informed by models, as well as the considerations
that could impact the extent to which a decision maker requests and uses
models for specific types of decisions. Based on interviews with agency
officials and our review of HHS models we identified examples of models
that were used to make specific decisions during response and non-
response times. Because we relied on officials to describe the extent to
which models inform decision making, we may not have captured all
relevant instances when models for the selected infectious diseases
informed decision makers.
HHS Coordination of To examine coordination and collaboration across HHS agencies, we
Modeling Efforts reviewed documents describing HHS agencies’ collaboration and
coordination mechanisms such as Memoranda of Understanding,
descriptions of Emergency Operations Center procedures, and after-
action reports following infectious disease outbreaks. We also conducted
interviews with and requested information from HHS officials, asking them
to provide information on their efforts to coordinate their infectious
disease modeling activities. In this report, and in our past work, we define
coordination broadly as any joint activity that is intended to produce more
public value than could be produced when organizations act alone. We
3For purposes of this review, we attribute comments from HHS officials to modelers,
officials, or modelers and officials.
Page 57 GAO-20-372 Infectious Disease Modeling
Appendix I: Objectives, Scope, and
Methodology
compared these actions to relevant selected collaboration leading
practices: 4
• define and articulate a common outcome;
• establish mutually reinforcing or joint strategies;
• identify and address needs by leveraging resources;
• agree on roles and responsibilities;
• establish compatible policies, procedures, and other means to operate
across agency boundaries; and
• develop mechanisms to monitor, evaluate, and report on results.
Because we judgmentally selected a group of experts and diseases, the
results of our review cannot be generalized to HHS coordination efforts
for other infectious diseases. However, our assessment of collaboration
and coordination activities did cover modeling efforts for the three
selected diseases.
Developing Infectious To identify steps that are generally considered when modelers develop
Disease Models and infectious disease models and assess their performance, we conducted
semi-structured interviews with relevant experts from academia and other
Assessing Their
organizations and CDC and ASPR officials, and reviewed literature
Performance identified by experts.
We used a snowball sampling approach to identify relevant experts and
groups. We initially identified five infectious disease modeling experts
through informal conversation with individuals working in the field,
infectious disease modeling experts known through GAO work, as well as
a review of websites, publications, and grants funded by NIH. Using a
snowball sampling approach, we reviewed key literature related to the
steps generally taken to develop models and assess their performance,
consulted with infectious disease modeling experts, and interviewed
agency officials to identify relevant groups, as well as individual experts,
4GAO, Results-Oriented Government: Practices That Can Help Enhance and Sustain
Collaboration among Federal Agencies, GAO-06-15 (Washington, D.C.: Oct. 21, 2005).
We excluded the remaining leading practices—reinforce individual accountability for
collaborative efforts through performance management system and reinforce agency
accountability for collaborative efforts through agency plans and reports—because
performance management and a review of agency strategic planning documents fell
outside of the scope of our review.
Page 58 GAO-20-372 Infectious Disease Modeling
Appendix I: Objectives, Scope, and
Methodology
who could convey to us the steps generally taken during infectious
disease modeling.
Through literature searches, the team identified literature from public
health journals or other major sources. The team applied personal
background and knowledge in public health, infectious disease modeling,
and statistics to help identify key sources. For the selected literature, we
reviewed references and used a snowball approach to identify further
relevant studies. Finally, we reviewed CDC guidance on decision making
for data access and long-term preservation as it related to documentation
standards.
Based on our review of identified literature, we developed a data
collection instrument to assess the extent to which CDC and ASPR used
the steps for infectious disease model development identified by experts
and in the literature. 5 Through this data collection instrument, we
gathered information about the elements of developing and assessing
model performance and the steps that could be taken within each
element. In order to develop the data collection instrument, based on our
review of literature, we mapped out steps to develop and assess model
performance, and developed broad categories of assessment elements.
Within each assessment element, we included steps modelers could take
as a part of each assessment element. For example, the data collection
instrument included items that recorded model verification steps that
might have been taken by modeler(s) within the broader model
verification element. The instrument was reviewed by internal
stakeholders, who provided feedback on its content. Prior to sending the
data collection instrument to the agency, we filled in information on
verification steps taken for each of the 10 selected models, based on
provided model documentation to reflect steps we determined modelers
took as a part of the model development and assessment process. In
order to provide officials with this information, two analysts reviewed each
model’s documentation, with one analyst providing an initial coding of the
model and the other reviewing and verifying the first analyst’s findings.
This method was first tested on one of the 10 selected models by two
analysts independently coding information from the model’s
documentation into the data collection instrument and then reviewing
coding choices to reconcile any differences found. We then sent the
instruments with filled-in information to CDC and ASPR modelers to
5We did not include NIH and FDA in this review because NIH funds, rather than conducts
modeling, and FDA has a limited role in modeling for our selected diseases.
Page 59 GAO-20-372 Infectious Disease Modeling
Appendix I: Objectives, Scope, and
Methodology
receive their feedback concerning the steps taken to develop models and
assess their performance, provide any missing information, and resolve
any ambiguities. See Appendix III for a list of the 10 selected models
reviewed and steps to develop and assess model performance included
in the data collection instrument. The data collection instrument was
intended to record whether a specific step had been taken, but did not
assess the quality of the modeling steps.
In order to determine steps CDC and ASPR took to develop and assess
its models, we selected a non-generalizable sample of 10 models for
review in our data collection instrument that demonstrated steps that HHS
agencies took to develop models and assess their performance. The
model selection process described above informed our selection of
infectious disease models. To be selected for inclusion in our non-
generalizable sample, the model had to be (1) developed by CDC, or
ASPR officials or contractors; (2) developed to answer a question about
Ebola, Zika, or pandemic influenza; and (3) used to inform public health
decision makers during an outbreak or for preparedness activities. We
selected 10 models that differed in form and answered different types of
questions, which included studies prepared during both outbreak
preparedness and response times, and covered topics such as the impact
of vaccination programs on deaths and hospitalization. For Ebola and
Zika, we focused on review of selected papers or memos produced since
2014 in order to capture the time period following the 2014-2016 Ebola
and 2015-2016 Zika outbreaks. For pandemic influenza, we focused on
papers and memos produced since 2009, when the H1N1 pandemic
occurred in the United States. Because we selected from a group of
models identified by HHS modelers and officials for Ebola, Zika, and
pandemic influenza, the results of our review cannot be generalized to
other diseases outside of the scope of this report. Furthermore, we
requested models that informed public health decision making, and did
not consider models that were not used for this purpose. Because we
reviewed a non-generalizable sample of 10 models, the results of our
review cannot be generalized to a larger population of models prepared
by HHS agencies.
Challenges to Effective To identify challenges associated with modeling for the selected
Modeling infectious diseases, we reviewed documents and reports to identify
modeling challenges and steps to address those challenges, and
interviewed agency officials and modelers, and experts identified through
the previously-described snowball sampling methodology. We used semi-
structured interview protocols that included open-ended questions about
challenges associated with infectious disease modeling and limitations
Page 60 GAO-20-372 Infectious Disease Modeling
Appendix I: Objectives, Scope, and
Methodology
associated with model development. Not all officials and experts we
interviewed provided comments on every challenge or limitation. In
addition, because we judgmentally selected a group of experts and
diseases, the results of our review cannot be generalized to all infectious
disease modeling efforts.
We conducted this performance audit from May 2018 to May 2020, in
accordance with generally accepted government auditing standards.
These standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe the
evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Page 61 GAO-20-372 Infectious Disease Modeling
Appendix II: Bibliography of Selected Model
Appendix II: Bibliography of Selected Model
Publications Reviewed
Publications Reviewed
Ebola Models Carias, Cristina, et al. “Preventive Malaria Treatment for Contacts of
Patients with Ebola Virus Disease in the Context of the West Africa 2014-
15 Ebola Virus Disease Response: An Economic Analysis.” The Lancet
Infectious Diseases, vol. 16, no. 4 (April 2016): pp. 449-458.
Christie, Athalia, et al. “Possible Sexual Transmission of Ebola Virus—
Liberia, 2015.” Morbidity and Mortality Weekly Report, vol. 64, no. 17
(May 8, 2015): pp. 479-481.
Martin I. Meltzer, et al. “Estimating the Future Number of Cases in the
Ebola Epidemic - Liberia and Sierra Leone, 2014-2015.” Morbidity and
Mortality Weekly Report, vol. 63, no. 3 suppl. (September 26, 2014): pp.
1-14.
Meltzer, Martin I., et al. “Modeling in Real Time during the Ebola
Response.” Morbidity and Mortality Weekly Report, vol. 65, no. 3 suppl.
(July 8, 2016): pp. 85-89.
Rainisch, Gabriel, et al. “Estimating Ebola Treatment Needs, United
States.” Emerging Infectious Diseases, vol. 21, no. 7 (July 2015): pp.
1273-1275.
Rainisch, Gabriel, et al. “Regional Spread of Ebola Virus, West Africa,
2014.” Emerging Infectious Diseases, vol. 21, no. 3 (March 2015): pp.
444-447.
Undurraga, Eduardo A., Cristina Carias, Martin I. Meltzer, Emily B. Kahn.
“Potential for Broad-Scale Transmission of Ebola Virus Disease during
the West Africa Crisis: Lessons for the Global Health Security Agenda.”
Infectious Diseases of Poverty, vol. 6, no. 159 (2017).
Washington, Michael L., Martin I. Meltzer. “Effectiveness of Ebola
Treatment Units and Community Care Centers Liberia, September 23-
October 31, 2014.” Morbidity and Mortality Weekly Report, vol. 64, no. 3
(January 30, 2015): pp. 67-69.
Zika Models Adamski, Alys, et al. “Estimating the Numbers of Pregnant Women
Infected with Zika Virus and Infants with Congenital Microcephaly in
Colombia, 2015–2017.” Journal of Infection, vol. 76 (2018): pp. 529-535.
Dirlikov, Emilio, et al. “Guillain-Barré Syndrome and Healthcare Needs
during Zika Virus Transmission, Puerto Rico, 2016.” Emerging Infectious
Diseases, vol. 23, no. 1 (January 2017): pp.134-136.
Page 62 GAO-20-372 Infectious Disease Modeling
Appendix II: Bibliography of Selected Model
Publications Reviewed
Ellington, Sascha R., et al. “Estimating the Number of Pregnant Women
Infected With Zika Virus and Expected Infants With Microcephaly
Following the Zika Virus Outbreak in Puerto Rico, 2016.” JAMA
Pediatrics, vol. 170, no. 10 (2016): pp. 940-945.
Grills, Ardath, et al. “Projected Zika Virus Importation and Subsequent
Ongoing Transmission after Travel to the 2016 Olympic and Paralympic
Games—Country-Specific Assessment, July 2016.” Morbidity and
Mortality Weekly Report, vol. 65, no. 28 (July 22, 2016): pp.711-715.
Johansson, Michael A., et al. “Zika and the Risk of Microcephaly.” The
New England Journal of Medicine, vol. 375 (July 7, 2016): pp.1-4.
Johnson, Tammi L., et al. “Modeling the Environmental Suitability for
Aedes (Stegomyia) aegypti and Aedes (Stegomyia) albopictus (Diptera:
Culicidae) in the Contiguous United States.” Journal of Medical
Entomology, vol. 54, no. 6 (November 7, 2017): pp. 1605-1614.
Mitchell, Patrick K. et al., “Reassessing Serosurvey-Based Estimates of
the Symptomatic Proportion of Zika Virus Infections.” American Journal of
Epidemiology, vol. 188, no. 1 (January 2019): pp. 206-213.
Mier-y-Teran-Romero, Luis, Mark J. Delorey, James J. Sejvar, Michael A.
Johansson. “Guillain-Barré Syndrome Risk Among Individuals Infected
with Zika Virus: a Multi-Country Assessment.” BMC Medicine, vol. 16, no.
67 (2018).
Mier-y-Teran-Romero, Luis, Andrew J. Tatem, Michael A. Johansson.
“Mosquitoes on a Plane: Disinsection Will Not Stop the Spread of Vector-
Borne Pathogens, a Simulation Study.” PLoS Neglected Tropical
Diseases, vol. 11, no. 7 (July 3, 2017).
Reefhuis, Jennita, et al. “Projecting Month of Birth for At-Risk Infants after
Zika Virus Disease Outbreaks.” Emerging Infectious Diseases, vol. 22,
no. 5 (May 2016): pp. 828-832.
Russell, Steven, et al. “Detecting Local Zika Virus Transmission in the
Continental United States: A Comparison of Surveillance Strategies.”
PLoS Currents Outbreaks (November 22, 2017).
Watts. Alexander G., et al. “Elevation as a Proxy for Mosquito-Borne Zika
Virus Transmission in the Americas.” PLoS ONE, vol. 12, no. 5 (May 24,
2017).
Page 63 GAO-20-372 Infectious Disease Modeling
Appendix II: Bibliography of Selected Model
Publications Reviewed
Influenza Models 1 Atkins, Charisma Y., et al. “Estimating Effect of Antiviral Drug Use during
Pandemic (H1N1) 2009 Outbreak, United States.” Emerging Infectious
Diseases, vol. 17. no. 9 (September 2011): pp. 1591-1598.
Biggerstaff, Matthew, et al. “Estimates of the Number of Human Infections
With Influenza A(H3N2) Variant Virus, United States, August 2011–April
2012.” Clinical Infectious Diseases, vol. 57, suppl. 1 (2013): pp. S12-S15.
Biggerstaff, Matthew, et al. “Estimating the Potential Effects of a Vaccine
Program Against an Emerging Influenza Pandemic—United States.”
Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S20-S29.
Carias, Cristina, et al. “Potential Demand for Respirators and Surgical
Masks during a Hypothetical Influenza Pandemic in the United States.”
Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S42-S51.
Cauchemez, Simon, et al. “Role of Social Networks in Shaping Disease
Transmission during a Community Outbreak of 2009 H1N1 Pandemic
Influenza.” Proceedings of the National Academy of Sciences of the
United States, vol. 108, no. 7 (February 15, 2011): pp. 2825-2830.
Dawood, Fatimah S., et al. “Estimated Global Mortality Associated with
the First 12 Months of 2009 Pandemic Influenza A H1N1 Virus
Circulation: a Modelling Study.” The Lancet Infectious Diseases, vol. 12
(September 2012): pp. 687-695.
Fung, Isaac Chun-Hai, et al. “Modeling the Effect of School Closures in a
Pandemic Scenario: Exploring Two Different Contact Matrices.” Clinical
Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S58-S63.
Iuliano, A. Danielle, et al. “Estimates of Global Seasonal Influenza-
Associated Respiratory Mortality: a Modelling Study.” The Lancet, vol.
391, no. 10127 (March 31, 2018): pp. 1285-1300.
Jain, Seema, et al. “Hospitalized Patients with 2009 H1N1 Influenza in the
United States, April–June 2009.” The New England Journal of Medicine,
vol. 361, no. 20 (November 12, 2009): pp. 1935-1944.
1While only pandemic influenza is within the scope of this audit, CDC officials said they
use seasonal influenza modeling to prepare for a pandemic and that seasonal models can
be used in a pandemic. Therefore, we are including seasonal models in our review of
pandemic influenza modeling.
Page 64 GAO-20-372 Infectious Disease Modeling
Appendix II: Bibliography of Selected Model
Publications Reviewed
Kostova, Deliana, et al. “Influenza Illness and Hospitalizations Averted by
Influenza Vaccination in the United States, 2005–2011.” PLoS ONE, vol.
8, no. 6 (June 19, 2013).
Lafond, Kathryn E., et al. “Global Role and Burden of Influenza in
Pediatric Respiratory Hospitalizations, 1982–2012: A Systematic
Analysis.” PLoS Medicine, vol. 13, no. 3 (March 24, 2016).
Meltzer, Martin I., Nancy J. Cox, Keiji Fukuda. “The Economic Impact of
Pandemic Influenza in the United States: Priorities for Intervention.”
Emerging Infectious Diseases, vol. 5, no. 5 (September-October 1999):
pp. 659-671.
Meltzer, Martin I., et al. “Estimates of the Demand for Mechanical
Ventilation in the United States during an Influenza Pandemic.” Clinical
Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S52-S57.
O’Hagan, Justin J., et al. “Estimating the United States Demand for
Influenza Antivirals and the Effect on Severe Influenza Disease during a
Potential Pandemic.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015):
pp. S30-S41.
Presanis, Anne M., et al. “The Severity of Pandemic H1N1 Influenza in
the United States, from April to July 2009: A Bayesian Analysis.” PLoS
Medicine, vol. 6, no. 12 (December 8, 2009).
Reed, Carrie, et al. “Estimates of the Prevalence of Pandemic (H1N1)
2009, United States, April-July 2009.” Emerging Infectious Diseases, vol.
15, no. 12 (December 2009): pp. 2004-2007.
Reed, Carrie, Martin I. Meltzer, Lyn Finelli, Anthony Fiore. “Public Health
Impact of Including Two Lineages of Influenza B in a Quadrivalent
Seasonal Influenza Vaccine.” Vaccine, vol. 30 (2012): pp. 1993-1998.
Reed, Carrie, et al. “Estimating Influenza Disease Burden from
Population-Based Surveillance Data in the United States.” PLoS ONE,
vol. 10, no. 3 (March 4, 2015).
Rolfes, Melissa A., et al. “Annual Estimates of the Burden of Seasonal
Influenza in the United States: A Tool for Strengthening Influenza
Surveillance and Preparedness.” Influenza and Other Respiratory
Viruses, vol. 12 (2018): pp. 132-137.
Page 65 GAO-20-372 Infectious Disease Modeling
Appendix II: Bibliography of Selected Model
Publications Reviewed
Russell, K., et al. “Utility of State-Level Influenza Disease Burden and
Severity Estimates to Investigate an Apparent Increase in Reported
Severe Cases of Influenza A(H1N1) pdm09 – Arizona, 2015–2016.”
Epidemiology and Infection, vol. 146 (June 14, 2018): pp. 1359-1365.
Shrestha, Sundar S., et al. “Estimating the Burden of 2009 Pandemic
Influenza A (H1N1) in the United States (April 2009–April 2010).” Clinical
Infectious Diseases, vol. 52, suppl. 1 (2011): pp. S75-S82.
Tokars, Jerome I., Melissa A. Rolfes, Ivo M. Foppa, Carrie Reed. “An
Evaluation and Update of Methods for Estimating the Number of Influenza
Cases Averted by Vaccination in the United States.” Vaccine, vol. 36
(2018): pp. 7331-7337.
Page 66 GAO-20-372 Infectious Disease Modeling
Appendix III: Ten Selected Infectious Appendix III: Ten Selected Infectious Disease
Models and Questions from Data Collection
Disease Models and Questions from Data
Instrument
Collection Instrument
Table 5: Documents Describing Models Evaluated in our Data Collection Instrument by Agency and Disease
Agency and disease Document describing model
Centers for Disease Control and Meltzer, Martin I., Charisma Y. Atkins, Scott Santibanez, Barbara Knust, Brett W. Petersen,
Prevention (CDC) Ebola Models Elizabeth D. Ervin, Stuart T. Nichol, Inger K. Damon, Michael L. Washington. Estimating the
Future Number of Cases in the Ebola Epidemic–Liberia and Sierra Leone, 2014-2015, MMWR.
Volume 63, Number 3, September 26, 2014.
Rainisch, Gabriel, Manjunath Shankar, Michael Wellman, Toby Merlin, and Martin I. Meltzer.
Regional Spread of Ebola Virus, West Africa, 2014. Emerging Infectious Diseases. Volume 21,
Number 3, March 2015.
Office of the Assistant Secretary for Asher, Jason. Forecasting Ebola with a Regression Transmission Model. Epidemics. Volume
Preparedness and Response 22, 2018.
(ASPR) Ebola Model
CDC Zika Models Ellington, Sascha R., Owen Devine, Jeanne Bertolli, Alma Martinez Quiñones, Carrie K.
Shapiro-Mendoza, Janice Perez-Padilla, Brenda Rivera-Garcia, Regina M. Simeone, Denise J.
Jamieson, Miguel Valencia-Prado, Suzanne M. Gilboa, Margaret A. Honein, Michael A.
Johansson. Estimating the Number of Pregnant Women Infected With Zika Virus and Expected
Infants With Microcephaly Following the Zika Virus Outbreak in Puerto Rico, 2016. JAMA
Pediatrics. Volume 170, Number 10, October 2016.
Johansson, Michael A., Luis Mier-y‑Teran-Romero, Jennita Reefhuis, Suzanne M. Gilboa, and
Susan L. Hills. Zika and the Risk of Microcephaly. New England Journal of Medicine. Volume
375, Number 1, July 7, 2016.
ASPR Zika Model Dirlikov, Emilio, Krista Kniss, Chelsea Major, Dana Thomas, Cesar A. Virgen, Marrielle
Mayshack, Jason Asher, Luis Mier-y-Teran-Romero, Jorge L. Salinas, Daniel M. Pastula, Tyler
M. Sharp, James Sejvar, Michael A. Johansson, Brenda Rivera-Garcia. Guillain-Barré
Syndrome and Healthcare Needs during Zika Virus Transmission, Puerto Rico, 2016. Emerging
Infectious Diseases. Volume 23, Number 1, January 2017.
CDC Pandemic Influenza Models Biggerstaff, Matthew, Carrie Reed, David L. Swerdlow, Manoj Gambhir, Samuel Graitcer, Lyn
Finelli, Rebekah H. Borse, Sonja A. Rasmussen, Martin I. Meltzer, Carolyn B. Bridges.
Estimating the Potential Effects of a Vaccine Program against an Emerging Influenza
Pandemic—United States, Clinical Infectious Diseases. Volume 60, Issue Supplement 1, 2015.
Carias, Cristina, Gabriel Rainisch, Manjunath Shankar, Bishwa B. Adhikari, David L. Swerdlow,
William A. Bower, Satish K. Pillai, Martin I. Meltzer, Lisa M. Koonin. Potential Demand for
Respirators and Surgical Masks during a Hypothetical Influenza Pandemic in the United States.
Clinical Infectious Disease. Volume 60, Issue Supplement 1, 2015.
Reed, Carrie, Frederick J. Angulo, David L. Swerdlow, Marc Lipsitch, Martin I. Meltzer, Daniel
Jernigan, and Lyn Finelli. Estimates of the Prevalence of Pandemic (H1N1) 2009, United States,
April–July 2009, Emerging Infectious Diseases. Volume 15, Number 12, December 2009.
ASPR Pandemic Influenza Model Asher, Jason, Matthew Clay. Deterministic compartmental models for influenza with mitigations.
R: “flumodels” package. Version: 1.0.7, April 24, 2017.
Source: GAO analysis of agency and other identified documents | GAO-20-372
Page 67 GAO-20-372 Infectious Disease Modeling
Appendix III: Ten Selected Infectious Disease
Models and Questions from Data Collection
Instrument
Data Collection
Instrument
GAO Review of Model Purpose: The Government Accountability Office has been asked by the
Assessment Steps for Congress to review the Department of Health and Human Services’
agency efforts to model infectious disease. As part of our methodology,
Selected Agency Models
we selected and reviewed published papers and internal memoranda
from the sources provided to us. We reviewed these sources to describe
the steps taken to describe, verify, validate, and communicate results of
these modeling efforts. The purpose of this inquiry is to provide the
authors of the selected papers the opportunity to confirm, clarify, or
provide additional information in the table below.
Instructions: In the table below, we have two sets of columns: one set
indicating GAO’s assessment of whether the document contained
information about a step being taken. The second set of columns is for
the authors of the selected paper to fill out. If you agree with information
in the GAO columns, please indicate your concurrence in the Reviewer
Comments column. Otherwise, please provide information accordingly.
If a step is marked “Step taken” please review the entries we have made
in the GAO Reviewer Comments column for accuracy and completeness
and indicate your concurrence in the Reviewer Comments column.
Please also provide additional supporting documentation if available. For
any steps that were taken, but where we indicated either “not taken” or
“not enough information to determine” in our review, please provide a
description of the actual steps and any documentation you may have. If a
step was not taken, please provide an indication as to why that step was
not taken and, if possible, please provide supporting documentation. For
example, if limited data availability impacted the ability to conduct a model
validation step(s), then please include this information in the appropriate
table cells.
In the table below, we reviewed the following: [first author, paper name,
year] Please indicate the names of staff who completed this form:
Page 68 GAO-20-372 Infectious Disease Modeling
Appendix III: Ten Selected Infectious Disease
Models and Questions from Data Collection
Instrument
Assessment Element
Clarify Objectives 1 Communication between decision maker or model requestor and
model developer to establish clear understanding of model
question(s), limitations, etc.
2 Question the model is designed to answer reflects the question posed
Model Description 3 Model assumptions are stated, for example:
• Population characteristics
• Transmission factors/equations
• Statistical/distributional (frequentist/Bayesian)
• Other
4 Model limitations are stated
5 Model inputs are defined (conceptual and operational definitions of
variables)
6 Model outputs are described
7 Type of model used is stated (phenomenological, mechanistic,
regression, simulation, deterministic SIR, etc.)
8 Underlying equation(s) or algorithms are included
9 Software/programming language used is stated
Model Verification (Internal 10 Independent expert (internal or external) review of key programming
Validation, Internal choices and approaches
Consistency, Technical 11 Debugging tests and checks for coding accuracy
Validity) 12 Model’s code or Excel spreadsheet is available
13 Test model assumptions (i.e. confirming model assumptions are
reasonable and appropriate for question), for example:
• Distributional assumptions about model residuals
• Form of the model
14 Model handling of input data/parameters is verified as correct (i.e. as
intended by developers)
15 Other
Model Validation
Page 69 GAO-20-372 Infectious Disease Modeling
Appendix III: Ten Selected Infectious Disease
Models and Questions from Data Collection
Instrument
16 Sensitivity analysis (assessing impact of assumption/parameter
uncertainty on output or model form)
17 Cross validation or between model comparisons: Compare results to
other models that address the same problem
18 External validation: Compare model results to actual event data
19 Predictive validation: Compare model predictions for future events to
actual outcomes.
20 Other
Communication 1 21 Modelers supply customer with agreed upon information, which may
vary depending on the model
22 Modeler provides customer with clear information on uncertainty in
model results, such as inclusion of standard errors or confidence
intervals, or qualitative explanations of uncertainty in the model results
Assessment Steps Question: Do you think that the assessment
elements identified in the table above sufficiently reflect the steps that
should generally be taken to develop and assess the performance of
models? Would you remove any steps, add any steps, or make any other
adjustments to these steps in order to consider them best practices in
assessing performance of models, generally? Please explain.
1Our discussion on developing and assessing model performance combines clarifying
objectives and communicating results. We provide a further discussion of agency officials’
answers to questions about communication, as it relates to supplying customers with a
model’s results in our discussion of communication challenges.
Page 70 GAO-20-372 Infectious Disease Modeling
Appendix IV: Comments from the
Appendix IV: Comments from the Department
of Health and Human Services
Department of Health and Human Services
Page 71 GAO-20-372 Infectious Disease Modeling
Appendix IV: Comments from the Department
of Health and Human Services
Page 72 GAO-20-372 Infectious Disease Modeling
Appendix V: GAO Contacts and Staff
Appendix V: GAO Contacts and Staff
Acknowledgments
Acknowledgments
Timothy M. Persons at 202-512-6888 or personst@gao.gov
GAO Contacts
In addition to the contact named above, the following individuals made
Staff contributions to this report: Sushil Sharma (Assistant Director), Charlotte
Acknowledgments E. Hinkle (Analyst-in-Charge), Sam Amrhein, Breanne Cave, Jehan
Chase, Carol A. Gotway Crawford, Justin Cubilo, Karen Doran, Nancy
Fasciano, Douglas G. Hunker, Dennis Mayo, Anika McMillon, Sarah
Resavy, Edward Rice, Ben Shouse, Amber Sinclair, Walter Vance, Sarah
Veale, and Richard Zarrella.
(102801)
Page 73 GAO-20-372 Infectious Disease Modeling
The Government Accountability Office, the audit, evaluation, and investigative
GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of the
federal government for the American people. GAO examines the use of public
funds; evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO’s commitment to good government
is reflected in its core values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly
GAO Reports and released reports, testimony, and correspondence. You can also subscribe to
GAO’s email updates to receive notification of newly posted products.
Testimony
Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and
distribution and depends on the number of pages in the publication and whether
the publication is printed in color or black and white. Pricing and ordering
information is posted on GAO’s website, https://www.gao.gov/ordering.htm.
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card, MasterCard,
Visa, check, or money order. Call for additional information.
Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts.
Visit GAO on the web at https://www.gao.gov.
Contact FraudNet:
To Report Fraud,
Website: https://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in
Automated answering system: (800) 424-5454 or (202) 512-7700
Federal Programs
Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400,
Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125,
Relations Washington, DC 20548
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548
James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707
Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814,
External Liaison Washington, DC 20548
Please Print on Recycled Paper.
File and source
- File
- gao-20-372.pdf
- Size
- 3,928,028 bytes
- SHA-256
- ff941df697280708daf1f49d669b4a361912955e64482bbb038ee02d019c0bec
- Our copy
- gao-20-372.pdf
- Original
- www.gao.gov