Census Bureau Needs to Ensure Transparency over Data Quality (GAO-21-262T)

Summary

GAO testimony, GAO-21-262T, before the House Committee on Oversight and Reform on December 3, 2020, the statement of J. Christopher Mihm, Managing Director, Strategic Issues. It discusses the U.S. Census Bureau's progress toward delivering apportionment counts for the 2020 Decennial Census. It states that in response to the COVID-19 pandemic and related executive branch decisions, the Bureau made a series of late changes to the census design that affected how and how long it did its work. GAO says the changes introduced risks to the quality of data the Bureau provides for congressional apportionment and redistricting, and that transparency over data quality will be important.

Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used

Full text

United States Government Accountability Office
Testimony
Before the Committee on Oversight and
Reform, House of Representatives

2020 CENSUS
For Release on Delivery
Expected at 10 a.m. ET
Thursday, December 3, 2020

Census Bureau Needs to
Ensure Transparency over
Data Quality
Statement of J. Christopher Mihm, Managing Director,
Strategic Issues

GAO-21-262T
Letter Letter

December 3, 2020

Chairwoman Maloney, Ranking Member Comer, and Members of the
Committee:

I am pleased to be here today to discuss the U.S. Census Bureau’s
(Bureau) progress to deliver apportionment counts for the 2020 Decennial
Census. As you know, the census was taken under extraordinary
circumstances. In response to the Coronavirus Disease 2019 (COVID-19)
pandemic and related executive branch decisions, the Bureau made a
series of late changes to the design of the census.

These changes affected the way the Bureau did its work as well as the
time it took to do the work. The changes also introduced risks to the
quality of data that the Bureau provides for congressional apportionment
and redistricting purposes. 1 My bottom line today is that it will be
important—both for transparency and ensuring public confidence in the
quality of the 2020 Census—that the Bureau share key quality indicators
in near real time as it releases apportionment and redistricting data.

Today, we are issuing the first in a series of planned reports that will
assess the operations of the 2020 Census and identify lessons learned as
planning begins for 2030. My statement today is based on that report
entitled, 2020 Census: Census Bureau Needs to Assess Data Quality
Concerns Stemming from Recent Design Changes. 2 The report describes
the key changes that the Bureau made in response to the COVID-19
outbreak and how those changes affect the quality of the census. For that
work, we reviewed Bureau decision memos and planning documents. We
also reviewed Bureau announcements related to the COVID-19 response
and any characterizations of the resulting operational changes.

We interviewed Bureau officials to better understand downstream effects
of operational changes made during the COVID-19 response, as well as
to learn about any actions the Bureau is taking to monitor cost and quality
effects of the changes. More detailed information on our objectives,
scope, and methodology can be found in the issued report. Our work was

1Census data are used, among other purposes, to apportion the seats of the U.S. House
of Representatives and redraw congressional districts in each state.
2GAO, 2020 Census: Census Bureau Needs to Assess Data Quality Concerns Stemming
from Recent Design Changes, GAO-21-142 (Washington, D.C.: Dec. 3, 2020).

Page 1 GAO-21-262T
conducted in accordance with generally accepted government auditing
standards.

As figure 1 shows, the changes to the Bureau’s operations took place
Bureau Needs to amid escalating COVID-19 case counts and state-level policy responses.
Examine Effects on
Data Quality of Its
Design Changes in
Response to COVID-
19 and Related
Executive Branch
Decisions

Page 2 GAO-21-262T
Figure 1: The Census Bureau Adjusted Its Operational Timeline during the COVID-19 Pandemic

Initial suspensions. As part of its initial COVID-19 response and to
promote the safety of its workforce and the public, the Bureau delayed its
field operations multiple times. On March 18 and 28, 2020, the Bureau

Page 3 GAO-21-262T
announced it would suspend operations for 2 weeks, respectively. In
recognition of the pandemic’s community spread, the Bureau on April 13,
2020, announced that it would extend the suspension of nonresponse
follow-up (NRFU)—an operation where enumerators collect responses
from households that had not yet self-responded—to a total of 3 months.

Plan for delayed delivery of population counts. Also on April 13, 2020,
the Bureau and the Department of Commerce announced plans to deliver
population counts for apportionment purposes by April 30, 2021, and for
redistricting purposes by July 31, 2021—4 months later than the
respective statutory deadlines. According to the announcement, the
Department of Commerce had requested that Congress grant a 120-day
extension to the Bureau’s reporting deadlines. The Bureau also
announced an extension to the self-response period and planned to
conduct the NRFU operation through October 31, 2020.

Gradual resumption of operations. The Bureau first resumed a revised
Update Leave operation—where temporary staff verify addresses and
deliver invitations to respond to the census—on May 4, 2020, and
announced a phased approach to starting NRFU. 3 Between July 16 and
August 9, 2020, the Bureau deployed three waves of a “soft launch” of
NRFU by area census offices (ACO) in areas that the Bureau deemed
sufficiently safe. 4

Time frames revised again to meet original statutory deadlines. On
August 3, 2020, the Bureau announced that it would accelerate its
operational time frames, as directed by the Secretary of Commerce, to
deliver population counts for apportionment and redistricting purposes by
the statutory deadlines—4 months earlier than the plan the Bureau was
implementing. To meet the revised timeline, the Bureau announced that
NRFU and the self-response period would now end on September 30,
2020—1 month earlier than previously announced. Bureau officials stated
that streamlined post-data collection processing would account for the
remainder of the necessary timeline compression.

3During the resumption of operations, the Bureau instructed enumerators to no longer
interview respondents about their address information when delivering questionnaires
during Update Leave in order to minimize possible in-person contacts.
4The Bureau implemented field operations through a nationwide network of 248 ACOs,
which oversee local recruiting and operations and are organizational subunits of offices in
the Bureau’s six regions.

Page 4 GAO-21-262T
The Bureau also announced a series of operational changes to meet the
new timeframes. During the final 2 planned months of NRFU,
enumerators received awards for working more hours in the field and
were also given the option in most places of making phone calls to collect
data instead of making personal visits. Additionally, the Bureau
implemented a revised NRFU contact strategy to make fewer-than-
originally-anticipated visits to households self-reported as vacant on
Census Day.

The Bureau also temporarily reduced the scope of its NRFU re-interview
quality control operation and compressed the time that internal Bureau
subject matter experts have to review apportionment population counts.

In September 2020, however, the U.S. District Court for the Northern
District of California issued a preliminary injunction, staying the August
2020 design changes, enjoining the Bureau’s adherence to the statutory
deadlines, and requiring the Bureau to continue data collection through
the end of October. 5 On October 13, 2020, the Supreme Court granted a
stay allowing the Bureau to cease data collection, while the appeal of the
preliminary injunction continues to be reviewed by the United States
Court of Appeals for the 9th Circuit. 6 On October 15, 2020, the Census
Bureau ended all data collection operations.

Processing census data. On November 20, 2020, senior Bureau
officials told us that they are running into what the Bureau is referring to
as anomalies as they process responses for the apportionment count. For
example, as they integrate data in group quarter counts they are finding
mismatches and duplicates. In addition, as they change the year for the
date of birth to align with a person’s age they have found that the system
is sometimes aging people incorrectly. According to Bureau officials,
processing anomalies are not unexpected, in that they occur with each
census, and time is typically built into the schedule to identify and address
them.

5National Urban League v. Ross, Case No. 20-cv-05799 (N.D. Cal. Sept. 24, 2020) (order
granting motion for stay and preliminary injunction). The Department of Justice, on behalf
of the Department of Commerce and the Bureau, appealed the order. National Urban
League v. Ross, Case No. 20-16868 (9th Cir., Sept. 25, 2020). When the 9th Circuit denied
a stay of part of the preliminary injunction, the ruling was appealed to the U.S. Supreme
Court.
6Ross v. National Urban League, Case No. 20A62, 592 U.S. ___ (2020).

Page 5 GAO-21-262T
However, due to the compressed processing schedule for 2020, Bureau
officials are unable to provide a firm date for when the apportionment
counts will be delivered to the President, reiterating that the Bureau’s plan
is to provide the counts as close to December 31, 2020 as possible. We
have requested, but have not yet received, additional information and
documentation from the Bureau on the anomalies, and will continue to
monitor this situation and keep the Congress informed of what we learn.

Initial Indicators of Census The report we are releasing today discusses how late changes to the
Quality census design could affect data quality. The Bureau has numerous
planned assessments and evaluations of operations which, in conjunction
with its post-enumeration survey (PES)—a survey conducted
independently of each census to determine how many people were
missed or counted more than once—help determine the overall quality of
the census and document lessons for future censuses.

The Bureau has multiple mechanisms to assess the quality of the 2020
Census. For example, through its Data Quality Executive Governance
Group, the Bureau oversees several teams of officials dedicated to
studying the effects on quality of the initial COVID-19 response, each with
a set of objectives or near and long-term deliverables that they are
planning in order to facilitate retrospective determinations of data quality.
Addressing the concerns we highlight in our report as part of updating the
Bureau’s overall assessment of the quality of the 2020 Census will help to
ensure public confidence and help the Bureau begin to plan for 2030.

We recommend in our report that the Secretary of Commerce and the
Director of the U.S. Census Bureau update and implement assessments,
evaluations, and coverage measurement efforts to address the effects of
the Bureau’s response to COVID-19 that we identified, including data
quality concerns and potential operational benefits from innovations. The
Department of Commerce agreed with that recommendation.

The American Statistical Association and the Census Scientific Advisory
Committee (CSAC) issued numerous recommendations in the fall of
2020, including for the Bureau to document what it knows in near real-
time about the quality of the population counts that it provides to the

Page 6 GAO-21-262T
President and to Congress. 7 The Bureau is also working with a group of
independent scientific consultants known as the JASON group to assess
Bureau plans, processes, procedures, and metrics around 2020 Census
data quality and help develop a near-term strategy for sharing indicators
of the quality of census data that are produced for apportionment and
redistricting purposes, according to Bureau draft documentation.

Transparency of 2020 Consistent with our report and recommendation and the work of both the
Census Quality Indicators ASA and CSAC, my statement today highlights some of the census
quality indicators that the Bureau should consider providing when it
Can Provide the Public
releases the apportionment counts. Transparency over what is known
with Confidence in the and not yet known about census quality would help to increase public and
Apportionment Counts congressional confidence in the quality and completeness of the census
despite all of the challenges the Bureau faced.

While the Bureau believes that self-response from households provides
the most accurate data, in an attempt to count every person, the Bureau
uses alternative data collection methods when it is unable to collect
census data directly from the household. Looking at the rates at which the
Bureau uses each of these alternative methods of data collection may
provide some insight into the quality of the data collected as part of the
2020 Census. 8 Nationwide rates provide a high-level indication of overall
census quality. However, and very importantly, the Bureau also would
need to examine the rates at which the Bureau uses these alternative
methods at smaller levels of geography and by demographic group to
provide a complete picture.

According to Bureau officials it has not provided data on use of these
indicators at smaller levels of geography or by demographic group in
previous censuses. Rather, since 1980, the Bureau has used its PES to
produce measures of coverage errors for geographic areas and
demographic groups. Bureau officials told us they understand the
importance of providing this information below the national level to the

7American Statistical Association 2020 Census Quality Indicators Task Force, 2020
Census Quality Indicators, (Alexandria, VA.: October 2020). The Census Scientific
Advisory Committee is an advisory body to the Director of the Census Bureau and
composes recommendations on major programs, such as the decennial census. The
members advise the Bureau on the uses of scientific developments in, among other
things, statistical data collection and statistical analysis, as they pertain to the full range of
Bureau programs and activities.
8In addition to the information presented here on the use of administrative records and
proxies in closing NRFU cases, the Bureau will likely provide additional information on the
use of these methods in the Operational Assessments produced after the 2020 Census.

Page 7 GAO-21-262T
public in as near real time as possible this decennial census and are
working to provide those data. Bureau officials cautioned that variations
will exist at the more granular level and they will not be able to fully
understand those variations and would need to further study those
variations before drawing any conclusions.

• Proxy responses for occupied, vacant, and nonexistent housing
units: The Bureau uses proxy responses—information from a
neighbor or other knowledgeable person about a household—to
collect data when a resident of the household is not available or
cannot be found during the NRFU operation. Proxy responses are
also allowable after one unsuccessful contact attempt to an address
considered vacant or nonexistent and after three unsuccessful contact
attempts for addresses considered occupied. For the 2020 Census,
preliminary results indicate that, nationally, the Bureau enumerated
approximately 7.4 million occupied households using proxies, or 24.1
percent of occupied households in the NRFU workload, and identified
approximately 9.9 million housing units as vacant and 3.4 million
housing units as nonexistent.
By comparison, for the 2010 Census, the Bureau reported that
nationally, a proxy was the respondent for approximately 23.8 percent
of occupied households (6.8 million households), approximately 97.5
percent of vacant housing units (13.8 million housing units), and
approximately 90 percent of nonexistent housing units (3.6 million
housing units) in the NRFU workload.
• Population-count only responses: The Bureau receives incomplete
NRFU interviews in cases where an enumerator is unable to complete
the entire census questionnaire for a household with either a
household member or a proxy. Enumerators are directed to try to
obtain, at a minimum, an indication of the household status (vacant,
not a household, or occupied) and, if occupied, the number of the
people in the housing unit by talking to a proxy. These are referred to
as population-count only responses. The total number of population-
count only responses can be an indicator of the quality of data
collected. According to Bureau officials, the number of population-
count only responses for the 2020 Census is not yet available.
Moreover, according to Bureau officials in 2010, the number of
households enumerated with population-count only responses was
not reported.
• Administrative records responses: The Bureau incorporated
increased use of administrative records into the design of the 2020
Census as a major cost saving innovation and to improve the overall

Page 8 GAO-21-262T
quality of the data. Administrative record data refer to information from
federal and state governments and third-party vendors that people
have already provided, such as, information from the United States
Postal Service, the Internal Revenue Service, the Social Security
Administration, the 2010 Census, and the American Community
Survey.
These data are used to identify vacant and non-housing units and to
enumerate occupied households if, after one visit, a NRFU contact
attempt was not successful. Specifically, for some households, after
one visit, the Bureau used administrative records it determined in
advance of NRFU met a certain quality threshold. According to
preliminary Bureau data, after a single NRFU attempt, the Bureau was
able to resolve 13 percent of NRFU cases (8 million cases) that were
either occupied, vacant, or not a housing unit using administrative
records.
The Bureau also used administrative records to resolve an additional
approximately 450,000 cases (0.73 percent of the NRFU workload,
according to preliminary data). Specifically, in the final phases of the
NRFU operation, households that received the maximum number of
contact attempts (typically six) were reopened. For these households,
if the Bureau was still unable to conduct an interview, then the Bureau
used administrative records that met the Bureau’s quality standards.
According to Bureau planning documents, while these administrative
records used at the end of NFRU do not meet the threshold for early
removal after the first visit, they do provide adequate data to support
apportionment.
The Bureau’s original plan had been to use an administrative record in
the final phase of NRFU to determine the population of a household
only when corroborated by a second administrative record. However,
as part of its plans to meet the statutory date for delivering
apportionment data on December 31, the Bureau decided, after
NRFU had already begun, to enumerate some cases using a single
source of administrative records when available.
This decision to use a single source of administrative records
introduced a data quality risk, since there is no corroborating source.
Bureau officials said that they will be examining the effects on quality
of the late design changes to the use of administrative records.
However, the Bureau believes that the risk to overall data quality
should be mitigated by the fact that the Bureau only used such single
source records for households that lack a population count during the
final stage of NRFU, after most households have already been
counted. The Bureau is still working on an estimate on the extent of

Page 9 GAO-21-262T
use of single source records. In total, according to preliminary data,
the Bureau used administrative records to close NRFU cases for 8.4
million households (approximately 14 percent of the NRFU workload),
or approximately 5.6 percent of households nationwide.
• Count imputation: To resolve missing household responses
following data collection, as it did in 2010, the Bureau is using for
2020 a technique it refers to as count imputation, which draws data
from similar nearby households to determine whether a housing unit
exists, whether it is occupied, and, if so, by how many people. The
Bureau has used some form of imputation since the 1940 Census,
and reported on the use of imputation in its operational assessment
reports for the 2010 Census.
According to Bureau reporting, in 2010, about 500,000 of 137 million
addresses counted in the decennial (0.4 percent) were missing an
entire response and the Bureau therefore used count imputation to
determine a combination of their residence and occupancy status and
household size. 9 The Bureau’s count imputation in 2010 added about
1.2 million people to the final census count. According to Bureau
officials, data on count imputation for the 2020 Census is not yet
available.

As the 2020 Census continues, we are monitoring the Bureau’s response
processing operations. Our work will continue to examine the quality and
cost implications of the Bureau’s COVID-19 response and late design
changes. As we begin to look forward to the initial planning for the 2030
Census, we will also report on the Bureau’s experience with its
innovations to the 2020 Census and any lessons learned that can inform
2030 planning.

Chairwoman Maloney, Ranking Member Comer, and Members of the
Committee, this completes my prepared statement. I would be pleased to
respond to any questions that you may have.

Lisa Pearson, (Assistant Director), Timothy Wexler (Analyst-in-Charge),
Staff Devin Braun, Mark Abraham, Ann L. Czapiewski, Alan Daigle, Alexandra
Acknowledgments Edwards, Rob Gebhart, Lisa Hardman, Ceara Lance, Hannah Laufe,
Jason T. Lee, Melissa Melvin, Kerstin Meyer, Ty Mitchell, Kate Sharkey,
Andrea Starosciak, Farrah Stone, Jon Ticehurst, Peter Verchinski, and

9The Bureau imputed whether an address was livable for 0.12 percent of addresses,
whether it was occupied or vacant for 0.03 percent, and its household size for 0.24
percent.

Page 10 GAO-21-262T
Shawn Ward made significant contributions to this statement and the
underlying report on which this statement is based.

(104645)
Page 11 GAO-21-262T
This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.
The Government Accountability Office, the audit, evaluation, and investigative
GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of the
federal government for the American people. GAO examines the use of public
funds; evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO’s commitment to good government
is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly
GAO Reports and released reports, testimony, and correspondence. You can also subscribe to
GAO’s email updates to receive notification of newly posted products.
Testimony
Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and
distribution and depends on the number of pages in the publication and whether
the publication is printed in color or black and white. Pricing and ordering
information is posted on GAO’s website, https://www.gao.gov/ordering.htm.
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card, MasterCard,
Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts.
Visit GAO on the web at https://www.gao.gov.

Contact FraudNet:
To Report Fraud,
Website: https://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in
Automated answering system: (800) 424-5454 or (202) 512-7700
Federal Programs
Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400,
Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125,
Relations Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548

Stephen J. Sanford, Acting Managing Director, spel@gao.gov, (202) 512-4707
Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814,
External Liaison Washington, DC 20548

Please Print on Recycled Paper.

File and source

File: gao-21-262t.pdf
Size: 808,275 bytes
SHA-256: f83ed2447b025e3c663a62300c2135ccbdfaf08c877123b93d0ceef85b28700d
Our copy: gao-21-262t.pdf
Original: www.gao.gov