Eggleston and Lieberman, Nonresponse and Coverage Bias in the Household Pulse Survey: Evidence from Administrative Data (2024-10)
Summary
CES working paper 24-60, dated October 2024, by Jonathan Eggleston and Carl Lieberman of the U.S. "Census Bureau", on nonresponse and coverage bias in the Household Pulse Survey (HPS). The authors match administrative data from government agencies and third-party data to HPS respondents and compare them with a benchmark of American Community Survey (ACS) respondents and nonrespondents. The paper states that the HPS has a single-digit unit response rate, with rates near the end of 2023 between 4 and 6 percent. It finds that the HPS is less representative of the U.S. population than the ACS, while the existing weighting adjustments appear to greatly improve the representativeness of the HPS. It finds no clear change in representativeness after email messages were limited in 2023 to addresses with at least 90% deliverability rates.
Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used
Cite this document
“Eggleston and Lieberman, Nonresponse and Coverage Bias in the Household Pulse Survey: Evidence from Administrative Data (2024-10).”
Original at www2.census.gov. Archived at Pandemic Darlings, https://pandemicdarlings.com/source-documents/eggleston-and-lieberman-nonresponse-and-coverage-bias-in-the-household-a425e977ad/
, SHA-256 31394d246256b795bc90f406f43963eb33b0486e28acdbe8307aca6e21225f46.Full text
Nonresponse and Coverage Bias in the Household Pulse Survey: Evidence from
Administrative Data
by
Jonathan Eggleston
U.S. Census Bureau
Carl Lieberman
U.S. Census Bureau
CES 24-60 October 2024
The research program of the Center for Economic Studies (CES) produces a wide range of
economic analyses to improve the statistical programs of the U.S. Census Bureau. Many of these
analyses take the form of CES research papers. The papers have not undergone the review
accorded Census Bureau publications and no endorsement should be inferred. Any opinions and
conclusions expressed herein are those of the author(s) and do not represent the views of the
U.S. Census Bureau. All results have been reviewed to ensure that no confidential information is
disclosed. Republication in whole or part must be cleared with the authors.
Papers in the series are posted on www.census.gov/library/working-papers/series/ces-wp.html.
For information about the series, contact Sean Wang, Editor, Discussion Papers, U.S. Census
Bureau, Center for Economic Studies, 4600 Silver Hill Road, Washington, DC 20233,
CES.Working.Papers@census.gov.
Abstract
The Household Pulse Survey (HPS) conducted by the U.S. Census Bureau is a unique survey that
provided timely data on the effects of the COVID-19 Pandemic on American households and
continues to provide data on other emergent social and economic issues. Because the survey has
a response rate in the single digits and only has an online response mode, there are concerns
about nonresponse and coverage bias. In this paper, we match administrative data from
government agencies and third-party data to HPS respondents to examine how representative
they are of the U.S. population. For comparison, we create a benchmark of American Community
Survey (ACS) respondents and nonrespondents and include the ACS respondents as another point
of reference. Overall, we find that the HPS is less representative of the U.S. population than the
ACS. However, performance varies across administrative variables, and the existing weighting
adjustments appear to greatly improve the representativeness of the HPS. Additionally, we look
at household characteristics by their email domain to examine the effects on coverage from
limiting email messages in 2023 to addresses from the contact frame with at least 90%
deliverability rates, finding no clear change in the representativeness of the HPS afterwards.
1
0F
*
Any opinions and conclusions expressed herein are those of the author and do not represent the views of the U.S.
Census Bureau. The Census Bureau has ensured appropriate access and use of confidential data and has reviewed
these results for disclosure avoidance protection (Project 7521191: CBDRB-FY24-CES005-007, CBDRB-FY24-CES005-
009, and CBDRB-FY24-CES005-014).
1. Introduction
The Census Bureau’s Household Pulse Survey (HPS) is an agile survey product that can quickly and
efficiently provide data on key social and economic measures. These data can then be disseminated
quickly to inform government action. HPS survey topics have included COVID-19 diagnosis and
vaccination, access to infant formula, food sufficiency, mental health, and the impact of living through
natural disasters.
The HPS has a single-digit unit response rate, with rates near the end of 2023 between 4 and 6 percent.
Unlike other Census Bureau surveys, the HPS contacts respondents through email and text messages,
and the questionnaire is only administered via web. Some research has expressed concern about the
representativeness of the HPS given these limitations. Bradley et al. (2021), for example, find that
limitations to the representativeness of the HPS result in overestimation of COVID-19 vaccination rates.
A Census Bureau report finds varying levels of nonresponse bias over time, documenting different
response rates across demographic groups, such as higher response propensities among college-
educated individuals and people in high-income areas 2 (Peterson et al., 2021).
1F
In this article, we use administrative and third-party data to estimate the representativeness of the
Household Pulse sample. We then merge the same administrative data to American Community Survey
(ACS) respondents and nonrespondents as a benchmark to evaluate the representativeness of the HPS
and compare it to ACS.
We use the term “representativeness” to describe the cumulative effect of various errors on the overall
deviation of the HPS respondent sample from the target population, which we generate using
administrative and third-party records. Groves and Lyberg (2010) list three sources of error that affect
the representativeness of the final respondent sample: coverage error, sampling error, and nonresponse
error. The HPS is at higher risk for nonresponse error given its low response rates. Additionally, the HPS
is also at risk for greater coverage error because only people with internet access can participate and
because the Census Bureau’s list of people’s email addresses and phone numbers may be incomplete.
Although sampling error can affect the precision of the estimates, it should not create bias. Therefore,
when we talk about the representativeness of the HPS sample, we are discussing the combined effect of
nonresponse and coverage errors on the accuracy of estimates. Due to the nature of the administrative
2
Survey respondents tending to have higher incomes has also been found in the American Community Survey
(Rothbaum et al., 2021) and Current Population Survey (Eggleston et al., 2024).
2
data to which we have access, our representativeness evaluations primarily use demographic and
economic characteristics. Given nonresponse bias is often variable-specific rather than survey-specific
(Groves and Peytcheva 2008), it is possible that the HPS could have larger bias in areas that are not
documented by the administrative records available at the Census Bureau, such as mental health or
vaccination status.
Based on comparisons of means of matched administrative records with a benchmark population,
although the HPS is generally not as representative or stable as the ACS, its performance is different
across variables and tends to be improved by the weighting process. After applying these weights, it is
often comparable in representativeness to the ACS, occasionally even outperforming it. For example,
the base HPS has the highest quartile of households by income overrepresented (30.94% when they
should, mechanically, be 25.00% of the sample, a 5.94 percentage point bias), but after weighting, the
bias is reduced to -0.24 percentage points, compared to 0.51 percentage points for the ACS. This
representativeness also often increases over time, with some variables showing noticeable
improvements to the HPS’ coverage bias as it incorporated updates to the Contact Frame from the 2020
Census. While these findings vary across variables or over time, as we discuss in detail later, the HPS is
broadly representative despite its response rate and changing sampling frame. However, some caution
is warranted in these comparisons given we find discrepancies between a respondent’s sampled address
and the address they report as their current address in the survey instrument.
Finally, we conduct a coverage analysis on the email contact frame and changes to how it is used. In
September 2022 (week 49 of the survey), the HPS switched to only using email addresses from the email
contact frame with at least 90% deliverability rates for their domain due to delivery issues with some
email providers (sampling with phone numbers was unaffected). Using the same administrative data, we
compare economic and demographic characteristics of households with different types of email
addresses. Overall, we find that compared to households with an email address with at least a 90%
deliverability rate, households with other email addresses are older, less likely to have children, and less
likely to include a person of color. However, we find that there is no clear change in the
representativeness of the HPS sample after this change, because the vast majority of respondents are
contacted via text message rather than email (e.g., 84% in week 40).
3
2. Data
2.1 Household Pulse Survey
The HPS is designed to be a low-cost and high-frequency survey to enable fast, informed decision
making across the federal government. Given these goals, potential respondents need to be contacted
quickly and cost-effectively. Using the Qualtrics platform, the survey sends invitations though text
messages and email. Consequentially, only addresses with a matched cell phone number or email
address are included in the sampling frame, which excludes a segment of the population. Reeves and
Varela (2024) discuss the design of the HPS’ survey invitations and reminders; they also present results
from experiments conducted on those invitations and reminders in 2022 like changing the login URL and
timing of reminders.
The HPS uses cell phone numbers and email addresses from the Census Bureau’s Phone Contact Frame
and Email Contact Frame. At the start of the HPS, the Contact Frames consisted mainly of phone
numbers from commercial providers, with additional information from sources such as prior American
Community Survey (ACS) responses and Supplemental Nutrition Assistance Program data for certain
states. These phone numbers were of varying quality and some households did not have a valid number
matched to their address. Eggleston (2021) presents evidence that lower income households were less
likely to have a valid phone number in the contact frame, implying that missingness was non-random.
The Contact Frames improved after the 2020 Census, when it incorporated phone numbers and email
addresses from the decennial census, greatly improving coverage. The HPS started using this new
version of the Contact Frames in December 2021.
Table 1: Important Changes in the Contact Strategy and Contact Inputs for the Household Pulse Survey
Week
Number Date Description
1 2020-04-24 Survey launch. Only email addresses are used in sampling. The survey is
longitudinal over three rounds of data collection.
2 2020-05-07 Text messaging introduced. Only addresses without an email address
received a text message.
3 2020-05-14 Full implementation of text messaging.
4
12 2020-08-19 Phase 2 of the survey started. The survey is no longer longitudinal. The
length of time to respond to the HPS increased from about one week to
about two weeks.
18 2020-10-28 Phase 3 started. No significant changes.
40 2021-12-01 Started using improved Contact Frames with updates from the 2020
Census.
49 2022-09-14 Started using only email addresses from the Email Contact Frame with
at least 90% deliverability rates.
Table 1 summarizes the changes made to the HPS contact strategies over time. Of note for its potential
impact is the week 49 decision to use only email addresses from the Email Contact Frame whose
deliverability rate is at least 90%. To alleviate recurring problems of HPS invitation emails being flagged
as spam by email providers due to the large quantity of emails sent, Census Bureau management
decided in 2022 to use only email addresses with at least 90% deliverability rates. Additionally, after
week 49, the email and text notifications were sent out in smaller batches to reduce the likelihood of
being falsely flagged as spam and improve delivery rates. This has the potential to change the
representativeness of the HPS sample.
2.2. American Community Survey
A common approach to evaluating the sample representativeness of a survey is comparing respondents
and nonrespondents matched to administrative records, restricting the analysis to occupied housing
units (thus excluding vacant homes, nonexistent addresses, and addresses without housing) (Eggleston
and Westra, 2020; Rothbaum and Bee, 2020; Rothbaum et al., 2021; Eggleston et al., 2024). This
approach cannot be directly applied to the HPS because it samples only addresses with a matched
phone number or email address. Further, if there is no response from someone at an address in the HPS,
then there is no interview to confirm the address’s occupancy status.
To make our comparisons, we use as our reference a survey that has in-person interviews: the American
Community Survey (ACS). The ACS is the largest household survey in the United States, sampling
approximately 3.5 million addresses each year, divided into twelve monthly panels for continuous
interviewing. It is a multimodal survey where most households are given two months to respond by web
or mail, followed by in-person follow-up in the third month for a subsample of nonrespondents. During
2020, in-person interviewing was completely suspended at the start of the COVID-19 Pandemic. After
5
July, in-person interviewing was resumed for much of the country, but in-person interviewing remained
suspended in areas with high COVID-19 case counts. These area-specific suspensions of in-person
interviewing continued until early 2021. The ACS contains questions on a variety of social, housing, and
economic topics, and is the key source of such statistics at the county or subcounty level in the United
States. Accuracy in the ACS is paramount, as it aids in the allocation of trillions of dollars in federal
funding every year.
2.3 Administrative Data
Our primary source of administrative data is annual tax filings from the Internal Revenue Service (IRS).
First, we use IRS Form 1040 individual income returns, which contain income from the prior year, filing
status as a proxy for marital status, and the identities of the first four children or other family members
claimed as dependents. Next, we use IRS information returns, which include W-2s from employers, 1099
forms like Form 1099-INT, and Form 1098, which lists mortgage information and can be used as a proxy
for homeownership. These forms give us additional information on different types of income and other
economic measures. We use tax data from the corresponding survey year for the HPS. Our matching and
data construction procedures resemble those of Eggleston and Westra (2020) and Eggleston et al.
(2024), who examine administrative data-based weights for the Survey of Income and Program
Participation (SIPP) and Current Population Survey (CPS), respectively.
To match survey and administrative records at the address level, we utilize the linking identifier in the
Master Address File (MAF), the Census Bureau’s frame of all known living quarters and certain
nonresidential addresses in the United States. The MAF is the sampling frame for the HPS, so all HPS
observations have this linking identifier, called the MAFID. The IRS data are linked to the MAF using a
probabilistic linking algorithm. Looking at 1040 data, Bee, Gathright, and Meyer (2015) find that about
90 percent of Form 1040 records match to a MAFID, with unmatched tax records tending to be in the
extreme upper and lower ends of the income distribution. There are two reasons a household does not
match to IRS data: either the household did not file taxes or receive any other tax form (among those
provided to the Census Bureau) from an employer or other party, or a problem with the address listed
on the tax form resulted in no MAFID being assigned. Later in this paper (Table 2), we see that 87.16% of
households in our benchmark are matched to our main administrative data, compared to 89.68% of HPS
respondents and 89.11% of ACS respondents, so the two surveys are very comparable in this respect.
6
2.3.1 Construction of Household Roster
Next, we use administrative data to construct a household roster for each address to use as a basis for
matching additional person-level datasets. Our primary sources for roster construction are the IRS Form
1040 returns and the IRS information returns (e.g., Forms 1098, 1099, and W-2), which allow for
identifying individuals who do not have any filing requirements, like certain retirees who receive only
Old-Age, Survivors, and Disability Insurance (OASDI).
We further use several supplemental data sources for households we are unable to match to any IRS
data, which can be because they are truly not in the IRS data or because their IRS records cannot be
assigned a MAFID The Census Bureau’s Master Address File Auxiliary Reference File (MAFARF) contains
residency information from IRS data and other administrative data sources, such as the United States
Department of Housing and Urban Development (HUD). SSA’s Supplemental Security Record (SSR)
covers recipients of Supplemental Security Income (SSI), which is not taxable. Finally, we use parent-
child linkages from the Census Bureau Household Composition Key to add children to the household
roster who do not appear in any of our other sources. This file is created using data Census receives
from SSA on parent names (but not SSNs), confirming that the parent and children live together using
other administrative data or a decennial census (see Genadek et al., 2021 for more details).
7
2.3.2 Additional Data Linkage
Figure 1: Data-Matching Diagram
MAF
MAFID
HH Survey
MAFID
CMS PIK
SSA PHUS
PIK
Medicaid SSA SSR
State SNAP/ SSA MBR
IRS 1040 IRS 1099 PIK
TANF/WIC
PIK
W2 MAFARF PIK
SSA
EIN Numident
Business PIK
Register 2010
Administrative Data Roster: Census
Linked by PIK and MAFID
We link several additional datasets to learn more about these households once the roster has been
finalized, depicted graphically in Figure 1. We use demographic data from the Social Security
Administration’s (SSA) Numident file and the 2010 Census. If race data are available from both sources
for a person, we use the 2010 Census data. The 2010 Census also provides information on Hispanic
status, while the Numident data contain information on citizenship and foreign-born status.
These datasets are linked to the IRS-based household rosters at the individual level using the Census
Bureau’s Person Identification Validation System (PVS), described by Wagner and Layne (2014). The PVS
matches both survey and administrative data to a master reference file; individuals who are matched
are given an identifier called a Protected Identification Key (PIK), which acts as an anonymized Social
Security number that can be used to link administrative datasets and surveys. For the 2010 Census,
8
about 90 percent of individuals were matched to a PIK (Wagner and Layne, 2014). 3 Bond et al. (2014)
2F
argue that inability to match to a PIK is nonrandom, finding that young children, racial and ethnic
minorities, immigrants, recent movers, low-income individuals, and non-employed individuals in the
2009 ACS are less likely to be matched to a PIK.
We then aggregate HPS data to the household/address level to create measures for comparing
respondent and nonrespondent households. For example, we create an indicator for the presence of a
household member over age 60 by taking the list of people on tax forms matched to an address and
linking these people in the Numident file to get the year of birth for all household members.
Additionally, we match OASDI program benefit data for SSA’s Master Beneficiary Record (MBR) and
Payment History Update System (PHUS) at the person-level using PIKs. For program participation, we
also match data at the person-level. We use Medicaid data from the Centers for Medicare & Medicaid
Services for every individual in the country, as well as children receiving health insurance through the
Children's Health Insurance Program. The comingled datasets also contain information on SNAP, WIC,
and TANF receipt from participating state agencies (not all states have agreements with the Census
Bureau to provide this data). For characteristics of a person’s employer, we use the Employer
Identification Numbers (EIN) on the W-2s matched to the household to find the employer in the Census
Bureau’s Business Register. Given that the employment of people in the hospitality and food service
industry was particularly affected by COVID-19 restrictions, the industry information available in the
Business Register may be especially valuable for economic outcomes in 2020. Note that this linkage by
EIN is imperfect for many businesses, as EIN is neither a firm nor an establishment identifier for multi-
establishment firms, so the number of people who share an EIN may not always equal the number of
people that work at a place of business. However, because our goal is to construct proxies for employer
characteristics rather than to construct precise establishment-level statistics, and we do not expect the
validity of these proxies to vary by survey response, we believe the EIN linkage is adequate as an input
for our weighting algorithm.
3
The 2010 Census is also linked to the MAF. Because many people have changed their residential location since
2010, however, we link decennial census data at the person level instead, to make sure we are capturing the
characteristics of the current household.
9
3. Methods
3.1 Primary Graphical Analysis of Means
Our primary graphical analysis consists of comparing means of administrative data variables over time
for different sets of survey respondents and nonrespondents. For the HPS group, we match the
administrative data to respondents in each week of the survey and calculate the means of each variable
by week, allowing us to monitor how changes in data collection practices affect sample
representativeness. We calculate these means using base weights for the point estimates and the base
weight replicate weights for the standard errors. We discuss the effects of the final weighting procedure
in Section 3.2.
Next, we also calculate the means of these variables using the same administrative data separately for
ACS respondents and all occupied housing units (including ACS nonrespondents), the latter of which is
our benchmark to evaluate HPS representativeness. 4 Comparisons between the HPS and the ACS
3F
benchmark allow for understanding the representativeness of the HPS, particularly the direction and
evolution of any biases. Including the ACS respondents provides context on representativeness from
another major survey product.
For our analysis periods, we take the month and year of the first day of a HPS week and compare
variable means to the corresponding ACS sample month. Although the ACS is designed to produce
annual estimates, its structure of randomly sampled monthly panels provides an appropriate sample of
occupied housing units to match administrative data and construct benchmarks for comparison with the
HPS sample. Appendix Table A.1 lists the start dates of the HPS weeks used in this report. For example,
HPS week 12 had a start date of July 16, 2020, so we compare HPS means to those for ACS respondents
and nonrespondents who were sampled in July 2020. As of writing, our ACS data goes through 2022,
while our HPS data goes through 2023; for 2023 HPS data, we compare with the ACS benchmark for the
3.2 Effects of Weighting Adjustments on Nonresponse Bias
The comparisons described above examine the representativeness of the HPS sample before weighting
adjustments. The HPS weighting procedures adjust to counts of age, sex, race, and Hispanic Origin data
from the Census Bureau’s Population Estimates Program and educational attainment distribution from
4
The nonrespondents benchmark for the ACS excludes cases marked as temporarily occupied, vacant, delete, or
non-respondents not sampled for CAPI. The ACS base weights also incorporate additional factors applied to
account for subsampling in the survey design. See Appendix A of Rothbaum et al. (2021) for more details
10
the ACS. These adjustments should improve the representativeness of the HPS sample for many of the
measures we analyze in this report. For example, adjusting for being a college graduate should reduce
the bias with respect to income in the HPS given the correlation between income and education.
To examine the effects of weighting on reducing nonresponse bias, we construct means for the same
administrative variables described above, but use the survey’s final weight instead of the base weight.
We subtract these mean constructed with final weights from the mean for the ACS benchmark. Analyses
of these differences compared to the analogous statistics constructed with base weights will show how
well these weighting variables correct for nonresponse and coverage bias with respect to measures not
included in the weighting procedure, such as income and program participation.
3.3 Contact Frame Analysis
To supplement these analyses with more precise results on how a household’s characteristics vary by
the type of email address they have and to evaluate the impact of the decision to use only email
addresses with at least 90% deliverability when sampling from the Email Contact Frame, here we
analyze the entire Email Contact Frame rather than only the HPS sample. For addresses with any data in
the Email Contact Frame, we match the same administrative data and compare variable means by email
domain. Using the entire universe of email addresses will provide more precise estimates on the
representativeness of households with email addresses at eligible domains. Additionally, we compare
the administrative data of households in the Email Contact Frame to our ACS baseline to provide context
on coverage bias prior to filtering on email domains. Specifically, we will compare administrative data
means for:
1. ACS baseline of occupied housing units
2. ACS occupied housing units either in the Email Contact Frame or with a cell phone number in
the Phone Contact Frame
3. Any address in the Email Contact Frame
4. Addresses with email addresses with at least 90% deliverability rates
5. Address in the Email Contact Frame without at least 90% deliverability rates
Comparing the means for 1 (which is assumed to have no or nearly no coverage bias) and 2 (which we
assume represents the HPS sample frame if the survey could observe households’ occupancy status)
provides information on coverage bias from only sampling addresses with a matched phone number or
email address. Comparing 3 and 4 provides further information on potential change in coverage bias in
11
the HPS from restricting email addresses in the contract frame. Finally, 5 provides information on
households dropped from the sample frame after filtering email addresses.
4. Results
4.1 Graphical Analysis
In this section, we present graphs to show the representativeness of the HPS respondent sample
compared with ACS respondents and the ACS occupied housing benchmark over time. We start with
Figure 2, which shows results for a proxy of a person residing in a multi-unit structure such as an
apartment or condo building. We construct this measure from the Master Address File by taking MAFIDs
at a basic street address marked as valid living quarters (i.e., that are not marked as demolished or a
duplicate) and then noting those with more than two MAFIDs.
Figure 2: Person Resides in Multi-Unit Structure
Source: American Community Survey, Household Pulse Survey, and Master Address File. Shaded regions
represent 95% confidence intervals.
12
Overall, we see that HPS and ACS respondents are less likely to live in a multi-unit structure compared to
the benchmark. For much of 2021 and 2022, the bias (distance from the ACS benchmark) in the ACS is
about -2.5 percentage points. The bias in the HPS is more volatile, with the largest change coming from
introducing the 2020 Census update to the Contact Frames. Before that, the bias was about -7.5
percentage points for almost a year, but after, the difference decreased in magnitude to less than
5 percentage points (all HPS biases after incorporating the updated Contact Frames are statistically
significantly smaller than the biases in Phase 3). We conclude that improvements in the Contact Frames
were greatly beneficial for improving representativeness of the sample with respect to this variable.
We also see changes in bias at the start of Phases 2 and 3. For Phase 2, we see a decrease in bias.
Because people were only asked to be in the HPS for one week during this phase rather than also asking
for additional responses after two weeks like in Phase 1, this improvement may be due to limiting the
potential for attrition bias. Giving people two weeks rather than one to respond may have also improved
representativeness. The increase in bias in Phase 3 compared to the end of Phase 2 is puzzling because
there were no substantive changes in the HPS contact or sampling strategies corresponding with this
phase.
When evaluating the possible impacts of the switch to using only high-deliverability email addresses in
week 49, there is the confounding factor of varying spam classification rates over time and across email
providers, with acute problems precipitating the email address restriction. Ideally, one would compare
survey results immediately before and after the change, minimizing or eliminating other differences
across weeks that could affect findings. But because different email providers at different times
bounced back or blocked HPS emails at different rates, it is difficult to separate any possible effects from
restricting email addresses and variation in spam rates across providers. HPS had elevated email bounce
rates at least in weeks 46 and 47 at several email providers and did not send any emails in week 48 in
response. Even if bias magnitude decreases relative to these weeks, the restriction on eligible email
addresses could still hurt representativeness. Further, examining longer time horizons as a possible
solution risks increased effects of unrelated temporal effects, such as time of year. We advise extra care
when examining results related to the email address restriction.
Continuing to examine Figure 2, upon the email address restriction, volatility of the weekly estimates
does seem to decrease, with estimates becoming more stable after the transition. This is likely due to
email bounce back issues leading up to the week 49 change. For example, in week 48, HPS did not send
out any emails. In that week, the HPS mean is higher (and closer to the benchmark) than in the weeks 44
13
and 45. Whether the estimated bias in the HPS changes after the email restriction may depend on the
exact weeks compared. For example, using week 45 as the reference, where the HPS mean is lower than
in all subsequent weeks, bias magnitude is smaller starting in week 49. But if using week 47 as the
reference, bias magnitude would not be significantly different after the change.
Results for other variables support many of the same qualitative conclusions about the
representativeness of the HPS. Figures 3 and 4 plot results for income-derived variables. In both, bias
shrinks over time, with noticeable improvements at Phase 2 and the incorporation of the updated
Contact Frames and no drop in representativeness from the email restriction (when using weeks 46 and
47 as the reference group, bias magnitude is smaller in some weeks after the email change, particularly
for the top income percentile, but if using week 45 as the comparison, differences are not significant).
Unlike Figure 2, there is not a pattern of immediately increased bias at the beginning of Phase 3. We do
again see greater volatility week-to-week in the HPS than the ACS, as expected. Note also the different
signs of the bias: higher-income households are overrepresented in the HPS, while lower-income
households are underrepresented.
Figure 3: Household Income Greater than 75th Percentile
14
Source: American Community Survey, Household Pulse Survey, and IRS tax filings. Shaded regions
represent 95% confidence intervals.
Figure 4: Household Income Less than 25th Percentile
Source: American Community Survey, Household Pulse Survey, and IRS tax filings. Shaded regions
represent 95% confidence intervals.
The results in Figure 5 for the household having a person over the age of 60 are particularly interesting
because the bias in the HPS here is sometimes smaller than that of the ACS, though much more volatile
(see the spikes even between the updates to the Contact Frames and the email address restriction;
households with any members above the age of 60 are over- and under-represented at different points
in the HPS). This is a setting where the impact of the restriction to high-deliverability email addresses
(and/or the increased spam classification rates leading up to that decision) may be especially salient
because the direction of the bias changes, and households with members over age 60 move from being
overrepresented to underrepresented for the first time in week 48, when email was not utilized. In week
48, the proportion of households with someone over the age of 60 was significantly lower than it was in
week 47, and all weeks after the email restriction have point estimates between those values (all are
15
statistically significantly different from the week 47 value, and most are significantly different from week
48's). The volatility of the HPS’ point estimates is also lower after the email restriction than before it,
and this variable may be one of the most impacted by the change.
Figure 5: Person over Age 60
Source: American Community Survey, Household Pulse Survey, and IRS tax filings. Shaded regions
represent 95% confidence intervals.
The results for having any children in the household under age 5 in Figure 6 also show abrupt changes:
now in week 48, we see a spike in the proportion of these households compared to weeks 47 and 49
that is near the benchmark measure. Again, the post-switch HPS point estimates tend to lie between the
values for weeks 47 and 48, and bias magnitudes after the email change tend to be smaller than relative
to weeks 45, 46, and 47, but not 48.
16
Figure 6: Any Children Under Age 5 in Household
Source: American Community Survey, Household Pulse Survey, and SSA Numident. Shaded regions
represent 95% confidence intervals.
Figure 7 shows results for another variable that may be affected by the email restriction: the presence of
a person of color in the household, defined as the household having any member who is Hispanic (any
race), Black, Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, or
another race besides White. Up until the email restriction, the HPS respondents are often close to the
ACS respondents. But when only high-deliverability email addresses are used, there is an abrupt change
in the proportion of households with people of color, though it quickly returns to levels closer to those
before the switch. Without a perfectly clean counterfactual, the precise impact of the email restriction is
difficult to ascertain: the change within the HPS from week 48 to 49 is significant, but these weeks may
both be different from the higher means during Phase 3, the trough slightly earlier in week 44, and the
more stable post-restriction means starting around week 53. A range of impact magnitudes are
plausible.
17
Figure 7: Any Person of Color in Household
Source: American Community Survey, Household Pulse Survey, 2010 Census, and SSA Numident. Shaded
regions represent 95% confidence intervals.
Other variables show some of the same patterns as these graphs, but with less apparent impact of the
email restriction. They have been collated into Figure 8. These variables differ in the evolution of their
representativeness but provide a broader picture of the HPS results.
18
Figure 8: Other Results
19
Source: American Community Survey, Household Pulse Survey, IRS tax filings, and SSA administrative
records. Shaded regions represent 95% confidence intervals.
4.2 Weighted Results
These graphs show the HPS before any weighting adjustments are made. Using the final weights rather
than base weights will change the means of these variables, and thus the representativeness of the HPS
estimates. In this section, we present tabular comparisons of the ACS and HPS with base and final
weights. We use data from 2022, pooling all HPS weeks together and present results in Table 2, with
comparisons to the same ACS benchmark as before. All variables presented in this tables are derived
from administrative data; the only survey data used are the response outcomes from the HPS and ACS
and the survey weights from these surveys. We discuss selected results below.
The final weights in many cases greatly reduce bias (again to the benchmark) in the HPS. For every
income quartile, moving from base weights to final weights significantly shrinks the bias: from 5.94% to -
0.24% for the top income quartile, -5.42% to -1.36% for the bottom quartile, -2.04% to -0.56% for the
25th-50th percentiles, and 1.52% to 1.05% for the 50th-75th percentiles. Not only do these significantly
reduce biases, but the remaining bias is much closer in magnitude to that of the ACS.
20
Table 2: HPS and ACS Means of Administrative Data Variables with Base, Final Weights
Base Weight Base Weight Final Weight
Mean Mean Bias Mean Bias
Variable Benchmark HPS ACS HPS ACS HPS ACS HPS ACS
Income Greater Than
75th Percentile 25.00% 30.94% 26.49% 5.94% 1.49% 24.76% 25.51% -0.24% 0.51%
(0.04%) (0.08%) (0.04%) (0.09%) (0.02%) (0.10%) (0.04%) (0.10%) (0.02%)
Income Less Than
25th Percentile 25.00% 19.58% 23.26% -5.42% -1.74% 23.64% 24.02% -1.36% -0.98%
(0.04%) (0.06%) (0.04%) (0.08%) (0.02%) (0.11%) (0.05%) (0.11%) (0.03%)
Income Between 25th
and 50th Percentiles 25.00% 22.96% 24.66% -2.04% -0.34% 25.56% 24.96% 0.56% -0.04%
(0.04%) (0.07%) (0.05%) (0.08%) (0.03%) (0.10%) (0.04%) (0.11%) (0.03%)
Income Between 50th
and 75th Percentiles 25.00% 26.52% 25.59% 1.52% 0.59% 26.05% 25.52% 1.05% 0.52%
(0.04%) (0.07%) (0.05%) (0.08%) (0.02%) (0.11%) (0.05%) (0.11%) (0.02%)
Any Over 60 44.23% 45.34% 47.02% 1.11% 2.78% 43.16% 44.11% -1.07% -0.12%
(0.05%) (0.08%) (0.05%) (0.09%) (0.03%) (0.10%) (0.04%) (0.11%) (0.03%)
Any Children Under 5 15.88% 14.74% 14.84% -1.15% -1.05% 16.73% 15.70% 0.84% -0.19%
(0.03%) (0.06%) (0.03%) (0.07%) (0.02%) (0.09%) (0.04%) (0.10%) (0.02%)
Any Children 41.35% 39.79% 39.14% -1.56% -2.21% 42.67% 40.76% 1.32% -0.59%
(0.04%) (0.09%) (0.05%) (0.09%) (0.02%) (0.11%) (0.07%) (0.11%) (0.03%)
21
Any Non-Citizen 13.09% 11.46% 12.48% -1.63% -0.61% 12.55% 12.92% -0.55% -0.17%
(0.04%) (0.05%) (0.04%) (0.06%) (0.02%) (0.08%) (0.04%) (0.09%) (0.02%)
Any Persons of Color 36.12% 32.48% 33.37% -3.64% -2.75% 37.40% 35.70% 1.29% -0.41%
(0.05%) (0.09%) (0.04%) (0.10%) (0.03%) (0.11%) (0.04%) (0.12%) (0.03%)
Disabled (SSI or
OASDI Disability) 9.35% 8.30% 8.97% -1.05% -0.38% 10.47% 9.22% 1.12% -0.14%
(0.03%) (0.05%) (0.03%) (0.06%) (0.02%) (0.08%) (0.03%) (0.08%) (0.02%)
Dividend Income
Receipt 37.10% 44.32% 39.32% 7.22% 2.21% 36.92% 37.86% -0.19% 0.75%
(0.05%) (0.07%) (0.05%) (0.09%) (0.03%) (0.10%) (0.06%) (0.11%) (0.03%)
Employed 79.95% 82.06% 79.12% 2.12% -0.83% 82.33% 80.26% 2.38% 0.31%
(0.04%) (0.06%) (0.04%) (0.07%) (0.02%) (0.08%) (0.03%) (0.09%) (0.02%)
1098 Receipt
(Mortgage Lender) 44.41% 50.28% 46.46% 5.87% 2.05% 45.48% 45.42% 1.07% 1.01%
(0.05%) (0.09%) (0.05%) (0.10%) (0.03%) (0.12%) (0.05%) (0.13%) (0.03%)
1099-G Receipt 41.83% 44.20% 42.53% 2.36% 0.70% 43.20% 42.24% 1.37% 0.40%
(0.04%) (0.09%) (0.04%) (0.09%) (0.03%) (0.11%) (0.04%) (0.11%) (0.03%)
1099-MISC Receipt 13.84% 15.34% 14.33% 1.49% 0.49% 14.10% 13.87% 0.26% 0.02%
(0.03%) (0.06%) (0.04%) (0.07%) (0.02%) (0.08%) (0.03%) (0.08%) (0.02%)
1099-R Receipt
(Retirement) 40.91% 44.78% 43.18% 3.87% 2.28% 41.08% 41.18% 0.18% 0.27%
(0.05%) (0.06%) (0.05%) (0.08%) (0.03%) (0.10%) (0.05%) (0.11%) (0.03%)
22
Matched to Main
Administrative Data 87.16% 89.68% 89.11% 2.52% 1.94% 88.71% 88.36% 1.54% 1.20%
(0.03%) (0.05%) (0.03%) (0.06%) (0.02%) (0.07%) (0.04%) (0.08%) (0.02%)
Interest Income
Receipt 55.23% 60.40% 57.65% 5.17% 2.42% 54.04% 55.96% -1.19% 0.73%
(0.04%) (0.08%) (0.05%) (0.09%) (0.03%) (0.13%) (0.06%) (0.14%) (0.03%)
Married (IRS Filing
Status) 43.41% 49.53% 46.44% 6.12% 3.03% 44.86% 44.87% 1.45% 1.46%
(0.04%) (0.09%) (0.05%) (0.10%) (0.03%) (0.14%) (0.07%) (0.14%) (0.03%)
Medicaid Receipt 36.15% 30.78% 33.10% -5.37% -3.05% 38.05% 34.86% 1.90% -1.29%
(0.05%) (0.08%) (0.05%) (0.10%) (0.03%) (0.11%) (0.09%) (0.12%) (0.03%)
Multi-Unit Structure 28.02% 23.60% 25.30% -4.42% -2.72% 26.74% 27.85% -1.28% -0.17%
(0.04%) (0.08%) (0.04%) (0.09%) (0.02%) (0.12%) (0.06%) (0.13%) (0.03%)
Rental Income
Receipt 9.26% 10.50% 9.69% 1.24% 0.44% 8.62% 9.22% -0.64% -0.04%
(0.03%) (0.06%) (0.03%) (0.06%) (0.01%) (0.06%) (0.03%) (0.07%) (0.01%)
SSA OASDI
Retirement 29.35% 30.32% 31.72% 0.97% 2.37% 28.56% 29.28% -0.79% -0.07%
(0.04%) (0.07%) (0.04%) (0.08%) (0.02%) (0.09%) (0.04%) (0.10%) (0.02%)
Same-Sex Marriage 0.69% 1.03% 0.73% 0.35% 0.05% 0.85% 0.74% 0.16% 0.05%
(0.01%) (0.02%) (0.01%) (0.02%) (0.00%) (0.01%) (0.01%) (0.02%) (0.00%)
23
Schedule A
(Itemization) 13.96% 16.65% 14.47% 2.69% 0.51% 13.10% 14.00% -0.86% 0.04%
(0.03%) (0.06%) (0.04%) (0.07%) (0.02%) (0.07%) (0.04%) (0.08%) (0.02%)
Schedule C (Business) 26.86% 27.70% 26.04% 0.83% -0.83% 27.66% 26.61% 0.80% -0.25%
(0.05%) (0.09%) (0.04%) (0.10%) (0.03%) (0.09%) (0.05%) (0.10%) (0.03%)
Schedule D (Capital
Gains) 29.92% 36.64% 32.30% 6.72% 2.38% 29.18% 30.70% -0.74% 0.78%
(0.04%) (0.07%) (0.04%) (0.08%) (0.02%) (0.09%) (0.06%) (0.10%) (0.02%)
Schedule E (Pass-
Through Income) 17.35% 19.76% 18.16% 2.42% 0.82% 16.09% 17.32% -1.25% -0.03%
(0.04%) (0.07%) (0.04%) (0.08%) (0.02%) (0.08%) (0.04%) (0.09%) (0.02%)
Schedule SE (Self-
Employment) 20.16% 20.70% 19.66% 0.54% -0.50% 20.53% 19.92% 0.37% -0.24%
(0.04%) (0.08%) (0.04%) (0.09%) (0.03%) (0.08%) (0.04%) (0.09%) (0.03%)
Sector 11
(Agriculture) 1.88% 1.45% 1.85% -0.43% -0.03% 1.69% 1.87% -0.19% -0.01%
(0.01%) (0.02%) (0.02%) (0.02%) (0.01%) (0.03%) (0.02%) (0.03%) (0.01%)
Sector 21 (Mining) 1.03% 0.87% 1.01% -0.16% -0.02% 0.95% 1.04% -0.08% 0.01%
(0.01%) (0.02%) (0.01%) (0.02%) (0.01%) (0.03%) (0.01%) (0.03%) (0.01%)
Sector 22 (Utilities) 1.30% 1.22% 1.36% -0.08% 0.06% 1.19% 1.34% -0.11% 0.04%
(0.01%) (0.02%) (0.01%) (0.02%) (0.01%) (0.03%) (0.01%) (0.03%) (0.01%)
24
Sector 23
(Construction) 10.17% 8.76% 9.90% -1.41% -0.28% 9.79% 9.89% -0.39% -0.28%
(0.03%) (0.04%) (0.03%) (0.06%) (0.02%) (0.08%) (0.03%) (0.09%) (0.02%)
Sectors 31-33
(Manufacturing) 22.92% 21.53% 22.88% -1.39% -0.04% 23.56% 22.91% 0.64% -0.01%
(0.05%) (0.08%) (0.05%) (0.10%) (0.03%) (0.12%) (0.05%) (0.13%) (0.03%)
Sector 42 (Wholesale
Trade) 16.23% 15.88% 16.31% -0.35% 0.08% 16.38% 16.25% 0.15% 0.02%
(0.05%) (0.06%) (0.04%) (0.07%) (0.02%) (0.09%) (0.04%) (0.10%) (0.02%)
Sectors 44-55 (Retail
Trade) 26.38% 25.06% 25.66% -1.32% -0.72% 27.54% 26.01% 1.16% -0.37%
(0.05%) (0.09%) (0.05%) (0.10%) (0.03%) (0.11%) (0.06%) (0.12%) (0.03%)
Sectors 48-49
(Transportation and
Warehousing) 24.21% 22.77% 23.49% -1.44% -0.72% 25.44% 23.92% 1.23% -0.29%
(0.05%) (0.08%) (0.05%) (0.09%) (0.03%) (0.11%) (0.05%) (0.12%) (0.03%)
Sector 51
(Information) 12.59% 14.32% 12.83% 1.72% 0.23% 13.42% 12.79% 0.83% 0.20%
(0.03%) (0.06%) (0.04%) (0.07%) (0.02%) (0.08%) (0.04%) (0.09%) (0.02%)
Sector 52 (Finance) 14.55% 14.70% 14.72% 0.15% 0.17% 14.49% 14.65% -0.06% 0.10%
(0.05%) (0.06%) (0.04%) (0.08%) (0.02%) (0.09%) (0.05%) (0.10%) (0.02%)
25
Sector 52 (Real
Estate) 5.85% 5.65% 5.69% -0.20% -0.16% 5.90% 5.76% 0.05% -0.09%
(0.03%) (0.04%) (0.03%) (0.05%) (0.02%) (0.05%) (0.03%) (0.06%) (0.02%)
Sector 54
(Professional and
Scientific) 24.36% 26.65% 24.80% 2.29% 0.44% 24.91% 24.61% 0.54% 0.25%
(0.06%) (0.08%) (0.06%) (0.10%) (0.03%) (0.10%) (0.06%) (0.12%) (0.03%)
Sector 55
(Management) 37.79% 36.80% 37.01% -0.99% -0.78% 39.17% 37.43% 1.38% -0.36%
(0.06%) (0.08%) (0.05%) (0.10%) (0.03%) (0.10%) (0.06%) (0.12%) (0.03%)
Sector 56 (Waste
Management) 28.88% 27.62% 27.72% -1.26% -1.16% 29.82% 28.34% 0.94% -0.54%
(0.05%) (0.09%) (0.05%) (0.10%) (0.03%) (0.13%) (0.06%) (0.14%) (0.03%)
Sector 61 (Education) 16.38% 19.24% 17.11% 2.86% 0.73% 16.49% 16.88% 0.11% 0.50%
(0.04%) (0.06%) (0.04%) (0.08%) (0.02%) (0.11%) (0.04%) (0.11%) (0.02%)
Sector 62 (Health
Care) 28.56% 29.37% 28.44% 0.81% -0.12% 28.98% 28.64% 0.42% 0.08%
(0.05%) (0.09%) (0.05%) (0.10%) (0.03%) (0.12%) (0.05%) (0.13%) (0.03%)
Sector 71 (Arts and
Entertainment) 7.68% 7.69% 7.55% 0.02% -0.13% 8.01% 7.58% 0.33% -0.09%
(0.03%) (0.05%) (0.03%) (0.06%) (0.02%) (0.07%) (0.03%) (0.08%) (0.02%)
26
Sector 72
(Accommodation and
Food Service) 19.71% 17.99% 18.58% -1.72% -1.13% 20.52% 18.99% 0.81% -0.72%
(0.05%) (0.05%) (0.04%) (0.07%) (0.03%) (0.10%) (0.06%) (0.11%) (0.03%)
Sector 81 (Other
Services) 14.98% 14.88% 14.86% -0.09% -0.11% 15.39% 14.89% 0.42% -0.09%
(0.04%) (0.06%) (0.04%) (0.07%) (0.02%) (0.09%) (0.05%) (0.10%) (0.02%)
Sector 92 (Public
Administration) 10.12% 10.73% 10.29% 0.60% 0.17% 10.37% 10.43% 0.25% 0.30%
(0.03%) (0.06%) (0.04%) (0.06%) (0.02%) (0.08%) (0.04%) (0.09%) (0.02%)
SNAP Receipt 21.33% 17.84% 18.89% -3.48% -2.43% 22.86% 20.28% 1.53% -1.05%
(0.13%) (0.15%) (0.12%) (0.19%) (0.09%) (0.23%) (0.15%) (0.26%) (0.09%)
Single-Adult
Household No
Children 21.88% 19.44% 21.93% -2.44% 0.05% 19.42% 21.89% -2.46% 0.01%
(0.04%) (0.06%) (0.05%) (0.07%) (0.02%) (0.13%) (0.05%) (0.14%) (0.02%)
WIC Receipt 7.85% 6.44% 6.85% -1.41% -1.00% 8.09% 7.36% 0.23% -0.49%
(0.18%) (0.24%) (0.18%) (0.29%) (0.14%) (0.31%) (0.19%) (0.36%) (0.14%)
Source: American Community Survey, Household Pulse Survey, and Administrative Records. Standard errors for each measure are in parentheses
under the corresponding means. Bias is the mean in question minus the mean for ACS benchmark.
27
The impacts of weighting are not uniform across variables. For labor market outcomes, results are
mixed. For the HPS, SSA OASDI retirement bias shrinks slightly in magnitude with the final weights but
changes in sign. The bias for having a disabled person in the household based on SSI or OASDI also
changes sign but with a comparable magnitude. The employment indicator shows a small and
insignificant increase in bias from the addition of final weights. These biases still tend to be larger than
those of the ACS, which captures these variables very well.
It may be surprising to see the sign of the bias change. As an example of how this may happen, consider
education in the weighting model. We can correct the data for having too few people with a high school
diploma. But within the high school diploma, if we do not properly distribute the weights groups (e.g.,
households with and without a disabled person), then we may give the high school diploma-disabled
group too much weight compared to high school diploma-not disabled.
Other variables show further heterogeneity in representativeness before and after applying the final
weights. SNAP, WIC, and Medicaid receipt all show large decreases in bias magnitude for the HPS when
final weights are applied. But their performances compared to the ACS depend on the specific variable:
the ACS has a significantly smaller bias magnitude for Medicaid receipt, but not for WIC receipt, where
the HPS’ point estimate has a smaller bias magnitude (the difference between the HPS ‘and ACS’ bias
magnitudes is not significant). There are variables where the HPS’ bias magnitude is smaller than that of
the ACS after weighting, like being in the top income quartile or having dividend income, but there are
many more where the ACS significantly outperforms the HPS. For sectors of employment, certain sectors
are slightly overrepresented or underrepresented in the HPS, such as accommodation and food services
and management, while the ACS performs almost universally very well.
These results and the others from Table 2 highlight some consistency and some variability. Applying final
weights tends to improve the bias of the HPS, but not always, and the sign of the bias may also change.
Generally, the ACS outperforms the HPS in representativeness, but this too is variable-by-variable, and
the HPS can be comparable or better.
4.3 Contact Frame Analysis
In Section 4.1, we did not see major changes in the representativeness of the HPS sample from the
restriction of using only high-deliverability email addresses. To better understand any differences
between the households included and excluded by this change, we compare means of the same
administrative data matched to the email addresses in the Email Contact Frame with delivery rates
28
above 90% and other households. For expositional purposes, we focus on our primary economic and
demographic outcomes of interest.
Table 3 presents the means for each type of household across several different measures: the ACS
occupied housing units benchmark, ACS occupied housing units matched to the Contact Frames,
households with an email address in the Email Contact Frames, and households that have at least one
high-deliverability email address and those that do not. The inclusion of the ACS addresses matched to
the Contact Frames (Phone or Email) and the Email Contact Frame allows for understanding the sources
of bias inherent to the universe from which the HPS samples. Restricting to only ACS addresses matched
to the Contact Frames, for example, overrepresents the top income quartile and underrepresents multi-
unit structures. The full Email Contact Frame tends to have slightly larger bias magnitudes than the ACS
addresses in the full Contact Frames.
29
Table 3: Means Across Contact Frames
Mean ACS Mean Mean Mean CF High- Mean CF No High-
Variable Benchmark ACS CF Email CF Deliverability Email Deliverability Email
Income Greater Than 75th Percentile 25.00% 25.24% 25.03% 25.71% 24.46%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Income Less Than 25th Percentile 25.00% 24.60% 25.58% 24.98% 26.09%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Income Between 25th and 50th
Percentiles 25.00% 25.01% 24.60% 24.29% 24.87%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Income Between 50th and 75th
Percentiles 25.00% 25.14% 24.78% 25.02% 24.58%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Any Over 60 44.23% 44.71% 44.47% 41.17% 47.28%
(0.05%) (0.05%) (0.00%) (0.01%) (0.01%)
Any Children Under 5 15.88% 15.86% 15.89% 17.67% 14.39%
(0.03%) (0.03%) (0.00%) (0.01%) (0.00%)
Any Children 41.35% 41.52% 42.45% 46.38% 39.12%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Any Non-Citizen 13.09% 13.01% 12.72% 12.99% 12.49%
(0.04%) (0.04%) (0.00%) (0.00%) (0.00%)
Any Persons of Color 36.12% 36.05% 36.40% 39.15% 34.07%
30
(0.05%) (0.05%) (0.00%) (0.01%) (0.01%)
Disabled (SSI or OASDI Disability) 9.35% 9.42% 9.99% 11.04% 9.09%
(0.03%) (0.03%) (0.00%) (0.00%) (0.00%)
Married (IRS Filing Status) 43.41% 43.85% 42.88% 42.92% 42.85%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Multi-Unit Structure 28.02% 26.28% 25.49% 25.02% 25.85%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Single-Adult Household No Children 21.88% 21.37% 20.60% 17.51% 23.23%
(0.04%) (0.04%) (0.00%) (0.01%) (0.01%)
Source: American Community Survey, Household Pulse Survey, and Administrative Records. Standard errors for each measure are in parentheses
under the corresponding means.
31
Focusing on the third and fourth columns to see how the representativeness of the email sampling
frame changes when restricting only to high-deliverability email addresses, the variables that change the
most are the indicator for having a household member over 60, having children in the household, being
a single-adult household without children, and whether there is a person of color in the household.
Households containing people of color, for example, are disproportionately more likely to have a high-
deliverability email address associated with them in the Email Contact Frame, and they are
overrepresented at 39.15% compared to 36.05%-36.40% in the benchmark, ACS matched with the
Contact Frames, and all households with emails in the Email Contact Frame.
4.4 Comparison of Sampled and Self-Reported Addresses
All analyses in this paper involve administrative data linked to HPS respondents via the MAFID for the
sampled addresses. Because of respondent mobility and errors or lags in the Contact Frames, some HPS
respondents may reside at a different address than the one at which we believe they were located at
the time of sampling. We can examine this by comparing the sampled MAFID to the MAFID assigned to
their current address from the survey response. Unlike the sampled addresses, which have a 100%
MAFID assignment rate, the MAFID assignment rate for self-reported current address is imperfect,
mostly due to item nonresponse or typographical errors in responses.
32
Figure 9: Percent of Self-Reported Addresses Assigned a MAFID
Source: Household Pulse Survey. Shaded regions represent 95% confidence intervals.
33
Figure 10: Percent with Same MAFID
Source: Household Pulse Survey. Shaded regions represent 95% confidence intervals.
In Figure 9, we plot the percentage of self-reported addresses that were assigned a MAFID by week of
the HPS. The MAFID assignment rate was higher in Phase 1, particularly in weeks 1 and 2 before more
people began responding via text message invitations. This could be due to mode effects where
respondents are more likely to type out their address on a computer rather than a smartphone. The
MAFID assignment rate is also higher after the 2020 updates to the Contact Frames and for many weeks
after the email address restriction compared to some weeks before the restriction. In all weeks, the
percentage of self-reported addresses that were not assigned a MAFID is sizable. After Phase 2, the
MAFID assignment rate is between 55% and 65% for most weeks.
In Figure 10, we see low concordance between MAFIDs for the sampling and self-reported addresses
among the universe of self-reported current addresses assigned any MAFID. The agreement rate is
about 50% in Weeks 1 and 2, weeks that had more responses via the email invitation. Before the 2020
34
update to the Contact Frames, the agreement rates are below 45%. After the 2020 updates to the
Contact Frames, the agreement rate increased substantially and was often between 55% and 60%. This
is further evidence that the 2020 updates to the Contact Frames were associated with improved data
quality. There is no clear change in the agreement rate after the email address restriction.
Taking the results from Figure 10, the MAFID agreement rates suggest that the administrative data
comparison for HPS respondents from Sections 4.1 through 4.3 are confounded by changes in address
quality. For example, Figure 6 shows that for some HPS weeks, about 15% of HPS respondents were in a
household with a child under 5. However, this rate is calculated by matching administrative data on
household (based on MAFID) composition to the sampled MAFID. It is clear from Figure 10 that many of
these respondents do not reside at their sampled MAFID, so they may not actually live with a child
under 5. This disagreement in MAFIDs could lead to attenuation bias in the presented statistics, so the
bias may be larger than what is depicted in these figures. There is no perfect solution to this problem,
and using the MAFIDs of the self-reported addresses would likely result in a biased sampled if MAFID
assignment is not missing at random. Despite this complication, these analyses still provide useful
information on data quality. For example, the 2020 updates to the Contract Frames were associated
with both an increase in the MAFID agreement rate and a reduction in the differences of administrative
data means between HPS respondents and the benchmark. Because an increase in the agreement rate
may be associated with a reduction in attenuation bias, the fact that the difference in administrative
data means still decreased suggests that data quality did improve after this change in frame quality.
Nevertheless, some caution is warranted when analyzing the point estimates in these analyses.
5. Conclusion
The Household Pulse Survey provides near real-time information on the US population to inform
decisionmakers and stakeholders on key social items like vaccination status and mental health. We
evaluate its representativeness and how it has evolved with changes made to the survey. Using detailed
administrative records matched with the HPS and ACS, we identify the extent to which nonresponse bias
and coverage bias affect the representativeness of the HPS. We find that overall, particularly after the
application of production weights and from the improvements made to its sampling frame from 2020
Census efforts, the HPS often is able to limit both of these risks to deliver representative results that are
comparable in quality to those of the ACS. This is variable-dependent: the ACS is more representative in
35
general, including for key topics such as income, employment, and housing structure, while HPS is more
representative for a handful of variables such as being in the top income quartile and working in
education. Some caution is warranted in interpreting these estimates, given the discrepancies we
document between a respondent’s sampled addresses and their self-reported current address. It is also
possible that representativeness may be different for topics outside of the demographic and economic
measures available for our analysis, such as the bias in vaccination status documented by Bradley et al.
(2021). The economic and demographic variables studied in this paper may also have a higher
correlation with the HPS weighting variables than vaccination status, which could help further explain
the reduction in bias we see as the result of weighting procedures. We also find that although the HPS
limited its sampling frame and contact strategy to use only email addresses with delivery rates of at
least 90% (use of phone numbers was unaffected), that it maintained similar levels of
representativeness, with caveats about the difficulty in identifying an appropriate comparison period
from before the change.
References
Bradley, Valerie C., Shiro Kuriwaki, Michael Isakov, Dino Sejdinovic, Xiao-Li Meng, and Seth Flaxman.
2021. "Unrepresentative big surveys significantly overestimated US vaccine
uptake." Nature 600, no. 7890: 695-700. https://doi.org/10.1038/s41586-021-04198-4.
Eggleston, Jonathan. 2021. “Survey Isolation During COVID-19: The Effects of Suddenly Relying on
Address-Matched Phone Numbers for Interviewing Households.” American Association of Public
Opinion Researchers Annual Conference.
Eggleston, Jonathan, Yarissa Gonzalez, Carl Lieberman, Tim Trudell, and John Voorheis. 2024.
"Incorporating Administrative Data in Survey Weights for the Basic Monthly Current Population
Survey." Center for Economic Studies Working Papers, 24-02. U.S. Census Bureau.
https://www.census.gov/library/working-papers/2024/adrm/CES-WP-24-02.html
Eggleston, Jonathan, and Ashley Westra. 2020. “Incorporating Administrative Data in Survey Weights for
the Survey of Income and Program Participation.” SIPP Working Paper No. 292. U.S. Census
Bureau. https://www.census.gov/content/dam/Census/library/working-
papers/2020/demo/SEHSD-WP2020-07.pdf.
Genadek, Katie, Jashua Sanders, and Amanda Stevenson. 2021. "Measuring U.S. Fertility using
Administrative Data from the Census Bureau." Associate Directorate for Economic Programs
36
Working Paper Series. 2021-02. https://www.census.gov/content/dam/Census/library/working-
papers/2021/econ/chck2-adep-working-paper.pdf.
Groves, Robert M., and Emilia Peytcheva. 2008. "The Impact of Nonresponse
Rates on Nonresponse Bias: A Meta-Analysis." Public Opinion Quarterly 72, no. 2: 167-189.
Available at https://doi.org/10.1093/poq/nfn011.
Peterson, Sandra, and Norilsa Toribio. 2021. "Nonresponse Bias Report for the 2020 Household Pulse
Survey." U.S. Census Bureau. Available at https://www2.census.gov/programs-
surveys/demo/technical-documentation/hhp/2020_HPS_NR_Bias_Report-final.pdf
Reeves, Renee, and Kayla Varela. Forthcoming. "Evaluation of the Household Pulse Survey Experiments."
Demographic Statistical Methods Division Report Series.
Rothbaum, Jonathan, and Adam Bee. 2020. “Coronavirus Infects Surveys, Too: Nonresponse Bias During
the Pandemic in the CPS ASEC.” Social, Economic, and Housing Statistics Division Working Paper
#2020-10. Available at https://www.census.gov/library/working-papers/2020/demo/
SEHSD-WP2020-10.html.
Rothbaum, Jonathan, Jonathan Eggleston, Adam Bee, Mark Klee, and Brian Mendez-Smith. 2021.
"Addressing Nonresponse Bias in the American Community Survey During the Pandemic Using
Administrative Data." 2021 AMERICAN COMMUNITY SURVEY RESEARCH AND EVALUATION
REPORT MEMORANDUM SERIES # ACS21-RER-05 and SEHSD Working Paper #2021-24, U.S.
Census Bureua. https://www.census.gov/library/working-
papers/2021/acs/2021_Rothbaum_01.html
Appendix
Table A.1 Mapping HPS Weeks to ACS Sample Months
Week Number Month Day Year
1 April 23 2020
2 May 7 2020
3 May 14 2020
4 May 21 2020
5 May 28 2020
6 June 4 2020
7 June 11 2020
37
8 June 18 2020
9 June 25 2020
10 July 2 2020
11 July 9 2020
12 July 16 2020
13 August 19 2020
14 September 2 2020
15 September 16 2020
16 September 30 2020
17 October 14 2020
18 October 28 2020
19 November 11 2020
20 November 25 2020
21 December 9 2020
22 January 6 2021
23 January 20 2021
24 February 3 2021
25 February 17 2021
26 March 3 2021
27 March 17 2021
28 April 14 2021
29 April 28 2021
30 May 12 2021
31 May 26 2021
32 June 9 2021
33 June 23 2021
34 July 21 2021
35 August 4 2021
36 August 18 2021
37 September 1 2021
38 September 15 2021
39 September 29 2021
38
40 December 1 2021
41 December 29 2021
42 January 26 2022
43 March 2 2022
44 March 30 2022
45 April 27 2022
46 June 1 2022
47 June 29 2022
48 July 27 2022
49 September 14 2022
50 October 5 2022
51 November 2 2022
52 December 9 2022
53 January 4 2023
54 February 1 2023
55 March 1 2023
56 March 29 2023
57 April 26 2023
58 June 7 2023
59 June 28 2023
60 July 26 2023
61 August 23 2023
62 September 20 2023
63 October 18 2023
39
File and source
- File
- eggleston-lieberman-2024-ces-wp-24-60-household-pulse-nonresponse-bias.pdf
- Size
- 2,249,119 bytes
- SHA-256
- 31394d246256b795bc90f406f43963eb33b0486e28acdbe8307aca6e21225f46
- Original
- www2.census.gov