Full text
Pandemic Response Accountability Committee
Agile Products Toolkit
iiii
AGILE PRODUCTS TOOLKIT
PURPOSE
To aid federal Offices of Inspectors General (OIGs), state, and local agencies that
conduct quick reviews as part of their duties to provide expeditious oversight of
federal funds. The guidelines laid out in this toolkit pertain to reviews related
not only to the effects of the coronavirus disease 2019 (COVID-19) on federal
operations (e.g., Coronavirus Aid, Relief, and Economic Security Act expenditures),
but also to reviews of other emergent or ongoing challenges external to COVID-19,
such as threats to health, safety, and the environment. Designed specifically for
senior management and Congressional stakeholders responsible for the oversight
of taxpayer funds, this toolkit includes a set of guidelines, best practices, and
lessons learned to help conduct, complete, and issue reports or other products
expeditiously.
DISCLAIMER
Emergent issues requiring a rapid response, per OIG discretion and/or as deemed
appropriate based on individual circumstances, may also involve full audits,
inspections, or evaluations. These will follow all applicable standards as well as
all relevant OIG-specific policy and guidance and are not covered within this Agile
Products Toolkit.
EXECUTIVE SUMMARY
The broad scope and elevated urgency of federal agencies’ responses
to the COVID-19 pandemic presents inherent risks to project and
funding management. Specifically, the formal and deliberative
processes with which federal agencies plan, fund, authorize, and
execute projects conflicts with the speed necessary for a federal
pandemic response the American public needs. Agencies must
continue to implement internal controls necessary to prudently
plan the awarding and managing of contracts, grants, loans, and
other forms of assistance, while also balancing the need to quickly
identify and mitigate risks of fraud, waste, abuse, and improper
payments. The American public and Congress have expressed a
desire for more expedient and transparent oversight from OIGs, which
has only intensified given the challenges brought on by the COVID-19
pandemic. While some OIGs maintain longstanding guidance, others
have recently developed guidance regarding the issuance of “agile
products." This toolkit focuses on such agile products to help agency
management and other key stakeholders increase transparency
around emergent concerns, such as the disbursement of federal
funds.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
Contents
1
Introduction
1
What is an Agile Product?
2
What Standards are Required?
5
When Should I Use an Agile Product?
13
Can Agile Products Include Recommendations?
13
What Key Principles Should I Follow?
17
Appendix A: Objective and Approach
19
Appendix B: Review Techniques to Consider when
Doing an Agile Product Related to the Pandemic
Response
1
Introduction
The Coronavirus Aid, Relief, and Economic Security (CARES) Act
and other related legislation provided approximately $2.6 trillion in
federal spending to address the public health and economic crises
resulting from the COVID-19 pandemic. Since these funds require
the use of grants and loans to disburse large amounts of funds
quickly, they not only require a significant amount of attention and
support from federal agencies, they are also at high risk of fraud and
misuse.
To address these concerns, the CARES Act appropriated supplemental
funds to OIGs to promote transparency and conduct oversight of the
covered funds and agencies’ COVID-19 pandemic response. OIGs are
expected to (1) ensure funds are used effectively and efficiently and
(2) prevent and detect fraud, waste, abuse, and mismanagement.
As a part of these responsibilities, OIGs must conduct their work in
accordance with generally accepted government auditing standards
or other professional standards as well as inspection and evaluation
standards. Unique to the pandemic response environment, OIGs
also have a duty to provide policymakers and the public with
expeditious reporting of potential risks and management concerns.
Agile products such as flash reports and other special bulletins can
be used to meet this need if they also comply with key aspects of OIG
professional standards, such as independence, due professional care,
and quality assurance. This toolkit is intended to give OIGs and other
oversight offices guidance to help prepare these types of reports for
CARES Act oversight and beyond.
What is an Agile Product?
Agile products can come in many forms depending on agency
guidance or expectations. Primarily, agile products highlight issues
requiring immediate action for oversight officials or Congressional
stakeholders and others who have requested reviews of high-
risk areas. Additionally, agile products can inform, without requiring
actions, by providing transparency and ensuring key agency
leadership, affected stakeholders, and the public have access to
information more quickly.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
2
What Standards are Required for Agile Products?
Incorporating agile project management principles into how your office
conducts its work allows you to issue products more quickly, be more
adaptable to changes, and focus on the essential work required to
issue products—all of which helps your office prevent and root out
waste, fraud, and abuse during a pandemic or an emergency. For
products issued using agile principles (i.e., agile products), your office
may choose to follow, but are not required to follow all Government
Auditing Standards (e.g., the Yellow Book or GAGAS) or CIGIE’s Quality
Standards for Inspections and Evaluations (e.g., the Blue Book). If
the agile product does not follow Yellow or Blue Book standards, your
office should follow CIGIE’s Quality Standards for Federal Offices
of Inspector General, or the Silver Book. For all agile products, it is
recommended the product include a standards policy statement
indicating the standards under which the report was issued and that
the work adheres to the professional standards of independence, due
professional care, and quality assurance and followed procedures to
ensure accuracy of the information presented.
The following sections provide more detail about the professional
standards of independence, due professional care, and quality
assurance, which are vital to ensuring the integrity of the work
performed and aid in securing the public trust and confidence of the
oversight agency. It is noted that these professional standards must
be present in all work conducted by OIGs.
Independence
In general, oversight professionals should be free from personal,
external, and organizational impairments to independence both in fact
and appearance, such that an informed third party would conclude
the professional proves capable of exercising objective and impartial
judgment on all issues being reported. Oversight professionals and
their oversight agencies must maintain their independence, so their
opinions, findings, conclusions, judgments, and recommendations
remain impartial and are viewed as impartial by others. Given the
nature of agile reporting, special attention and new processes may be
necessary to ensure independence related to undue influence, bias,
and management participation.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
3
Oversight agencies should establish standard processes to identify
and evaluate threats to independence and apply safeguards to reduce
those threats and maintain the integrity of their agile products.
THREATS TO INDEPENDENCE
(1) Undue Influence
The threat of external pressures affecting
an oversight professional’s ability to make
independent and objective judgments.
(2) Bias
The threat that an oversight professional will,
because of political, ideological, social, or
other convictions, take a position that is not
objective.
(3) Management Participation
The threat resulting from an oversight
professional taking on the role of management
or otherwise performing management
functions on behalf of the agency or entity,
which will lead the professional to take a
position that is not objective.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
4
Due Professional Care
Oversight professionals should conduct reporting with due
professional care. This requires acting with integrity, exercising
objectivity, and maintaining professional skepticism. Professional
skepticism requires keeping an eye out for evidence that contradicts
other evidence obtained or for any information that brings into
question the reliability of evidentiary support. It also involves a
mindset in which professionals do not assume management is
dishonest.
Due professional care ensures the oversight team members
collectively possess the professional knowledge, skills, and abilities;
and in good faith and with integrity, are capable of diligently
gathering information and objectively evaluating the sufficiency and
appropriateness of evidence.
Quality Assurance
Oversight agencies should have appropriate quality assurance
controls in place to provide reasonable assurances their reports are
accurate and supported by evidence. The quality assurance process
should ensure statements of facts, figures, and dates are correctly
reported; the findings are adequately supported by the evidence; and
if applicable, recommended or required actions flow logically from the
evidence. See What Key Principles Should I Follow? below for more
suggestions related to ensuring sufficient quality assurance for agile
products.
IN TEGRI T Y
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
5
When Should I Use an Agile Product?
Based on our outreach to the OIG community (Appendix A: Objective
and Approach), we identified four primary situations in which an
agile product may be used by your office: Urgent Risk Identification,
Information Sharing, Interim Assessments, and Summarizing Lessons
Learned.
The following section provides more information about each situation.
For each, you will find examples of the agile products used and
explanations from various offices on why they chose a particular type
of agile product.
PRIMARY SITUATIONS FOR AGILE PRODUCTS
URGENT RISK
IDENTIFICATION
INFORMATION
SHARING
SUMMARIZING
LESSONS LEARNED
INTERIM
ASSESSMENT
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
6
A. URGENT RISK IDENTIFICATION
This is typically a descriptive product that outlines risk areas for
agency consideration based on prior OIG work.
Examples of product formats include:
White Papers
Capstone Reports
Management Advisory Reports
Memorandums
To determine if an agile product identifying an urgent risk is right
for your office, use the following examples as tools to inform your
decision.
Example A1
WHO
Small Business Administration OIG
WHAT
Serious Concerns of Potential Fraud in the EIDL Program
Pertaining to the Response to COVID-19
Strong indicators of widespread potential fraud and
internal control deficiencies associated with SBA’s
Economic Injury Disaster Loan program required
a flexible format to quickly communicate information to SBA OIG
stakeholders. The Management Alert format allowed SBA OIG to first
highlight suspicious activities and suspected fraud and then make
immediate suggestions for improvement.
Example A2
WHO
Department of Labor OIG
WHAT
The Pandemic Unemployment Assistance Program Needs
Proactive Measures to Detect and Prevent Improper
Payments and Fraud
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
7
The DOL OIG Alert Reports generally address a single
time-sensitive issue with the agency, requesting the
agency provide a response within one week. The DOL
OIG issued this Alert Report to notify the department of an issue
amidst an ongoing audit of the unemployment insurance program’s
expansion under the CARES Act. They included the alert issue in the
final audit report as well.
Example A3
WHO
Special Inspector General for the Troubled Assets Relief
Program
WHAT
Letter to the Treasury Secretary Regarding Better Use of
Funds
The need for immediate Treasury action prompted
the Special IG’s letter to the Secretary. The Special
IG wanted to bring to the Secretary’s attention that
SIGTARP’s foreclosure prevention program held billions of dollars they
could put to better use by repurposing the funds towards foreclosure
prevention related to COVID-19.
Example A4
WHO
Department of Transportation OIG
WHAT
Key Potential Risk Areas for the Department of
Transportation in Overseeing CARES Act Requirements
DOT OIG chose to release this as an official
correspondence instead of a lengthy audit report format
for two reasons: speed and quality. DOT OIG's decision
was driven by DOT’s need to quickly distribute $36 billion in CARES
Act funds and our desire to help identify oversight risks for those
funds as soon as possible, all while following the full measure of our
usual quality assurance process.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
8
B. INFORMATION SHARING
This is typically a descriptive product that provides a status or
snapshot on agency or program activities or funding (e.g., allocation,
use, etc.).
Examples of product formats include:
Status Reports
Flash Reports
Information Memorandums
To determine if an agile product to share information with agency
management or other decision makers is right for your office, use the
following examples as tools to inform your decision.
Example B1
WHO
Architect of the Capitol OIG
WHAT
Architect of the Capitol Status of CARES Act Funding
In response to rapidly changing resource requirements
and the need for increased transparency in government
budget functions, the AOC OIG chose the status report
format to provide a readily accessible presentation of data that
clearly outlines both the purpose and disbursement of allocated
funds. Under the guidance of leadership, AOC OIG tailored the data
for stakeholders who may use it to inform programmatic, operational,
and legislative processes.
Example B2
WHO
Department of Justice OIG
WHAT
Status of CARES Act Funding as of June 12, 2020
(Unaudited)
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
9
Example B3
WHO
Department of the Interior OIG
WHAT
The National Park Service’s Coronavirus Response Operating
Plans
The DOI OIG developed the “flash report” format to
relay key information quickly and concisely. These
reports are rapidly issued to decision makers, and the
format—two columns, clearly structured, easy to read, with high-level
overviews and data visualizations—provides actionable information
quickly and concisely to support the oversight mission and promote
accountability.
Example B4
WHO
Small Business Administration OIG
WHAT
Small Business Administration’s Implementation of the
Paycheck Protection Program Requirements
SBA OIG chose the Flash Report format because it
needed to quickly provide information to three Senators,
and this agile format was fast and flexible. The Flash
Report allowed SBA OIG to summarize its ongoing oversight work and
present a comparative analysis of key provisions of the CARES Act
and SBA’s rules, procedures, and public guidance for implementing
the Paycheck Protection Program. SBA OIG highlighted gaps and
suggested improvements to align Paycheck Protection Program
requirements with the CARES Act.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
10
10
C. INTERIM ASSESSMENT
An interim assessment can also be used to share information more
quickly with stakeholders during a broad scope review, a targeted
assessment of a large program, or a series of reviews.
Examples of product formats include:
Management Alerts
Management Advisory Reports
Memorandums
Interim Reports
To determine if an agile product providing an interim
assessment is right for your office, use the following
examples as tools to inform your decision.
Example C1
WHO
Amtrak OIG
WHAT
Observations on Amtrak’s Use of CARES Act Funds
Example C2
WHO
Department of the Treasury OIG
WHAT
Interim Audit Update – Coronavirus Relief Fund Recipient
Reporting
Treasury OIG identified a need to report an initial finding
and recommendation prior to completion of all audit
work, and therefore, selected the interim report format.
Example C3
WHO
Department of Labor OIG
WHAT
COVID-19: MSHA Faces Multiple Challenges in Responding
to the Pandemic
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
11
11
DOL OIG issued this limited scope audit report to
provide the Mine Safety and Health Administration with
useful and timely information about its initial response
to the COVID-19 pandemic. DOL OIG conducts limited scope audits
during the earliest stages of a new or revised program, funding, or
expenditure to evaluate the adequacy of initial implementation.
D. SUMMARIZING LESSONS LEARNED
This is typically a descriptive product that outlines lessons learned for
agency consideration based on prior OIG work.
Examples of product formats include:
White Papers
Capstone Reports
Management Advisory Reports
Memorandums
To determine if an agile product summarizing lessons
learned is right for your office, use the following
examples as tools to inform your decision.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
12
12
Example D1
WHO
Small Business Administration OIG
WHAT
Risk Awareness and Lessons Learned from Prior Audits of
Entrepreneurial Development Programs
The white paper descriptive format allowed SBA
OIG to quickly share details about its work on SBA’s
entrepreneurial development programs. This agile
format allowed SBA OIG to succinctly summarize previous findings,
highlight relevant risk areas, and focus on key points to consider
when administering COVID-19 funds.
Example D2
WHO
Department of Labor OIG
WHAT
CARES Act: Key Areas of Concern Regarding Implementation
of Dislocated Worker Grant Provisions
DOL OIG issued this advisory report based on years of
prior oversight work, reiterating issues raised applicable
to the dislocated worker provisions included in the
CARES Act. DOL OIG’s advisory reports do not include new work or
new recommendations. They are generally issued at the start of a
new or revised program, funding, or expenditure and serve as a notice
of known key findings.
Example D3
WHO
Department of Education OIG
WHAT
Challenges for Consideration in Implementing and
Overseeing the CARES Act
Example D4
WHO
Department of Defense OIG
WHAT
COVID-19 Expenditures — Lessons Learned Regarding
Awareness of Potential Fraud, Waste, and Abuse Risk
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
13
Can Agile Products Include Recommendations?
An agile product can have a recommendation, but it is more likely that
your product will include an informal recommendation, suggestion,
key action, area of consideration, or nothing at all. For example, a
product summarizing lessons learned may include considerations or
advice based upon previous work completed by your office. In another
example, a data brief with the intent of sharing information would
likely not include a recommendation. However, it is at the discretion
of each Inspector General to determine if a formal recommendation is
needed.
What Key Principles Should I Follow?
The following key principles apply to agile products issued quickly,
performed in accordance with CIGIE’s Quality Standards for Federal
Offices of Inspector General. Please note these key principles do not
apply to non-audit services your office agrees to perform for agency
management.
Quality Assurance Guidelines
Designed specifically for rapid delivery
of information, agile products do
not require the same level of quality
assurance as an audit, inspection, or
evaluation; however, the following key
principles should be applied to agile
products.
9
STAFF ASSIGNMENT Employ staff who are competent and
independent, and make sure enough staff are assigned according
to the scope and timeframe of the product.
9
EVIDENCE REVIEW Assess evidence for relevance, reliability,
and sufficiency. Document the approach for collecting evidence
according to the objective and nature of the project.
9
SUPERVISION Ensure proper supervision of staff, especially for
those teams with less experienced staff.
QUALI T Y
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
14
14
9
INDEX AND REFERENCING Complete indexing and referencing,
or other quality assurance review, for key facts and findings. Keep
in mind that the Silver Book does not require that a product be
indexed and referenced in its entirety, as with audits conducted
under the Yellow Book. Indexing and referencing should also be
completed as you go so you can release the product quickly.
9
LEGAL REVIEW Conduct legal or other reviews at the discretion of
your office based upon the content/objective of your agile product.
Milestones and Timeframes
for Completion
Milestones and timeframes for
completion should be set based
on the nature of the review and
requirements of the requesting
party (internal or external stakeholder). The following best practices,
identified by various OIGs, should help keep your agile product moving
quickly through the process.
9
INTERNAL BUY-IN Ensure buy-in of rapid cycle by OIG senior
leaders.
9
EFFICIENT ANALYSIS METHODS Consider efficient timesaving
data collection and analysis methods such as surveys and data-
driven findings. (See Appendix B: Review Techniques to Consider
when Creating an Agile Product Related to the Pandemic
Response for more information on these analysis methods.)
9
CONCURRENT REVIEW Expedite review by implementing a
concurrent review with internal stakeholders (such as internal
senior leaders, editors, visual communications specialists, etc.).
Communication with Agency Management
Agile products using the Silver Book standards may be new to agency
management. To ensure that all stakeholders understand the use of
these standards, be sure to have open and ongoing communication
with everyone involved, especially management. Various OIGs
identified the following principles to follow when communicating with
agency management about your agile product.
MILESTONES
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
15
9
COMMUNICATE EARLY As soon as you know about an agile
product, make sure management knows why the product requires
a short timeline and how they can support an expedited review.
Explain how the process may differ from the usual processes,
including the Silver Book standards used to validate the product.
9
COMMUNICATE REGULARLY Ensure regular and open
communication to maintain a positive rapport with the agency
throughout the process. Meet with senior agency management on
a regular basis to share information and discuss broader issues.
Meetings help keep the work on track with the rigorous timeline.
9
AVOID SURPRISES Share findings or observations, as appropriate,
throughout the project to reduce surprises and conflicts during
the reporting stage. When you share these findings, you should
encourage agency management to start contemplating their
responses to help keep pace with the shorter timeframe. Ongoing
communication is key to avoiding surprises or confusion, which in
turn increases the likelihood of receiving an expedited response.
9
USE ENTRANCE AND EXIT CONFERENCES Formal entrance and
exit conferences can be used as a structured communication
channel to inform agency management on latest the findings. Use
these according to your office’s preference.
Product Distribution and
Follow-up
How you classify your agile
product, whether it is Sensitive
But Unclassified or For Official Use
Only, and how you distribute your
agile product will be based on your
office’s preferences. If possible, we
encourage you to post your agile products online in the interest of
transparency and accountability. This will depend on the content of
your agile product and your office’s discretion.
DISTRIBUTION
COMMUNICATION
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
16
16
9
MANAGEMENT RESPONSE Send your agile product to agency
management for review and response. Your office should
determine whether to incorporate management's response into
the report, and if so, whether it should be a summary or the full
response. This will depend on the format and content of your agile
product.
9
PRODUCT FOLLOW-UP Identify, if applicable, the policies and
procedures for following up on agile products. Since agile products
will likely not include a formal recommendation, there may be
some differences in follow-up procedures compared to non-agile
products. Determine how your office plans to handle issuing,
resolving, and following up on agile products and be sure agency
management knows about and agrees with the plan.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
17
17
Appendix A: Objective and Approach
Objective
This Agile Products Toolkit is
designed to aid both federal Offices
of Inspectors General (OIGs) as well
as State and local agencies that
provide oversight of Federal funds as
they seek to conduct quick-response
reviews. These reviews relate not only to addressing the effects of
the coronavirus disease 2019 (COVID-19) on federal operations
(e.g., CARES Act expenditures, among others) but other emergent or
ongoing challenges external to COVID-19 that require an expedited
review (e.g., threats to health, safety, the environment, or to ensure
effective stewardship in the expenditure of federal funds). This
Toolkit is intended to provide a set of guidelines, best practices, and
lessons learned, which we refer to as “key principles” to consider
in conducting, completing, and expeditiously issuing reports or
OBJECTIVE
other products. These products seek to provide the greatest benefit
to senior agency management and Congressional stakeholders
responsible for managing, overseeing, and ensuring effective
stewardship of taxpayer funds. This Agile Products Toolkit does not
create or mandate any new professional guidance, requirements, or
standards. Oversight agencies remain responsible for adhering to
applicable standards.
Approach
To accomplish the above objective,
the Audit Subcommittee under the
PRAC established a working group
comprised of representatives from five
OIGs to identify the various types of
agile products used across the community as well as best practices
for developing and issuing such products. The working group solicited
examples of product types and guidance from across CIGIE and
conferred with the CIGIE Peer Review team to gather questions related
APPROACH
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
18
18
to policy. The working group researched the various types of products
using information provided on the PRAC’s website, Oversight.gov, and
individual OIG websites.
The best of existing guidance and products from across the CIGIE
community has been distilled into this user-friendly, quick-reference
guide for federal, state, and local agencies, to help provide oversight
of federal funds through quality agile products.
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
19
19
Appendix B: Review Techniques to Consider when
Creating an Agile Product Related to the Pandemic
Response
As part of the COVID-19 pandemic response, federal offices have
operated in an increased telework environment and travel for on-site
reviews has been limited. Within this operating model, your office will
need to utilize new or additional techniques to obtain information it
usually would through in-person interviews, physical observations, and
controls testing.
This appendix discusses three review techniques to assist you in
completing your agile product and may help identify additional types
of reviews to consider for agile reporting. These review techniques
include remote surveys, virtual “on-site” reviews, and increased data
usage.
Remote Surveys
Conducting a remote survey can help
you collect information and data
without having to travel. Design your
remote survey in such a way that it
encourages a high-response rate
and rapid return. Review the list of
recommendations below to help structure your survey accordingly.
9
Acknowledge that the current operating environment is unusual
and might make it more difficult to respond. Including this type of
acknowledgement at the start of your survey helps build goodwill
with the potential participants. It shows you know their time is
valuable and you appreciate their efforts.
9
Focus on questions that will help you complete your objectives.
Make the distinction between what you need to know and what is
good to know.
REMOTE SURVEYS
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
20
20
9
Limit the number of questions asked. The more questions you
ask, the longer it will take to complete the survey. Remember the
people you are surveying may have limited time to respond.
9
Keep it simple. Including questions that may be difficult to answer
or require the participants to do a lot of work could decrease your
response rate.
9
Avoid using “agree or disagree” questions. These questions could
influence the responses based upon the language used.
9
To speed up your analysis of responses, use close-ended
questions as much as possible. These include “yes” or “no”
questions or questions that use a rating scale.
9
Be careful not to ask double-barreled questions. These questions
address more than one issue yet allow for only one response
(Example: How satisfied are you with the response time and
quality of service?)
9
Ask someone to review your survey questions. Complete a pre-test
of your survey questions by having volunteers complete the survey
to ensure the questions are easy to understand and answer. Adapt
the survey based on the pre-test issues encountered or where
questions were not clear.
9
Consider including the survey questions and response summary in
an appendix of your agile product.
Examples of Survey-Driven Findings
(1) Department of Homeland Security OIG, Early Experiences with
COVID-19 at CBP Border Patrol Stations and OFO Ports of Entry, OIG-
20-69, September 4, 2020
(2) Department of Justice OIG, Remote Inspection of Federal Bureau
of Prisons Contract Correctional Institution McRae, Operated by
CoreCivic, 20-098, August 31, 2020
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
21
21
Virtual “On-site” Reviews
Offices of Inspectors General
have increased their use of virtual
collaboration platforms to conduct
their work remotely and to create
a virtual “on-site” environment.
Consider the following virtual
options and techniques to help you complete your work remotely:
9
Virtual one-on-one video interviews with agency staff
9
Virtual panels and group discussions using videoconferencing
capabilities
9
Video “walkthroughs” where agency staff use video capabilities to
help a review team obtain physical evidence
9
Photos with time and date stamps as physical evidence
9
Screensharing to observe testing of various controls
Increased Data Usage
Completing data-driven reviews can
provide your office with insight into
agency activities without completing
on-site work. While the Silver Book
standard does not require in-depth
systems control testing, you can
use some of those principles for assessing the appropriateness and
sufficiency of the data. When collecting data to support your agile
product, consider the following tips:
9
Clearly identify the data source and how the selection was made
for samples.
9
Identify any limitations associated with the data and include a
section to outline them for your readers.
9
Attribute the information to the source, such as the Federal
Procurement Data System or an internal agency data system.
VIRTUAL REVIEW
DATA USAGE
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
22
22
9
Obtain a data attestation statement from the agency/data system
managers regarding the validity, the completeness, and any
limitations of the data.
9
Validate the data or information to the extent possible.
Examples of Data-Driven Findings
(1) Department of the Interior OIG, August 2020: Where’s the Money?
2020-FIN-068, September 28, 2020
(2) U.S. Department of Agriculture OIG, USDA Coronavirus Disease
Funding, September 30, 2020
Data on Diversity, Equity, and Inclusion
Multiple studies and reports have looked at how the COVID-19
pandemic has affected racial and ethnic groups, both in terms of
health and economics. One report focuses on the pandemic’s effect
on small businesses and the fact that Black, Latinx, and Asian
businesses saw a greater percentage drop in ownership than white
businesses. In addition, numerous reports from the Centers for
Disease Control and Prevention document how long-standing systemic
health and social inequities have increased the likelihood of adverse
outcomes for many people from racial, ethnic, and socioeconomic
minority groups, including dying from COVID-19. The Government
Accountability Office released recommendations for the Centers
INCLUSION
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
23
23
for Disease Control and Prevention to take additional measures
to help ensure it maintains complete and consistent collection
of demographic data in order to more clearly assess the impacts
COVID-19 has on racial and ethnic minority groups. Similarly, the
Small Business Administration’s OIG issued a flash report revealing
that SBA did not require demographic data collection related to the
Paycheck Protection Program and, as such, would not be able to
determine the loan amounts provided to prioritized underserved
groups.
While not every agile product related to the pandemic response will
assess the impacts on minority individuals or groups, we encourage
OIGs to consider if any of their agile projects could address diversity,
equity. and inclusion. Some areas to explore when planning agile
projects are:
9
Whether the agency collects relevant demographic data for its
programs and operations.
9
If the datapoints are collected, whether the information
represents disproportionate impacts on protected classes of
individuals (for example, race, gender, age, ethnic origin, religion,
sexual orientation, or disability).
9
Whether the agency should collect additional demographic data
to identify relevant issues.
Examples of Reports with DE&I-Related Findings
(1) Federal Reserve Bank of New York, Double Jeopardy: COVID-19’s
Concentrated Health and Wealth Effects in Black Communities,
August 2020
(2) Special Inspector General for the Troubled Asset Relief
Program, Mismanagement of the Hardest Hit Fund in Georgia,
SIGTARP-18-001, October 13, 2017
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B
24
24
Acknowledgements
This report was prepared under the guidance of the
PRAC audit subcommittee, chaired by Small Business
Administration Inspector General Mike Ware.
Special thanks to the working group:
Toayoa Aldridge, USAID OIG
Eric Anderson, DOE OIG
Terry Gibson, FDIC OIG
Erika Lang, DHS OIG
Jenniffer Wilson, SIGTARP
A Committee of the
Council of the Inspectors General
on Integrity and Efficiency
I
T
PRAC Point of Contact
Lisa Reijula
Associate Director of Outreach and Engagement
Lisa.Reijula@cigie.gov
Visit our website at
www.PandemicOversight.gov
INTRODUCTION
WHAT IS
AN AGILE
PRODUCT?
WHAT
STANDARDS
ARE REQUIRED?
WHEN SHOULD
I USE AN AGILE
PRODUCT?
WHAT KEY
PRINCIPLES
SHOULD I FOLLOW?
APPENDIX A
CAN AGILE
PRODUCTS INCLUDE
RECOMMENDATIONS?
APPENDIX B