Agile Oversight in a Time of Crisis - Lessons learned and best practices in conducting oversight during the COVID 19 pandemic
- Issuer
- Government Accountability Office (GAO)
- Document type
- Report
- Date
- 2024-03-05
Summary
Agile Oversight in a Time of Crisis, a report dated March 5, 2024 that captures lessons learned and best practices shared at an Agile Oversight Forum hosted in January 2023 by the Pandemic Response Accountability Committee (PRAC) with the Council of the Inspectors General on Integrity and Efficiency. It summarizes a fireside chat with Comptroller General Gene Dodaro and Inspectors General Michael Horowitz, Christi Grimm and Mike Ware on leading with an agile mindset. Panel highlights cover enterprise risk management and risk-based planning, professional standards, sharing and integrating information, and communicating agile work. Case examples describe the DOL OIG's alert memorandum on unemployment insurance data, a USAID OIG advisory product on Ukraine and 12 pandemic-related audit reports by the California State Auditor's Office. Appendices list acronyms and forum credits.
Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used
Full text
Agile Oversight in a
Time of Crisis
Lessons learned and best practices in conducting
oversight during the COVID-19 pandemic
March 5, 2024
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
CONTENTS
Background.............................................................................................................................. 3
Panel Highlights | Enterprise Risk Management and Risk-Based Planning Enhance Oversight Agility.....5
Panel Highlights | Leveraging Professional Standards to Increase Innovation, Collaboration, and
Opportunities in Agile Work........................................................................................................ 9
Panel Highlights | We Can Share and Integrate Information to Perform Agile Work....................... 13
Panel Highlights | How to Communicate Agile Work ................................................................... .16
Appendix A | Acronyms............................................................................................................ 18
Appendix B | Document Preparation and Agile Oversight Forum Credits ....................................... 19
page | 2
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Our fast-paced world with its complex and evolving problems makes it
important to get information into the hands of government decision-
makers in real time. An agile approach to oversight is results oriented,
responsive, and reflective so we can learn, adjust, and move forward in
a more positive direction. The agile oversight process provides
stakeholders with access to information quickly so decisions can be
made to increase program integrity.
What Is Agile Oversight?
Recognizing the importance of an agile process, the Pandemic
Agile oversight is an efficient and
Response Accountability Committee (PRAC), in coordination with the
innovative approach to
Council of the Inspectors General on Integrity and Efficiency (CIGIE),
transparency and accountability.
hosted an Agile Oversight Forum (Forum) in January 2023. This report
It highlights issues and provides
captures the lessons learned and best practices shared by oversight
timely information about high-risk
experts from around the country at the Forum. Four topics drove the
areas or high priority topics and
Forum’s focus and conversation:
programs.
1. Incorporating enterprise risk management and risk-based
Who Should Read This planning
Report? 2. Embracing the flexibility of oversight standards
3. Sharing and integrating information and data
Federal, state, and local oversight 4. Communicating to help people better understand your agile
professionals interested in products
learning more about the leading
agile practices in use across our When used in tandem with our previously issued Agile Products Toolkit,
oversight community. this report can help oversight organizations better determine how to
adopt agile practices to keep our community’s work relevant and timely.
Background
After the publication of our Agile Products Toolkit in November 2020, our partners in the oversight
community were interested in learning more about how to implement agile oversight in their
organizations. We decided to examine this issue in the first-ever Agile Oversight Forum.
The Forum featured a “Fireside Chat” with four prominent oversight leaders who continue to
emphasize the need for agile approaches in their own organizations and across the broader
oversight and accountability community. In addition, the Forum included four panels that examined
important aspects of implementing an agile oversight approach (see Appendix B for full list of Forum
participants).
“Moving more broadly into an agile posture is a shift in culture,
and this takes time.”
-- Inspector General Christi Grimm,
Department of Health and Human Services
page | 3
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Fireside Chat Highlights | Leading an Organization with an
Agile Mindset
Gene Dodaro, Comptroller General, Government Accountability Office (GAO)
Michael Horowitz, Department of Justice (DOJ) Inspector General; PRAC Chair
Christi Grimm, Department of Health and Human Services (HHS) Inspector General
Mike Ware, Small Business Administration (SBA) Inspector General
Matt Winchell, Associate Program Analyst, PRAC (Moderator)
The Comptroller General and the three Inspectors General began their discussion with
the importance of embracing an “agile mindset.” They emphasized that this takes
commitment from the top as well as at all levels of the organization. This is consistent
with the GAO “Green Book” on internal controls, which states that leaders should set
the “tone at the top” for their organization by demonstrating their values, philosophy,
and operating style.
These speakers noted that to become more agile, leadership plays a key role in setting
the vision and creating the infrastructure and opportunities for agile oversight
processes. They discussed the importance of developing and fostering a culture that
can respond quickly to “meet the moment.” Panelists spoke about other essential
leadership actions, such as:
Being open and accepting of new ideas and possibilities.
Encouraging innovation by seeking diversity of thought and taking risks with
experimenting with new ideas.
Investing in resources, data analytics, and subject matter experts.
Cultivating a talented and motivated workforce that works well together and
leverages their strengths.
Prioritizing continuous learning.
Sharing stories and leadership practices to inspire teams.
Celebrating the value and impact this work creates.
page | 4
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Panel Highlights | Enterprise Risk Management and Risk-Based Planning
Enhance Oversight Agility
The Forum’s first panel titled, “How Enterprise Risk Management and Risk-Based Planning Enhance
Oversight Agility,” focused on using ERM and risk planning to help oversight entities be agile. For
more information about the panelists, see Appendix B.
What is Enterprise Risk Management (ERM)?
In the Enterprise Risk Management Practitioner’s Guide for Offices of Inspectors General, ERM is
defined as a process “applied in strategy settings across an entity, designed to identify potential
events that may affect the organization, and to manage risks within the organization’s risk appetite
to provide assurance regarding the achievement of entity objectives.” 1
How Does ERM Relate to Agile Oversight?
ERM can be applied in both “operational risk” and “oversight activity risk” planning and can help to
identify the highest oversight priorities and to adjust those priorities as new risks emerge. When
incorporating ERM into oversight work, our panelists recommended that organizations focus on the
following:
1. Creating a culture that considers and
measures risk. Operational Risk | Identifying
2. Using data to assess risks. and preparing for risks that
3. Understanding emerging risks. could impact the organizational
health.
Creating a Culture that Considers and Oversight Activity Risk |
Measures Risk Assessing the associated risk of
inaccurate or ineffective
Strong support from senior leaders for a risk-planning
oversight results for an area or
approach can be key to an organization’s risk
program.
identification and response. Planning should happen
collaboratively, so staff see the value that ERM provides, Risk Planning | Identifying,
not just hear about it. Panelists identified the following prioritizing, and managing risk.
best practices for creating a culture of risk planning:
Understand that processes and culture are related, but distinct. They should work in tandem.
Give concrete examples of how incorporating ERM during oversight planning adds value so
staff understand the purpose of ERM strategies.
Provide opportunities for staff to be part of the decision-making processes.
1 Risk appetite can be qualitative or quantitative and describes the amount and type of risk an organization chooses to
accept to achieve its mission and vision. The appetite is established by senior leadership and serves as the guidepost in
setting strategic goals and objectives. Descriptions of risk appetite should reflect your culture.
page | 5
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Including staff in the ERM process gets more buy-in and
"While leaders may understand creates a culture focused on risk planning. When staff are
the importance of agile work, brought together to understand and participate in the
due to their visibility on key process, they take ownership and may incorporate risk
issues, it is critical to provide assessment into their work. As a result, ERM processes are
necessary context and need for strengthened and solidified within the office culture,
increasing the likelihood that ERM strategies are applied to
shifts to agile to their broader
future work.
staff to continue to engrain the
mindset into the work the Using Data to Assess Risks
organization is doing."
Data is critical to informing agencies of their highest risks
Inspector General Michael and oversight entities’ subsequent responses. Once
Horowitz, Department of Justice identified, risks can then impact the plan, timing, and scope
and PRAC Chair of oversight work. When using data to assess risk and inform
planning, organizations should consider:
Mapping key risks and developing composite risk scores to identify the highest risk
programs.
Developing a predictive analytics model to identify emerging risks.
Using identified risks and risk scores to adjust models and mappings.
Promoting data sharing and data accessibility.
While data sharing across agencies is improving, data availability challenges remain that should
inform the risk assessment process (discussed in How We Can Share and Integrate Information to
Perform Agile Work). Offices may consider promoting data sharing capabilities, such as identifying
needed data sources in advance and using standardized security agreements.
The Importance of Data Access for Assessing Risk
The Department of Labor (DOL) Office of the Inspector General (OIG) had trouble assessing risk
in pandemic unemployment insurance programs because they did not have regular access to
state workforce data. DOL OIG used their legal authority to obtain the data and once they had
access determined that their analysis needed to be released quickly. They adjusted their work
accordingly and released an Alert Memorandum that identified one of the largest estimates of
potential fraud in the history of the DOL OIG (more detail in Agile Oversight in Practice – U.S.
Department Of Labor).
page | 6
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Multidisciplinary Structure Helps
Figure 1: Moving from Siloed Structure to a
Inform Emerging Risks Multidisciplinary Planning Method
Awareness of emerging risks, and the ability
to respond to them, are critical components
of ERM. An office’s understanding of
emerging risks will significantly impact its
oversight plan and work. To see the entire Audit Investigations Evaluations
risk picture, our panelists suggested shifting
from disparate work streams to a more
holistic, cross-cutting approach using a
multidisciplinary team (Figure 1).
Multidisciplinary structures bring together
subject matter experts from multiple mission
areas. Collaboration between these cross-
cutting teams can avoid silos, ensure proper
communication between stakeholders, and Multidisciplinary Team
ensure the most pressing work is addressed
Support Legal
in a timely manner. Audit Evaluations Investigations
Agile Oversight in Practice | DOL OIG
Why Agile | The OIG determined that the unprecedented infusion of federal funds into DOL’s
unemployment insurance programs, in addition to the limited controls protecting these funds,
created a high-value target and “perfect storm” for individuals and organized crime to exploit. The
OIG determined they should use a collaborative, agile approach in their audits and investigations.
This collaboration allowed the agile team to report results and recommendations quicker. Through its
analysis of data from all 54 state workforce agencies the OIG was able to identify four common, high-
risk areas related to the unemployment insurance programs. Using an Alert Memorandum, DOL OIG
shortened the project timeline to 4.5 months, compared to its normal 12-month timeline.
Best Practices or Lessons Learned
Increase collaboration and connection across directorates and between the OIG and the
agency to assure agile product timelines can be met.
Leverage data analytics to assess risk and use throughout the project’s lifecycle.
Use new ways of approaching issues and be creative in how information is presented. For
example, one of DOL OIG’s alert memos was completed using subpoenaed data from state
workforce agencies, which required the use of their Audit Business Intelligence Unit to help
analyze and present the data.
Shorten the review phase to align with the agile approach.
Conduct frequent meetings with internal stakeholders to reach consensus on conclusions
and recommendations.
page | 7
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Watch | DOL OIG - Agile Oversight in Practice - YouTube
Agile Product | Potentially Fraudulent Unemployment Insurance Payments in High-Risk Areas
Increased to $45.6 Billion
Agile Oversight in Practice | U.S. Agency for International Development (USAID) OIG
Why Agile | The OIG issued an advisory product on USAID’s $4 million in direct support to the
Ukrainian government. This agile approach was inspired by a 2021 memo issued by the Office of
Management and Budget that promoted accountability and cooperation between the OIG and its
agency by sharing risks and challenges at the outset rather than in retrospect. The advisory product
highlighted key considerations to inform the agency’s Ukraine response based on lessons from prior
oversight work examining other complex emergencies.
Best Practices or Lessons Learned
Leverage insights from previous work to connect issues and identify risks.
Budget time for changing circumstances as rapidly changing events mean rapidly changing
facts and figures.
Develop a reporting template to ensure the report is succinct, messaged appropriately, and
includes high-level and impactful findings.
Develop an effective communication process to increase the speed of development and
support the product’s overall impact.
Draw upon staff experience across the organization.
Watch | USAID OIG - Agile Oversight in Practice - YouTube
Agile Product | Advisory Notice - Key Considerations to Inform USAID’s Response in Ukraine
Agile Oversight in Practice | California State Auditor’s Office
Why Agile | Due to the amount of funding, changes to program eligibility, and the rapid nature of
allocations, the California State Auditor’s Office designated pandemic programs as high-risk. This
designation allowed the Auditor’s Office to respond to programs that needed immediate attention.
The California State Auditor’s Office issued 12 pandemic-related audit reports between November
2020 and November 2021. Findings resulted in bipartisan legislation to better prevent and detect
fraud. This legislation not only assisted the state workforce agency during the pandemic, but it also
supported fraud prevention and detection post-pandemic.
page | 8
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Best Practices or Lessons Learned
Consider emerging risks and the impact each government program has on the broader
population when identifying what issues need to be addressed first.
Determine availability of staff and other resources during planning to inform how many
programs and state agencies can be reviewed.
Adjust processes as needed to expedite work.
Find opportunities to leverage different types of products depending on overall goals.
Time and sequence final products to coincide with key decision-making processes in case
recommendations require shifts in funding to help administer the program or changes to
legislation.
Watch | California State Auditor – Agile Oversight in Practice - YouTube
Report | State High Risk State Management of Federal COVID-19 Funding
Panel Highlights | Embracing Professional Standards to Increase
Innovation, Collaboration, and Opportunities in Agile Work
The Forum’s second panel titled, “Agile Products and Professional Standards Increase Innovation,
Collaboration, and Opportunities,” focused on conducting agile work in alignment with quality
assurance standards. For more information about the panelists, see Appendix B.
All agile oversight work can meet professional standards with the proper design and scope. The
purpose and urgency of the project should drive which standards are necessary and appropriate. As
discussed in the Agile Products Toolkit, the professional standards of independence, due
professional care, and quality assurance are vital to ensure integrity and must be present in all
oversight entities’ work. To move towards agile principles in organizational processes, sharing,
learning, and adapting are essential. The three broad concepts below are designed to help
organizations think through incorporating agile processes into their operations while meeting
standards and increasing innovation and collaboration.
Organize and Design
Planning is an essential piece of all oversight products and is even more significant when conducting
agile work. Planning for agile products from the beginning can help your office better align resources,
strategize intentionally, and determine how to apply relevant standards. Our panelists shared ideas
for organizing and designing agile oversight work:
page | 9
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Keep an open mind on how to apply
professional standards to agile work.
"The concepts of agile and
Understand when the use of agile products is
standards are not in conflict if
most appropriate and impactful.
you have the right mindset and
Focus on the impact, timeframes, and approach…There is nothing in
appropriate standards to reach intended goals. the [yellow] book that says the
Revisit as necessary. longer you take, the higher
Coordinate early with partners and the quality of work."
stakeholders.
Comptroller General Gene
Leverage data analytics to focus scope and Dodaro, Government
methodology. Accountability Office
Communicate early to the subject(s) being
reviewed about the processes and product.
Adapt Internal Processes to Agile
In an agile approach, the focus should be on exploring efficient and creative ways to understand the
objective and quickly identify issues. Embracing agile disciplines is important to cultivate continuous
learning and innovation. Additionally, an agile approach can refine processes to help highlight and
report on risks and issues in a timely manner. In more traditional oversight, auditors and evaluators
tend to establish procedures to review and test widely. Below are techniques identified by our
panelists to adapt internal processes to agile:
Be flexible to adapt quickly to program and operational changes.
Break down procedures into smaller increments to help prioritize and stay focused on the
objective(s).
Establish a risk matrix, or other risk assessment methodologies, at the beginning of oversight
work. Later, this may help determine if results are significant or critical enough to warrant
separate reporting.
Capture observations via photography, video, or audio to help augment the message and
impact.
Schedule multidisciplinary team meetings to discuss progress, results, and refinements.
Use technology to streamline the review process.
Develop iterative processes and feedback to continuously improve the oversight approach.
page | 10
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Quality Assurance
All oversight standards provide a framework for performing high quality, accurate, and supported
oversight. At its core, the quality control function does not differ between traditional and agile
products. The minimum standards of independence, quality assurance, and due professional care
provide reasonable assurance that the oversight is conducted in an objective and accurate manner.
In addition, professional judgment is a powerful concept engrained in all independent oversight
standards and provides the flexibility for determining how to apply the standards. According to our
panelists, the following considerations may help with maintaining quality assurance processes in
your agile products:
Develop guidance that integrates the minimum internal
processes and standards required for completion, and
identifies expectations for determining whether evidence is
sufficient, valid, and reliable.
Focus on support needed to demonstrate ineffective
controls, needed improvements, and risk areas instead of
completing a full sample review.
Document the techniques and methodologies used to
determine the level of evidence required to support findings
or conclusions to help facilitate the independent verification
process, e.g., referencing.
Due to the significance of risk or urgency of the communication, it may be appropriate to
index and reference only critical facts or conclusions or incorporate tracing and verifying
techniques throughout the review.
Agile Oversight in Practice | Department of the Interior (DOI) OIG
Why Agile | The Infrastructure Investment and Jobs Act authorized nearly $4.7 billion to reclaim and
plug inoperable and noncompliant wells, known as orphaned wells, on federal, state, Tribal and
private lands. The amount of funding made it necessary for DOI to quickly identify how it planned to
use the funding and the OIG determined that an informational flash report was the best approach to
quickly provide information to stakeholders about the agency’s plans. The agile process laid the
groundwork for additional flash reports and the use of video in future reports. Constant
communication built strong working relationships with stakeholders. The team shared their fieldwork
success with the entire DOI OIG.
Best Practices or Lessons Learned
Plan what you want your report to look like early after considering the audience, messaging,
branding, and graphics.
Coordinate with all parties involved in the project; be open-minded to all ideas.
page | 11
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Conduct fieldwork and onsite visits to gather compelling observations, visual, and evidence-
based insights with the final report structure in mind.
Be clear about the type of product being issued. Stakeholders should understand the
standards being followed during the information gathering process.
Bring team members to site visits to learn about the assignment and help document
observations to include in the report.
Agile Product | Infrastructure Investment and Jobs Act, Orphaned Wells Programs
Agile Oversight in Practice | King County, WA Auditor’s Office
Why Agile | In 2020, Washington state expanded eligibility for property tax exemptions. However,
the move to telework and changes in processes led to a 19-month backlog in property tax exemption
applications. The King County Auditor’s Office needed to provide timely and independent analyses to
stakeholders reviewing the Assessor’s budget for resource considerations. The Auditor’s Office
produced an agile report on systems that process property tax exemptions. The Department of
Assessments concurred with all 12 recommendations in the report and quickly began to execute
corrective actions. The recommendations focused on developing or improving processes to help with
program execution, to comply with state rules for exemption renewals, and to meet increased
demand for exemptions.
Best Practices or Lessons Learned
All performance audits can benefit from agile practices. King County has taken an agile
approach for 10 years and has high rates of recommendation concurrence and compliance.
Audit plans do not need to be fully developed all at once. They can change. Though it may be
difficult at first, being agile means adapting internal processes to start with a broad plan and
refine along the way.
Develop goals and deliverables for established
Sprint | A short, time-boxed
sprint-based time frames.
period when the team works to
Document meeting minutes to support complete a set amount of work.
compliance with standards and for peer reviews.
Use tools, such as a researchable questions matrix, to help apply the agile process to meet
standards. Use this tool to map out the sub-questions from the objectives, sources of
information, criteria, scope and methodology, issues and limitations, and impact for each or
grouping of standards.
Watch | King County, Washington – Agile Oversight in Practice - YouTube
Agile Product | Property Tax Exemptions: Stronger Systems Needed to Meet Demand - King County
page | 12
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Panel Highlights | We Can Share and Integrate Information to
Perform Agile Work
The Forum’s third panel titled, “How We Can Share and Integrate Information to Perform Agile
Work,” focused on sharing information and working with data to be agile. For more information
about the panelists, see Appendix B.
Accurate, complete, and relevant data is essential to oversight. Understanding the data is equally
important to ensure quality and preserve credibility so that recommendations are taken seriously
and acted upon. Sharing and integrating data into agile processes can result in more efficient
analyses that create risk-based products and support timely decision-making. The panelists
discussed best practices for incorporating data into agile methods and products.
Early Identification of Needed Data
Obtaining the data can often be the most difficult and time-consuming activity, and the panelists
encouraged offices to identify the necessary data as early as possible. To do this, create a flexible
roadmap that the team can use to plan out major activities and data requirements.
Legal Requirements for Data Access
Access to data is governed by existing legal requirements, such as System of Records Notices, and
obtaining the appropriate legal rights to the data can add time to the process and even impact how
the data can be used. The panelists suggested leveraging already approved language by your own
legal counsel and/or by other agencies for data sharing agreements and using an established
precedent to expedite the legal process. It’s important to note that time spent coordinating with legal
counsel may require changes to a project’s schedule. Be ready and flexible to reallocate resources to
other projects to minimize time spent on projects that are not ready to begin.
Activities While Awaiting Access to the Data
Coordinating with legal counsel may take some time, so obtaining a copy of the data dictionary can
help analysts plan next steps once the data is available. The agency furnishing the data should share
the data dictionary and send the requesting agency the methodology for cleaning, or “normalizing,”
the data so the requestor can begin to understand it. Also, sending sample data will help the
receiving agency understand and test the import process.
Data Validation
Once you have the data, it is important to take time to validate it to
reasonably ensure that the information is complete and accurate.
This is an essential process because it helps avoid using invalid data
for testing, resulting in inaccurate conclusions that can threaten the
accuracy and integrity of the report.
page | 13
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Collaboration
Sharing experiences and best practices with partners is key to data sharing, data integration, and
developing standards for common data points across different datasets. As lessons are learned
about data, document and share them.
Create an Enterprise Data Management Structure
Review memoranda of understanding for data use and sharing, data use agreements, and other
documents for common attributes, and build an inventory of data acquired. Maintaining an inventory
of the data use attributes can help better plan work, timing, and data disposition. The panelists
stated that common attributes may include:
Privacy Level: Is the data public, nonpublic, or shareable across agencies?
Data Use: Can the data only be used for a specific project or activity?
Data Records Disposal: Once the project is complete, do you need to delete the data?
Using Data to Improve Internal Processes
One OIG’s data analytics office built an application that allows OIG staff who need access to
a system or data to make the requests directly in the system. This not only helped to track
overall requests and response times, but also helped to identify common requests and
streamline processes through the creation of self-service data tools available to staff. This
streamlining helped to free resources for more complex work.
Agile Oversight in Practice | SBA OIG
Why Agile | The urgency of getting COVID-19 assistance to businesses required the OIG to apply
agile oversight methodologies. The OIG’s preliminary review and investigative findings identified
weak internal controls and red flag indicators of potential fraud in both the Paycheck Protection
Program (PPP) and the COVID-19 Economic Injury Disaster Loan program. The OIG issued two reports
in the first 30 days after the passage of the CARES Act to provide guidance to stakeholders before
the first PPP loans were issued. Two weeks after the first loan was approved, the OIG issued another
report that resulted in legislative and program changes.
Best Practices or Lessons Learned
Collaborate with different groups and subject matter experts. For example, the OIG and the
Department of the Treasury’s Do Not Pay staff held multiple meetings to develop data
exchange agreements.
Hone your emotional intelligence and have courage, patience, and grace as obstacles are
encountered.
Design timely feedback loops to verify data matches to drive process improvements.
page | 14
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Perform continuous monitoring and use “boots on the ground” reviews to identify effective
controls.
Utilize your existing body of work to report on items requiring immediate attention.
Watch | SBA OIG – Agile Oversight in Practice – YouTube
Agile Products | PPP Loan Recipients on the Department of Treasury’s Do No Pay List, and
COVID-19 EIDL Program Recipients on the Department of Treasury’s Do No Pay List
Agile Oversight in Practice | Office of the New York State Comptroller
Why Agile | In a changing world with big challenges facing the nation, organizations able to
successfully readjust are, by definition, resilient. The Office of the New York State Comptroller’s
development of Smart Audit—their approach to digital transformation—combined investments in staff
and strategic partnerships, with investment in high-end analytic tools and technologies. This
approach produced speedier results leading to more informed and smarter decisions that give rise to
greater return on investment of taxpayer dollars. Investing in human and technological infrastructure
has driven innovation and led to impactful audits that save taxpayers money and improve New
Yorkers’ lives.
Best Practices or Lessons Learned
Allow staff proficient in data to serve as data consultants to help teams access and
understand public data sources, identify the appropriate analytic tool for each project, and
coach staff on using these tools.
Take an inventory of staff’s technical skills and interests and create rotational opportunities
for staff to work on data intensive projects to sharpen their skills.
Establish an Audit Innovation Unit that evaluates new technology and helps data consultants
roll out analytic solutions.
Conduct contests to engage staff in the creation of high-impact product ideas.
Focus on developing the right data visualizations and provide staff with the tools, trainings,
and templates to communicate the work most effectively.
Incorporate artificial intelligence into your business processes.
Watch | New York State Comptroller – Agile Oversight in Practice - YouTube
Agile Products | Department of Health: Use, Collection, and Reporting of Infection Control Data,
March 2022 Report; and Department of Environmental Conservation: Oversight of New York State
Forest Tax Programs, Aprill 2022 Report and Accompanying Story Map
page | 15
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Panel Highlights | How to Communicate Agile Work
The Forum’s final panel titled, “How to Communicate Agile Work,” focused on different approaches
to messaging and highlighting agile work. For more information about the panelists, see Appendix B.
Agile products allow oversight agencies to effectively communicate important
information to stakeholders about emerging issues in an easily digestible and
timely manner. Communication can take many forms, including snapshots,
advisories, reports, or web products. Many agile products consolidate the
results of lessons learned from previous work to address current events,
thereby giving agencies and policymakers information to facilitate decision
making. When thoughtfully designed, such products can accelerate agency
and policymaker decision making by condensing complex ideas into
digestible narratives and providing a quick understanding of program risks.
Agile Product Types and Designs
Oversight agencies should know their audience and design agile products accordingly. The content,
findings, and intended audience should drive the final product choice and design (see the Agile
Products Toolkit for more information). The panelists shared tips and best practices for designing
and communicating final products:
Consider whether the product is intended to reach a broader audience, such as the public, or
to inform specific policymakers about an emerging issue. This will help drive the structure.
Be clear, brief, and visually appealing so that busy stakeholders receive a product that
informs them to make better, more timely decisions about key issues.
Use innovative techniques such as linking to videos, visual cues, and using sidebars to tie
contextual information to other oversight products.
Consistently and clearly communicate agile product results to subject matter experts
responsible for this information.
Conduct a fatal flaw review before releasing the agile product to ensure the accuracy of all
information.
Have stakeholders conduct a final review to ensure the report has the most up-to-date data
and milestones.
Different Types of Agile Products
Panelists highlighted the following agile products during the Forum as innovative product types
and designs:
• GAO Snapshot, Electricity Grid Resilience
• DOI OIG Web Product, DOI Pandemic Response Spending: Where’s the Money
page | 16
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Agency and Stakeholder Reception
Agile products have received very positive feedback from agencies and external stakeholders.
According to our panelists, a short, story-telling approach offers stakeholders a quick education and
minimizes the time your office needs to do a press release for the media. The brevity, clarity, and
timeliness of agile products have assisted policymakers in quickly digesting information about
ongoing or significant issues and aided in discussions with federal agencies.
page | 17
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Appendix A | Acronyms
CIGIE Council of the Inspectors General for Integrity and Efficiency
DNP U.S. Department of the Treasury’s Do Not Pay Business Center
DOJ U.S. Department of Justice
DOL U.S. Department of Labor
DOI U.S. Department of the Interior
EIN Employer Identification Number
ERM Enterprise Risk Management
FDIC Federal Deposit Insurance Corporation
GAO U.S. Government Accountability Office
HHS U.S. Department of Health and Human Services
HUD U.S. Department of Housing and Urban Development
OIG Office of Inspector General
PPP Paycheck Protection Program
PRAC Pandemic Response Accountability Committee
SBA U.S. Small Business Administration
USAID U.S. Agency for International Development
page | 18
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Appendix B | Document Preparation and Agile Oversight Forum Credits
Preparation of this Report
The PRAC, in coordination with CIGIE, hosted an Agile Oversight Forum (Forum) on January 25, 2023.
The PRAC’s Audit, Inspection, and Evaluation Subcommittee coordinated with CIGIE’s Audit
Committee and the CIGIE Enterprise Risk Management Work Group to organize and conduct the
Forum. The day-long, virtual event featured experts from federal, state, and local oversight offices
and brought together more than 1,500 professionals from across the country who were eager to
learn more about agile practices and approaches. In addition to highlighting several specific
examples of agile oversight in practice (highlighted in call out boxes below and also posted on the
PRAC’s YouTube page), Forum presenters shared their thoughts and insights on what we can do now
as an oversight community to start performing with an agile mindset. After the Forum, PRAC staff
compiled and synthesized all the insights and information shared by the panelists and presenters to
create this agile oversight product. The product is intended to be a starting point for offices and is
not a standard for conducting oversight work. This product aims to capture the lessons learned and
best practices shared by oversight experts throughout the Forum.
DISCLAIMER
Emergent issues requiring a rapid response, as determined by an oversight entity, may also involve
full audits, inspections, or evaluations. These will follow all applicable standards as well as all
relevant oversight entity-specific policy and guidance and are not covered by this product.
Agile Oversight Forum Panelists and Moderators
Many thanks to the wonderful presenters and organizers who provided great insights and
discussions about how the oversight community can be more agile in our processes and products.
Their efforts in designing, developing, and executing their sessions were invaluable.
Introductory Fireside Chat on Agile Oversight
Gene Dodaro, Comptroller General, GAO
Michael Horowitz, DOJ Inspector General; PRAC Chair
Christi Grimm, HHS Inspector General
Mike Ware, SBA Inspector General
Matt Winchell, Associate Program Analyst, PRAC (Moderator)
How Enterprise Risk Management and Risk-Based Planning Enhance Oversight Agility
Jason Malmstrom, Assistant Inspector General for Audit, DOJ OIG
Jonna Mueller, Oversight Planning Program Manager, HUD OIG
Laura Nicolosi, Deputy Assistant Inspector General for Audit, DOL OIG
Jessica Southwell, Chief Performance and Risk Management Officer, DOL OIG (Moderator)
page | 19
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Agile Oversight in Practice Vignettes
Carolyn Hantz, Assistant Inspector General for Audit, DOL OIG
Marc Meyer, Assistant Inspector General for Investigations, USAID OIG
Elaine Howle, Special Advisor for State, Local, Tribal, and Territorial Oversight, PRAC and former
California State Auditor
Agile Products and Professional Standards Increase Innovation, Collaboration, and Opportunities
Andrea Deadwyler, Assistant Inspector General for Audits, SBA OIG
Luke Itnyre, Audit and Evaluation Manager, Federal Deposit Insurance Corporation (FDIC) OIG
Heather (Phoenix) Atkins, Operations Research Analyst, DOL OIG
Michael Frickel, Operations Research Analyst, DOL OIG
Marcos Contreras, Chief Management Officer, PRAC (Moderator)
Agile Oversight in Practice Vignettes
Kymber Waltmunson, King County, Washington Auditor
Eric Atkinson, Supervisory Auditor, DOI OIG
How We Can Share and Integrate Information to Perform Agile Work
Hong “Jo” Zhou, Analytics and Data Science Lead, PRAC
Tim Eagle, Director of Division of Analysis and Analytics, DOL OIG
Jennifer Wagner, Assistant Director for Office of the Chief Data Officer, HHS OIG
Kaitlyn Braswell, Management Analyst, HHS OIG (Moderator)
Agile Oversight in Practice Vignettes
Jeff Brindle, Director of Information Technology and Financial Management, SBA OIG
Tina Kim, Deputy Comptroller, Office of the New York State Comptroller
How to Communicate Agile Work
Cardell Johnson, Senior Executive for Office of Natural Resources and Environment, GAO
Hannah Maloney, Director of Audit Support Division, USAID OIG
Troy Dopke, Supervisory Auditor, DOI OIG
Christopher Stubbs, Director of Audit Inspection Evaluations Infrastructure, DOI OIG (Moderator)
Agile Oversight Forum Organizers
This virtual learning event required the assistance of professionals from the PRAC and across the
community. Without their help, such a successful event would not have been possible. In addition to
the moderators and panelists highlighted above, we would like to thank the following individuals for
their support designing, developing, and executing the Forum:
AmeriCorps OIG, Dave Karakashian U.S. Department of Defense OIG, Marcus
Gullett, Clorinda Hernandez, Kevin O’Connor,
CIGIE, Amanda Freeman and Doug Holt
and Megan Tormey
page | 20
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
DOI OIG, Nicole Miller, Kathleen Sedney, and PRAC, Jenn Contreras, Marcos Contreras,
Christopher Stubbs George Goodwin, Lynn Houston, Elaine Howle,
Ashleigh Miller, Laura Nichols, Lisa Reijula,
DOJ OIG, Jason Malmstrom
Ami Schaefer, Amanda Seese, Jenniffer
DOL OIG, Heather Atkins, Jessica Southwell Wilson, and Matt Winchell
FDIC OIG, Wendy Alvarado, Luke Itnyre, Terry SBA OIG, Mary Kazarian
Gibson, and Melissa Mulhollen
USAID OIG, Hannah Maloney, and Ryan
HUD OIG, Temika Edwards, and Martin Werner
Herrera
National Aeronautics and Space
Administration OIG, George Scott
page | 21
ENTERPRISE RISK AGILE PRODUCTS SHARING AND
COMMUNICATING
INTRODUCTION MANAGEMENT AND AND PROFESSIONAL INTEGRATING
AGILE PRODUCTS
RISK-BASED PLANNING STANDARDS INFORMATION
Acknowledgements
This product was prepared under the guidance of the PRAC Audit,
Inspection, and Evaluation Subcommittee, chaired by the U.S. Small
Business Administration Inspector General Mike Ware.
PRAC Point of Contact
Lisa Reijula
Associate Director of Outreach and Engagement
Lisa.Reijula@cigie.gov
Visit our website at
www.PandemicOversight.gov
Follow us at
page | 22
File and source
- File
- 2024-03-05-agile-oversight-in-a-time-of-crisis.pdf
- Size
- 619,440 bytes
- SHA-256
- fd9d8f917f1fbfe1cfde3e6b93c3e2e1a348424c5401cb772593fa18fc396af8
- Original
- www.oversight.gov