COVID-19 Affected Ongoing Construction of Major Facilities Projects (GAO-21-417)
Summary
A GAO report to congressional committees, GAO-21-417, dated June 2021, on the National Science Foundation's (NSF) major facilities projects, research infrastructure such as telescopes and research vessels that cost over $100 million. It reports that since GAO's April 2020 report, the Large Hadron Collider High Luminosity Upgrade program began construction. GAO finds that it and the four other major facilities projects in construction have weathered schedule delays associated with the COVID-19 pandemic. The report also describes steps to partially account for increased costs associated with the pandemic, such as paying project staff.
Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used
Full text
United States Government Accountability Office
Report to Congressional Committees
NATIONAL SCIENCE
June 2021
FOUNDATION
COVID-19 Affected
Ongoing Construction
of Major Facilities
Projects
GAO-21-417
June 2021
NATIONAL SCIENCE FOUNDATION
COVID-19 Affected Ongoing Construction of Major
Facilities Projects
Highlights of GAO-21-417, a report to
congressional committees
Why GAO Did This Study What GAO Found
NSF supports the design, construction, Since GAO’s April 2020 report on the status of the National Science Foundation's
and operations of science and (NSF) major facilities projects, the Large Hadron Collider High Luminosity
engineering research infrastructure Upgrade program began construction, and it along with the four other major
such as telescopes and research facilities projects in construction (see figure), have weathered schedule delays
vessels. These projects include major associated with the COVID-19 pandemic. To partially account for increased costs
facilities that cost over $100 million to associated with the pandemic, such as the cost of paying project staff while work
construct or acquire, and mid-scale is paused, NSF has authorized $38.9 million in total project cost increases to the
research infrastructure projects. Over award recipients constructing three of the five projects:
the past 5 fiscal years, NSF has
received over $1 billion in • $18.9 million for the Daniel K. Inouye Solar Telescope,
appropriations for these projects. Prior • $10.0 million for the Vera C. Rubin Observatory, and
GAO reports reviewed NSF’s oversight • $10.0 million for Regional Class Research Vessels.
of the projects, its cost estimating and
schedule policies, and the project Because the pandemic is ongoing and its full effects are not yet known, NSF
management expertise of its oversight expects to make further adjustments to the cost and schedule of all five major
workforce. projects in construction. Design work on an additional major facility project
continued without significant interruption from the pandemic. Further, NSF made
Senate Report 115-275, Senate Report
114-239, and House Report 114-605
awards to begin the agency's first three mid-scale research infrastructure
included provisions for GAO to review projects.
and report annually on projects funded National Science Foundation Major Facilities Projects in Construction
from NSF’s Major Research Equipment
and Facilities Construction account.
This report, the fourth, examines (1)
the cost and schedule performance of
NSF’s ongoing major facilities and mid-
scale research infrastructure projects
and (2) the extent to which NSF has
implemented prior GAO
recommendations related to its
management of major facilities. GAO
reviewed NSF and award recipient
documents for the projects. GAO
examined policies and procedures to
identify efforts to implement
recommendations and interviewed
NSF officials for clarifying information.
What GAO Recommends
NSF agreed with and has taken steps
to address three remaining NSF has fully implemented GAO’s prior recommendation on information sharing
recommendations from GAO’s prior among award recipients and has drafted guidance or taken other steps towards
work to improve the project addressing GAO’s three remaining recommendations. To enhance information
management skills of its staff and sharing among award recipients, NSF added a section to its terms and conditions
award recipients and to align schedule in its major facilities agreements that encourages awardees to share information
guidance to GAO’s best practices. among awardees and participate in a knowledge management program.
View GAO-21-417. For more information,
contact Candice Wright at (202) 512-6888 or
WrightC@gao.gov.
United States Government Accountability Office
Contents
Letter 1
Background 3
NSF Experienced Cost or Schedule Increases on Five of Its Nine
Major Projects 10
NSF Implemented a Recommendation on Information Sharing and
Took Steps to Address Others 16
Agency Comments 20
Appendix I Summaries of the National Science Foundation’s Major Facilities
Projects under Construction 21
Appendix II Summary of the National Science Foundation’s Future Major Facilities
Project in Design 33
Appendix III GAO Contact and Staff Acknowledgments 36
Tables
Table 1: Appropriations for NSF Major Research Equipment and
Facilities Construction 2
Table 2: Completion Dates and Additional Estimated Delays to
NSF Major Facilities Projects in Construction, as of March
2021 12
Table 3: Not-to-Exceed Cost Changes for NSF Major Facilities
Projects in Construction, March 2021 14
Table 4: NSF Mid-scale Research Infrastructure Projects, March
2021 15
Figures
Figure 1: Organization of NSF Oversight of Major Research
Equipment and Facilities 6
Figure 2: Total Project Cost in Relation to the Not-To-Exceed
Award Amount for NSF Construction Projects 10
Page i GAO-21-417 National Science Foundation
Abbreviations
AIMS Antarctic Infrastructure Modernization for Science
ATLAS A Toroidal Large Hadron Collider Apparatus
CERN European Organization for Nuclear Research
CMS Compact Muon Solenoid
COVID-19 Coronavirus Disease 2019
DKIST Daniel K. Inouye Solar Telescope
DOE Department of Energy
HL-LHC Large Hadron Collider High Luminosity Upgrade
LCCF Leadership Class Computing Facility
LHC Large Hadron Collider
LSST Large Synoptic Survey Telescope
MREFC Major Research Equipment and Facilities Construction
NSF National Science Foundation
OMB Office of Management and Budget
RCRV Regional Class Research Vessels
This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.
Page ii GAO-21-417 National Science Foundation
Letter
441 G St. N.W.
Washington, DC 20548
June 8, 2021
The Honorable Jeanne Shaheen
Chair
The Honorable Jerry Moran
Ranking Member
Subcommittee on Commerce, Justice, Science,
and Related Agencies
Committee on Appropriations
United States Senate
The Honorable Matt Cartwright
Chairman
The Honorable Robert Aderholt
Ranking Member
Subcommittee on Commerce, Justice, Science,
and Related Agencies
Committee on Appropriations
House of Representatives
The National Science Foundation (NSF) supports the design,
construction, and operations of various research infrastructure projects,
which are research tools such as telescopes and research vessels.
Research infrastructure projects include both major facilities that cost
over $100 million to build and mid-scale research infrastructure projects
that cost between $4 million and $100 million to build. 1 These projects are
designed, constructed, and operated in collaboration with the scientific
community. The longevity of these projects—some may operate for 50
years—and complexity of their construction or acquisition heighten the
need for NSF to provide rigorous oversight.
NSF uses cooperative agreements and contracts to fund and oversee the
projects throughout their life cycles. Award recipients of cooperative
agreements and contracts—which may include universities, nonprofit
associations, and companies—manage the projects’ day-to-day activities.
1Mid-scale research infrastructure projects are those with a total project cost above the
upper limit for NSF's Major Research Instrumentation program and below the threshold for
a major facility. American Innovation and Competitiveness Act (AICA), Pub. L. No. 114-
329, § 109(b)(4), 130 Stat. 2969, 2988 (2017). Major facilities are those that cost over
$100 million to construct regardless of the funding account. William M. (Mac) Thornberry
National Defense Authorization Act for Fiscal Year 2021, Pub. L. No. 116-283, § 267, 134
Stat. 3388, 3502 (amending section 110 of the AICA).
Page 1 GAO-21-417 National Science Foundation
NSF generally funds these construction efforts through its Major
Research Equipment and Facilities Construction (MREFC) account. Over
the past 5 fiscal years, NSF has received over $1 billion in appropriations
to this account, as shown in table 1. In fiscal year 2021, Congress
appropriated $241 million for the MREFC account.
Table 1: Appropriations for NSF Major Research Equipment and Facilities
Construction
Dollars in millions
Fiscal year
2017 2018 2019 2020 2021
Appropriations 209 183 296 243 241
Source: National Science Foundation (NSF) and congressional budget documents. | GAO-21-417
Since 2018, we have issued three reports focused on NSF’s oversight of
cost and schedule performance of the agency’s major facilities
construction projects and on NSF’s efforts to ensure the project
management expertise of NSF oversight staff and award recipients for its
major facilities projects. 2 In addition, we also reported on how NSF shares
lessons learned and best practices for the construction of major facilities
projects. We made a total of six recommendations to improve NSF’s
ability to provide oversight over the design and construction of these
major facilities projects. NSF concurred with our recommendations in
these past reports. In April 2020, we reported that NSF had implemented
two of the six recommendations.
Senate Report 115-275, Senate Report 114-239, and House Report 114-
605 included provisions for GAO to review and report on projects funded
from the NSF MREFC account. This report, the fourth of our annual
reports in response to the Senate and House report provisions, (1)
describes the cost and schedule performance of NSF’s ongoing major
facilities and mid-scale research infrastructure projects and (2) assesses
2GAO, National Science Foundation: Revised Policies on Developing Costs and
Schedules Could Improve Estimates for Large Facilities, GAO-18-370, (Washington, D.C.:
June 1, 2018); GAO, National Science Foundation: Cost and Schedule Performance of
Large Facilities Construction Projects and Opportunities to Improve Project Management,
GAO-19-227, (Washington, D.C.: Mar. 27, 2019); GAO, National Science Foundation:
Cost and Schedule Performance of Major Facilities Construction Projects and Progress on
Prior GAO Recommendations, GAO-20-268, (Washington, D.C.: April 3, 2020).
Page 2 GAO-21-417 National Science Foundation
the extent to which NSF has implemented prior GAO recommendations
related to its management of major facilities.
To describe the cost and schedule performance of NSF’s major facilities
projects, we reviewed NSF and award recipient documents that detailed
project cost, schedule, and risks for each major facility. We also reviewed
initial documents on NSF’s mid-scale research infrastructure projects.
Additionally, we interviewed NSF officials to obtain clarifying information
about these projects and NSF’s oversight.
To assess the extent to which NSF has implemented the four
recommendations from two prior GAO reports on NSF major facilities that
had not been implemented as of April 2020, we reviewed NSF policies
and procedures as well as documentation related to actions the agency
took to implement these recommendations. 3 We also interviewed NSF
officials to discuss NSF’s ability to implement the recommendations, as
well as their estimated time frames for implementation.
We conducted this performance audit from September 2020 to June 2021
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Background
Stages in the Life Cycles Each major facility and mid-scale research infrastructure project has a
of NSF’s Major Research sponsoring organization from within NSF’s seven research directorates. 4
Equipment and Facilities The sponsoring organization assesses the scientific merit of a potential
project, proposes projects for funding through NSF’s MREFC account,
Projects and is responsible for overseeing the project during the following five
stages of its life cycle.
• Development. Initial project ideas emerge; and a broad consensus is
built within the relevant scientific community for the potential long-term
3GAO-18-370 and GAO-19-227.
4NSF is divided into the following seven research directorates that support science and
engineering research and education: biological sciences; computer and information
science and engineering; engineering; geosciences; mathematical and physical sciences;
social, behavioral, and economic sciences; and education and human resources.
Page 3 GAO-21-417 National Science Foundation
needs, priorities, and general requirements for research infrastructure
that NSF may consider funding.
• Design. Entrance into this stage occurs when the NSF Director
approves the proposed research infrastructure as a national priority
and the sponsoring directorate makes an award (either through a
cooperative agreement or contract) for developing detailed project
cost, scope, and schedule for possible construction. This stage is
divided into conceptual, preliminary, and final design phases. A
candidate project will exit the design stage and enter the construction
stage after a successful review by the NSF director and other key
stakeholders of its project execution plan and authorization of its not-
to-exceed total project cost by the National Science Board, as
discussed below.
• Construction. The construction stage begins when NSF makes
awards to external recipients for acquisition or construction of
research infrastructure. 5 The construction stage ends after final
delivery and acceptance of the defined scope of work and facility
performance per terms of the award instrument.
• Operations. The operations stage includes the day-to-day work
necessary to operate and maintain the research infrastructure
(including refurbishment or upgrade activities) and to perform
research.
• Divestment. Divestment can include the transfer of the research
infrastructure to another entity’s operational and financial control or
the decommissioning of the research infrastructure, including its
complete deconstruction and removal. NSF generally decides to
divest when the agency or the scientific community determines that
5Awards generally take the form of cooperative agreements, although NSF occasionally
uses contracts, according to agency officials. The policies and procedures in NSF’s Major
Facilities Guide apply to research infrastructure projects regardless of the award
instrument employed. In addition, cooperative agreements with universities, consortia of
universities, or nonprofit organizations are governed by the Office of Management and
Budget’s (OMB) Uniform Administrative Requirements, Cost Principles, and Audit
Requirements for Federal Awards (Uniform Guidance). 78 Fed. Reg. 78,590 (Dec. 26,
2013) (codified as amended at 2 C.F.R. pt. 200). In December 2014, NSF and other
federal awarding agencies issued a joint interim final rule to implement this Uniform
Guidance. 79 Fed. Reg. 75,871 (Dec. 19, 2014). NSF received approval from OMB to
implement the Uniform Guidance using a policy rather than a regulation. Acquisitions by
contract of supplies or services by and for the use of the federal government are governed
by the Federal Acquisition Regulation. See 48 C.F.R. §§ 1.104, 2.101(b); see also chapter
25 of title 48 of the Code of Federal Regulations for NSF-specific provisions. According to
NSF’s Major Facilities Guide, contracts with nonprofit and educational institutions are also
governed by the Uniform Guidance.
Page 4 GAO-21-417 National Science Foundation
the facility is no longer considered an operational priority with regard
to advancing science, according to NSF’s Major Facilities Guide.
NSF funding for the development, design, operations, and divestment
stages generally comes from the sponsoring directorate. Funding for the
construction stage generally comes from the MREFC account.
NSF Oversight of Major NSF has established an oversight structure for major facilities projects
Research Equipment and that includes organizations from across the agency (see fig. 1). This
includes the National Science Board, a policy and advisory body that is
Facilities Projects
part of NSF and consists of the NSF Director and 24 members, drawn
from industry and universities, who represent a variety of science and
engineering disciplines. The NSF Office of the Director and the National
Science Board provide high-level, ongoing oversight of major facilities
projects, including the approval of new projects to be included in NSF’s
annual budget request.
Page 5 GAO-21-417 National Science Foundation
Figure 1: Organization of NSF Oversight of Major Research Equipment and Facilities
Note: Figure does not include all NSF organizations or interactions between them and includes only
the major facilities projects and mid-scale research infrastructure projects in design or construction at
the time of GAO’s review.
Page 6 GAO-21-417 National Science Foundation
Within NSF’s Office of Budget, Finance, and Award Management, the
Large Facilities Office (1) develops business-related oversight policies for
all life-cycle stages, with a focus on the design and construction stages,
and (2) provides assistance on nonscientific and nontechnical aspects of
project planning, budgeting, implementation, and management. To that
end, the office maintains the Major Facilities Guide, which contains NSF
policies for agency staff and recipients on the planning, management, and
oversight of major facilities. The guide also tailors the policies for
application to mid-scale research infrastructure projects.
NSF also uses external panels of experts to review projects at several
points during their life cycles. An external panel may first review a project
proposal during the development stage. Separate panels then review the
project at the culmination of each of its design phases. In addition, an
external panel periodically reviews each project during both construction
and operations; according to NSF officials, those reviews are generally
annual.
Construction Costs and Under NSF’s major facilities construction process, the recipients of design
Schedules of Major awards develop construction cost and schedule estimates for projects
and submit them to NSF for review. In particular, after a project’s final
Facilities Projects
design review, the National Science Board authorizes a not-to-exceed
award amount and duration. The not-to-exceed award amount that the
National Science Board authorizes is the amount against which NSF
measures cost increases to implement its no cost overrun policy.
NSF’s Major Facilities Guide defines the following components, which
together make up the total project cost and schedule for the construction
of major facilities projects. The total project cost awarded in a project’s
construction agreement may be less than the not-to-exceed cost but it is
not to exceed it. These components of the total project cost and schedule
include the following:
• Performance measurement baseline. During design, the scope,
cost, and schedule are refined and eventually become the project
baseline. Once the baseline has been authorized and included in a
construction award, it is known as the performance measurement
baseline. NSF documents the performance measurement baseline in
the terms and conditions of the award instrument and requires that
any changes to it be made through a formal change control process.
The performance measurement baseline does not include the
project’s budget or schedule contingency.
Page 7 GAO-21-417 National Science Foundation
• Contingency. This is an amount of budget or time for covering the
cost increases or delays that would result if foreseen project risks
were to occur, such as price changes of goods in future years. During
development of a total project cost estimate, the timing and impacts of
such risks are uncertain. As a project progresses, the impacts of risks
that materialize may exceed the cost or schedule in the performance
measurement baseline and lead to use of the project’s budget or
schedule contingency. 6 The amount of contingency needed for a
project is typically estimated using statistical analysis or judgment
based on past project experience. According to NSF’s Standard
Operating Guidance on budget contingency, it is likely that no
contingency will be left over by the end of a project because all of it
will have been used during normal execution of the project to manage
known risks and uncertainties. NSF approval is needed when use of
contingency exceeds certain project-specific thresholds, which are
described in the project’s execution plan and codified in the award.
In this report, we identify total project costs for the construction of major
facility and mid-scale research infrastructure projects, which were
developed during the design phase based on the latest estimates
available from NSF officials; those estimates are subject to change before
construction awards are made. For projects under construction, we
identified total project costs based on the amounts awarded in the
cooperative support agreements for construction and the not-to-exceed
amount authorized by the National Science Board. Only at the end of the
project—when construction is complete and the awards have been closed
out—will the final total project costs be known.
In addition to the performance measurement baseline and budget
contingency, a project’s not-to-exceed cost that the National Science
Board authorized may include the following:
• Fee. NSF may provide recipients the opportunity to earn a fee for
major facilities projects.
6Use of budget contingency is governed by OMB’s Uniform Guidance. See 2 C.F.R. §
200.433. OMB’s Uniform Guidance and NSF’s Standard Operating Guidance on budget
contingency define contingency as that part of a budget estimate of future costs (typically
of large construction projects, information technology systems, or other items as approved
by the federal awarding agency) which is associated with possible events or conditions
arising from causes the precise outcome of which is indeterminable at the time of
estimate, and that experience shows will likely result, in aggregate, in additional costs for
the approved activity or project. Amounts for major project scope changes, unforeseen
risks, or extraordinary events may not be included.
Page 8 GAO-21-417 National Science Foundation
• Management reserve. NSF, not the award recipient, holds
management reserve to manage budget uncertainties, unforeseeable
events, and risks not manageable by the recipient.
NSF’s No Cost Overrun Since February 2008, NSF has had a policy to manage cost overruns on
Policy for Major Facilities major facilities projects. 7 Under this policy, the cost estimate developed at
Projects the preliminary design review should have adequate contingency to cover
all foreseeable risks. 8 Any cost increases not covered by contingency are
generally to be accommodated by reductions in scope. 9 Figure 2 provides
a breakdown of the total project cost and its relation to the not-to-exceed
award amount.
7See GAO-18-370 for additional details on the history of this policy.
8According to the September 2019 update to NSF’s Major Facilities Guide, while the
policy requires that the total project cost estimate established following the preliminary
design review have adequate contingency to cover all foreseeable risks, NSF will conduct
oversight of major facilities projects against the total project cost authorized by the NSB
following final design review.
9These reductions in scope differ from re-planning actions on a project. NSF’s Major
Facilities Guide defines re-planning as a normal project management process to modify or
re-organize the performance measurement baseline cost and/or schedule plans for future
work without impacting total project cost, project end date, or overall scope objectives or
the implementation of approved de-scoping options.
Page 9 GAO-21-417 National Science Foundation
Figure 2: Total Project Cost in Relation to the Not-To-Exceed Award Amount for NSF Construction Projects
Note: Figure does not include other components of the not-to-exceed award amount that the National
Science Board may authorize, such as fees or management reserves.
Since our April 2020 report, NSF has continued construction on four
NSF Experienced major facility projects—the Daniel K. Inouye Solar Telescope (DKIST), the
Cost or Schedule Vera C. Rubin Observatory (Rubin Observatory), Regional Class
Research Vessels (RCRV), and Antarctic Infrastructure Modernization for
Increases on Five of Science (AIMS)—and begun construction of a fifth, the Large Hadron
Its Nine Major Collider High Luminosity Upgrade (HL-LHC) program. The Coronavirus
Disease 2019 (COVID-19) pandemic, however, caused unexpected
Projects delays to each of the projects in construction. As a result of the pandemic
and other factors, all five of these projects have experienced schedule
delay and may cost more than initially budgeted. NSF’s four other
projects, an existing major facility project in design and three newly
announced mid-scale projects proceeded without significant interruption.
Page 10 GAO-21-417 National Science Foundation
All Five Projects in The COVID-19 pandemic has caused unexpected delays to NSF’s five
Construction Had projects that are in construction to varying degrees depending on the
geographic location, level of completion, and technical nature of the
Schedule Delays and
project. NSF considers all five major facility projects that were in
Several Have Requested construction to be behind schedule and intends to increase authorized
Additional Funds not-to-exceed total project costs and supplement awards, or re-evaluate
project scope, if necessary, to account for the pandemic’s effects. Given
Travel and Quarantine Restrictions the pandemic’s on-going domestic and global impacts to personnel and
Enacted Because of the COVID-19
Pandemic Affected Construction of NSF’s
supply chains, NSF officials expect that all of its major facility projects
Major Facilities Projects currently in construction will be extensively re-planned or completely re-
Travel restrictions or quarantines caused baselined to account for the effects of the pandemic. 10
significant work stoppages at three major
facilities projects.
• At Daniel K Inouye Solar Telescope,
Travel restrictions and social distancing requirements during the
Hawaii’s travel restrictions prevented staff pandemic have affected each of the projects in construction (see
from reaching the telescope for sidebar). However, these delays have generally not yet been fully
commissioning activities.
• At the Vera C. Rubin Observatory, the
incorporated into the projects’ performance measurement baselines,
Chilean government’s quarantine because the pandemic is ongoing. As of March 2021, two of the five
measures resulted in work stoppage projects in construction have had their schedules extended (see table 2).
between March and September 2020.
NSF officials approved a schedule extension of 18 months for DKIST and
• On Antarctic Infrastructure Modernization
for Science, NSF suspended work at the a schedule extension of 6 months for RCRV to address COVID-related
project site to avoid the possibility of and other delays. For the three other projects in construction, NSF has
introducing COVID-19 to Antarctica. This
stoppage may continue for another
not yet adjusted the schedules because these projects are earlier in their
season, effectively halting all construction construction and not in immediate need of schedule extensions. Officials
for two years. estimate that the HL-LHC is currently approximately 7 months behind
Source: GAO analysis of National Science Foundation (NSF)
information. | GAO-21-417 schedule, and the Rubin Observatory is 10 months behind schedule.
Having lost a construction season, AIMS is also running behind schedule.
10NSF defines re-planning as a normal project management process to re-organize
without impacting the total project cost, schedule or scope. Re-baselining is a re-planning
that results in a change that is outside the terms set forth in the award for either the total
project cost, the project duration, or the project scope that is not in the approved options
for scope management.
Page 11 GAO-21-417 National Science Foundation
Table 2: Completion Dates and Additional Estimated Delays to NSF Major Facilities
Projects in Construction, as of March 2021
Current
scheduled Additional
Social Distancing Restrictions Imposed Original completion estimated
Because of the COVID-19 Pandemic Project Name completion date date delaya
Affected Construction of NSF’s Major
Facilities Projects Daniel K Inouye Solar Telescopeb December 2017 December 2021 0 months
Social distancing requirements, which Vera C. Rubin Observatoryc October 2022 October 2022 10 monthsc
reduced the number of employees that could
work simultaneously, affected four major Regional Class Research Vessels July 2024 January 2025 No estimate
facilities projects. Large Hadron Collider High December 2026 December 2026 7 months
• On Regional Class Research Vessels, Luminosity Upgrade
fewer shipyard workers could be on site
Antarctic Infrastructure January 2028 January 2028 No estimatec
to work on the research vessels.
Modernization for Sciencec
• On the Large Hadron Collider High
Luminosity Upgrade, fewer research Source: GAO analysis of National Science Foundation (NSF) information. | GAO-21-417
employees could conduct lab work a
Estimated delays are based on NSF projections and are not incorporated into the projects current
necessary to advance upgrades to the performance measurement baseline
detector components. b
This project was re-baselined in 2014. Before the Coronavirus Disease 2019 pandemic, its
• At the Daniel K Inouye Solar Telescope scheduled completion date was June 2020. In May 2021, NSF updated its project schedule to reflect
and the Vera C. Rubin Observatory, December 2021 as the completion date.
social distancing requirements limited the c
Re-baseline expected in late 2021.
number of individuals on site for
construction, integration, and testing. In some cases, NSF officials expect that re-baselining will be necessary
Source: GAO analysis of National Science Foundation (NSF)
information. | GAO-21-417
to develop a reliable completion schedule for projects in construction. For
example, NSF officials expect to re-baseline the Rubin Observatory later
in 2021 and develop a new completion date at that time. Shutdown of
work at the Rubin Observatory’s construction site in Cerro Pachón, Chile,
stopped progress on two significant components, the dome that will
house the telescope and the telescope mount assembly that moves and
stabilizes it. Work at McMurdo Station in Antarctica for the AIMS project
was suspended in March 2020 and has not yet resumed. Construction
activities in Antarctica are concentrated over the austral summer with
work beginning in October and concluding in March. With the pandemic
and its effects ongoing, construction work at the site may not resume for
the 2021-2022 season, according to NSF officials. A new estimated
completion date for AIMS will depend on how the project is re-baselined,
including a re-evaluation of the project scope. On April 30, 2021, NSF
determined that it would not be funding additional components of AIMS
due to the significant impacts of the pandemic. Current components of
AIMS include the Vehicle Equipment and Operations Center, the first
lodging unit, and associated utilities.
In addition to the pandemic, other factors contributed to schedule delay
since our April 2020 report. Major facility projects try to identify
foreseeable delays and other schedule risks during design and include
Page 12 GAO-21-417 National Science Foundation
schedule contingency to account for these risks as part of the project
management process. For example, the DKIST project site used
schedule contingency to account for time lost due to an adverse weather
event at its construction site in Hawaii, and the RCRV project site in
Houma, Louisiana used scheduled contingency when it experienced
shipyard performance challenges. However because the pandemic was
unforeseen a project’s remaining schedule contingency may not
accurately reflect its total schedule risk.
The delays and inefficiencies caused by the pandemic will have cost
impacts on the projects under construction as well. Some construction
costs, such as those associated with project staff waiting for work to
resume, continue to accrue even if no or little construction can be
performed. Other costs are new, such as the leasing of additional storage
space for materials waiting to be shipped to a project site and time added
to tasks as a result of social distancing. For example, on HL-LHC NSF
officials estimated that the current cumulative effects of COVID-19 totaled
between $2 million and $3 million dollars as of January 2021.
NSF’s major facilities projects generally use their budget contingency
funds to compensate for changes in expected costs resulting from known
project risks. Such contingency is built into the total project cost and is
estimated at the start of construction. Consistent with NSF policy, NSF
does not plan to use budget contingency to offset cost increases related
to the pandemic. For unforeseen events beyond the award recipient’s
control—like the pandemic—NSF’s Major Facilities Guide allows for the
use of management reserve, which is an authorized amount of money in
addition to a project’s baseline estimate and contingency to address the
additional work associated with unforeseen events.
To address costs associated with the pandemic, NSF has sought
authorization through the National Science Board or used delegated
authority to create management reserves. Prior to the pandemic, none of
the projects under construction had management reserves. As of March
2021, however, three projects were authorized management reserves. 11
To create management reserves, the National Science Board and NSF
increased the not-to-exceed amounts cumulatively by $38.9 million. The
not-to-exceed cost for DKIST has increased by $18.9 million, and the not-
11Since NSF does not normally have management reserves, they must be identified by
NSF from available MREFC funds, if needed in the near-term. Long-term additional funds
may be appropriated by Congress.
Page 13 GAO-21-417 National Science Foundation
to-exceed costs for both the Rubin Observatory and RCRV have
increased by $10 million (see table 3). 12 NSF officials’ adjustments to
account for the pandemic were ongoing as of March 2021 because the
full effects of the pandemic are not yet known. NSF officials told us they
expect to seek authorization for additional reserves through the National
Science Board to account for the unexpected costs the pandemic has
created.
Table 3: Not-to-Exceed Cost Changes for NSF Major Facilities Projects in
Construction, March 2021
Dollars in millions
Original Increase in not-to- Current
authorized not- exceed cost as a result authorized not-
to-exceed cost of management to-exceed cost
Project name reserves
Daniel K. Inouye 297.9 18.9 363.0a
Solar Telescopea
Vera C. Rubin 473.0 10.0 483.0
Observatoryb
Regional Class 365.0 10.0 375.0
Research Vessels
Large Hadron Collider 153.0 0 153.0
High Luminosity
Upgradec
Antarctic 410.4 0 410.4
Infrastructure
Modernization for
Scienceb
Total 1,699.3 38.9 1,784.4
Source: GAO analysis of National Science Foundation (NSF) information. | GAO-21-417
a
DKIST has been re-baselined since its original award, so the original authorized not-to-exceed cost
and the increase in not-to-exceed cost as a result of management reserves will not total to the current
authorized not-to-exceed cost. Before the Coronavirus Disease 2019 (COVID-19) pandemic, the
authorized total project cost was $344.1 million.
b
NSF officials expect to re-baseline this project in late 2021.
c
This project is in its first year of construction, so NSF officials may adjust its cost to reflect the effects
of the COVID-19 pandemic at a later time.
12NSF was able to authorize $14.1 million in additional management reserve for RCRV
because the project’s construction award was below its not-to-exceed amount.
Considering this, total authorized management reserve was about $43 million as of March
2021.
Page 14 GAO-21-417 National Science Foundation
Work Continued on One A major facility project in design and newly announced mid-scale
Project in Design and NSF research infrastructure projects proceeded without significant interruption.
The Leadership Class Computing Facility (LCCF), which is the only major
Announced Three New
facility project in the design stage, underwent conceptual design review in
Projects June 2020. LCCF’s total project cost is still under development. Under
NSF policy, a major facility’s total project cost is not final until after the
final design review when the National Science Board authorizes a not-to-
exceed cost and an award duration for construction.
Additionally, in 2020 the National Science Board authorized the first mid-
scale research infrastructure projects from the MREFC account and,
following NSF review, awarded funds to three projects. NSF sought
proposals for potential projects with total project costs between $20
million and $70 million. 13 The three new projects funded by NSF have a
combined total cost of $125.1 million and scheduled completion dates in
2025 (see table 4).
Table 4: NSF Mid-scale Research Infrastructure Projects, March 2021
Authorized
award amount Scheduled
(dollars in completion
Project Name Awardee Project description millions) date
Distributed Energy University of To address the long-term challenges of integrating 39.5 October 2025
Resources Connect California, renewable and distributed energy resources into the
(DERConnect) San Diego power grid, DERConnect aims to establish a large-scale
grid-connected experimental research facility. This testbed
will also help to develop a new generation of workforce to
successfully modernize the power grid.
Global Ocean Monterey Bay To better study and monitor the effects of climate change 52.9 October 2025
Biogeochemistry Aquarium on the oceans, this project aims to construct 500
Array Research biogeochemical Argo floats with oxygen, nitrate, pH, and
Institute other sensors to be released into the ocean to collect
chemical and biological data at depths of over a mile. The
network would deliver data in real-time to an established
global data system, where it would be freely available at
no cost.
13Since the initial project solicitation, NSF has increased the upper limit for a mid-scale
research infrastructure project to $100 million to align with the definition of a major facility
project as amended by the William M. (Mac) Thornberry National Defense Authorization
Act for Fiscal Year 2021. NSF received 48 initial proposals and chose award recipients
from 11 full proposals.
Page 15 GAO-21-417 National Science Foundation
Authorized
award amount Scheduled
(dollars in completion
Project Name Awardee Project description millions) date
High Magnetic Field Cornell To enable new science with X-rays, this project will 32.7 December 2025
Beamline University establish a High Magnetic Field Beamline. The beamline
can be used to observe the underlying correlations and
symmetries of new phases of matter induced by high
magnetic fields and obtain insights into electronic
symmetry breaking, unconventional superconductivity, and
quantum magnetism.
Source: GAO analysis of National Science Foundation (NSF) information. | GAO-21-417
NSF issued guidance to improve its process for sharing information and
NSF Implemented a best practices among major facilities projects, which fully implements a
Recommendation on recommendation we made in March 2019. In addition, NSF has taken
preliminary steps such as drafting guidance to address three other
Information Sharing recommendations from our June 2018 and March 2019 reports related to
and Took Steps to its management of its major facilities. 14 Once NSF completes these steps,
we will evaluate its actions to determine whether they are sufficient to fully
Address Others address our recommendations.
NSF Issued Guidance to NSF has updated the terms and conditions of its major facilities
Improve Information agreements to improve collection and sharing of lessons learned and best
practices among major facilities project teams. This action addresses a
Sharing Among Awardees
recommendation in our March 2019 report where we recommended that
NSF ensure, through a requirement or other means, that award recipients
for major facilities projects provide information to NSF on any lessons
learned or best practices. 15 At that time, we found that NSF had a process
to share lessons learned but provided inconsistent direction to projects on
how to participate in this process. For example, the RCRV project had a
requirement in its award terms and conditions to annually share lessons
learned, but the Rubin Observatory’s award did not.
In our April 2020 report, we found that NSF had drafted preliminary award
terms and conditions that would require all recipients to participate in
NSF’s program and that they planned to finalize and publish this guidance
14GAO-18-370, GAO-19-227.
15The process for identifying and sharing lessons learned on major facilities projects,
which NSF refers to as its Knowledge Management program, was designed to respond to
a 2015 recommendation by the National Academy of Public Administration and to the
American Innovation and Competitiveness Act’s requirements that NSF coordinate the
sharing of best management practices and lessons learned from major facilities projects.
Page 16 GAO-21-417 National Science Foundation
in fiscal year 2020. 16 NSF implemented the updated terms and conditions
in all its major facilities agreements on October 5, 2020, thereby fully
addressing the recommendation. The updated terms encourage all
awardees to share information on lessons learned and best practices and
participate in NSF’s annual Major Facilities Workshop. 17 According to
NSF officials, the agency chose not to make this a requirement because
other agencies running similar programs found that making participation
voluntary yields more meaningful results. The most recent Major Facilities
Workshop included representatives from all current major facilities.
NSF Has Taken Additional NSF has also taken additional steps since our April 2020 report to
Steps to Address the address the three remaining recommendations from our June 2018 and
March 2019 reports, but has not fully implemented them. 18
Three Remaining
Recommendations Policies for developing project schedules. NSF is in the process of
Related to Its Oversight of issuing guidance that could improve the accuracy and reliability of project
Major Facilities schedules. In our June 2018 report, we found that the majority of NSF’s
scheduling policies did not fully align with GAO’s best practices for
scheduling which could lead to increased costs and schedule delays. We
recommended that NSF revise its policies for developing schedules for
major facilities projects, and for reviewing those schedules, to better
incorporate best practices in GAO’s schedule guide. 19
In April 2020, we reported that NSF had updated internal guidance
documents to include an analysis of project schedules by the Large
Facilities Office as part of the NSF cost analysis process. 20 In addition, we
reported that NSF planned to include additional guidance on scheduling
in its Major Facilities Guide. The guide contains NSF policy on planning
and managing major facilities, including required policies and procedures
16GAO-20-268.
17NSF hosts its major facilities workshop annually to provide a collaborative forum for
continuous learning and information sharing among participants. During the COVID-19
pandemic, NSF transitioned to quarterly virtual webinars. Based on this experience, NSF
may use a combination of an in-person workshop and webinars in the future, according to
NSF officials.
18GAO-18-370, GAO-19-227.
19GAO’s schedule guide develops the scheduling concepts introduced in our cost
estimating guide and presents them as 10 best practices associated with developing and
maintaining a reliable, high-quality schedule. See GAO, Schedule Assessment Guide:
Best Practices for Project Schedules, GAO-16-89G (Washington, D.C.: December 2015).
20GAO-20-268.
Page 17 GAO-21-417 National Science Foundation
and related guidance. According to NSF, they revised their pre-award
internal guidance to evaluate and document that award recipients develop
their schedule estimates in alignment with the four GAO characteristics of
a high-quality schedule (comprehensive, well-constructed, credible, and
controlled).
In February 2021, NSF released a draft for public comment of the Major
Facilities Guide that requires award recipients to use GAO's scheduling
guide when developing and managing project schedules (a requirement
added in 2017) and provides additional guidance for implementation of
the GAO guide on NSF-funded projects. According to NSF officials, they
expect to issue the updated guide by the end of fiscal year 2021. Once
issued, we will review the final version of the guide to determine if the
recommendation was met.
Project management competencies of NSF’s major facilities
oversight workforce. NSF has taken steps to evaluate the project
management competencies of its oversight staff and is considering
additional actions. The American Innovation and Competitiveness Act
directed NSF to determine what project management and financial
management expertise NSF staff need to effectively oversee major
facilities projects. However, in our March 2019 report, we found that NSF
had not assessed NSF major facilities oversight staff for potential
competency gaps in these areas of expertise nor taken steps to close any
such gaps. We recommended that NSF assess the agency’s oversight
workforce to identify any competency gaps, develop a plan to address
them and time frames for doing so, and monitor progress in closing them.
In April 2020, we reported that NSF had hired a contractor to conduct a
proficiency assessment and workforce gap analysis. 21 Since then, the
contractor has helped NSF develop a competency model for its staff and
complete surveys of NSF staff who oversee major facilities projects to
assess their competencies in project management and other professional
and technical areas. NSF’s contractor used three surveys to identify any
competency gaps. Employees rated their own proficiency in
competencies, supervisors rated the proficiency of their employees, and
supervisors rated the expected minimum proficiency levels for a
21In addition to responding to our recommendation, this analysis is part of NSF's
implementation of the Program Management Improvement Accountability Act. Pub. L. No.
114-264, § 2(b)(1), 103 Stat.1371, 1372 (2016) (codified as amended at 31 U.S.C. §
1126).
Page 18 GAO-21-417 National Science Foundation
hypothetical, experienced employee. Using data from these surveys, the
contractor found that certain individuals had gaps in award management,
budget management, financial cost/analysis, project management, and
risk and opportunity management, among others. However, they found
that NSF’s project teams as a single group possessed the competencies
expected. 22
NSF is now planning how to address the identified gaps. According to
NSF officials, supervisors have used the survey results to help staff
create individualized training plans. In addition, NSF commissioned a
study of its training capabilities. In January 2021, the study identified
areas where NSF does not provide the training in-house that employees
need to reach some mid-level proficiencies and most expert level
proficiencies and made recommendations to improve the agency’s
internal and external training opportunities. For example, according to the
study, the agency did not provide sufficient training opportunities in areas
such as award management and risk and opportunity management. The
study recommended that NSF develop new courses and leverage training
resources at other agencies. According to NSF officials, the agency is
now taking steps to implement the recommendations and to determine
how it will monitor staff competencies, such as by performing regular self-
assessment surveys. We will continue to review NSF’s actions as it works
to identify, address, and monitor competency gaps.
Project management expertise of award recipients for major
facilities projects. NSF is in the process of issuing guidance that could
improve the project management expertise of its major facilities award
recipients. In our 2019 report, we recommended that NSF establish
criteria for the project management expertise of major facilities project
recipients and incorporate the criteria in project requirements and external
panel reviews. In response, NSF identified criteria that award recipients
must be able to demonstrate, but these criteria will not be fully integrated
into NSF’s oversight practices until the end of fiscal year 2021.
As we reported in April 2020, NSF established criteria for expertise in
project management, program management, earned value management,
risk management, cost estimating, and other areas in draft updates to its
Major Facilities Guide. As discussed above, the draft guide was released
for public comment in February 2021 and NSF expects to issue the guide
22NSF's implementation of the Program Management Improvement Accountability Act
looks for the three core members of the NSF Integrated Project Team to hold the
necessary competencies for award oversight, as opposed to a single individual.
Page 19 GAO-21-417 National Science Foundation
by the end of fiscal year 2021. Further, according to NSF officials,
requirements for expert panels to assess recipients’ project management
expertise will be included in internal guidance they plan to complete in the
third quarter of fiscal year 2021. Once issued, we will review the final
versions of these documents to determine if the recommendation was
met.
We provided a draft of this report to NSF for review and comment. NSF
Agency Comments provided technical comments, which we incorporated as appropriate.
We are sending copies of this report to the appropriate congressional
committees and the Director of the NSF. In addition, the report is
available at no charge on the GAO website at https://www.gao.gov.
If you or your staff have any questions about this report, please contact
me at (202) 512-6888 or WrightC@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made key contributions to this
report are listed in appendix III.
Candice N. Wright
Acting Director
Science, Technology Assessment, and Analytics
Page 20 GAO-21-417 National Science Foundation
Appendix I: Summaries of the National
Science Foundation’s Major Facilities
Projects under Construction
This appendix provides individual summaries of the National Science
Foundation’s (NSF) five major facilities projects under construction: (1)
the Daniel K. Inouye Solar Telescope, (2) the Vera C. Rubin Observatory,
(3) the Regional Class Research Vessels, (4) the Antarctic Infrastructure
Modernization for Science, and (5) Large Hadron Collider High
Luminosity Upgrade Program.
Each project’s summary is based on project documents and other
information that NSF officials provided and includes the following:
• an overview of the project and its purpose;
• a timeline identifying key project dates, including the date of the
original construction award, which we report as the start of
construction;
• project information, such as the project’s scheduled completion date
for construction (including schedule contingency), the type and latest
amounts of the awards for construction, 1 the responsible NSF
directorate, project partners, and expected duration of operations;
• tables summarizing the project’s current status and its cost and any
cost 2 or schedule 3 increases or scope reductions made under NSF’s
no cost overrun policy and changes since our April 2020 report; 4
• a summary of the project’s cost and schedule performance history;
• a chart depicting the latest construction award’s total project cost for
construction, including the performance measurement baseline and
budget contingency;
1Costs are reported in then-year dollars, which means that NSF or the recipient converted
base-year dollars by applying an inflation index. According to NSF policy, inflation is a part
of NSF’s budgeting and project planning.
2NSF measures cost increases against the not-to-exceed cost that the National Science
Board authorized under the agency’s no cost overrun policy. Therefore, we define cost
increases since starting construction as increases to the not-to-exceed cost that the Board
authorized.
3We identified schedule increases by comparing the project’s scheduled completion date
in the construction award as of December 2020 with the scheduled completion date in the
original construction award. When a project’s scheduled completion date was not
identified in the award, we used the expiration date of the award.
4GAO, National Science Foundation: Cost and Schedule Performance of Major Facilities
Construction Projects and Progress on Prior GAO Recommendations, GAO-20-268
(Washington D.C.: April 3, 2020).
Page 21 GAO-21-417 Natioanl Science Foundation
• if applicable, a chart showing the increase in the construction award’s
total project cost since the original construction award; and
• information on remaining project risks and potential for cost or
schedule increases, including the amount of remaining contingency
and scope reduction options. 5
5We report each project’s estimate of remaining risk exposure as weighted by the
recipients for the probability of the risks occurring. According to NSF’s Major Facility
Guide, risk exposure is the quantitative impact of risks. We report the risk exposure as
determined by the Monte Carlo method when available.
Page 22 GAO-21-417 Natioanl Science Foundation
DANIEL K. INOUYE SOLAR TELESCOPE
When completed, the National Science Foundation’s (NSF) Daniel K. Inouye Solar
Telescope (DKIST), formerly named the Advanced Technology Solar Telescope, will be
the world’s flagship facility for the study of magnetic phenomena in the solar atmosphere.
It will help answer fundamental questions in solar physics and enable understanding of
solar variability and activity, which can affect Earth through phenomena generally
described as space weather.
Project Information Project Summary
Location: Maui, Hawaii. Construction of NSF’s DKIST project was 98 percent complete, as of
December 2020. The project was in its 11th year of construction and in
the process of testing and commissioning the facility. Since our April 2020
report, which included data from September 2019, the project installed
and aligned a key component of the telescope's optics. However, the
Coronavirus Disease 2019 (COVID-19) pandemic and extreme weather
events in Hawaii in early 2020 delayed the project. As of December 2020,
the estimated end date for construction was June 2021, but in May 2021
NSF staff said they expect the end of construction in December 2021. As
of February 2021, the National Science Board approved a total
management reserve of $18.9 million to cover expenses related to the
pandemic, such as increased labor due to delays or inefficiencies. NSF
Expected construction completion applied $6.7 million of this reserve to the project in December 2020 and
date, including schedule $2.7 million in March 2021.
contingency:
December 2021. Construction Status of the Daniel K. Inouye Solar Telescope, as of March 2021
Construction award: Percentage complete 98
Cooperative support agreements with Dollars in millions
the Association of Universities for Not-to-exceed cost that the National Science Board authorized 363.0
Research in Astronomy, Inc., consisting Total project cost in latest construction awards a 353.4
of 42 U.S. institutional members and
five international affiliates. National Science Foundation (NSF) funding obligated to date 353.4
Responsible NSF directorate:
Mathematical and Physical Sciences. Changes in Cost, Schedule, and Scope
Dollars in millions
Project partners:
Cumulative changes since Changes since
More than 20 U.S. and international original construction award April 2020
organizations. Leibniz-Institut für
Sonnenphysik (Germany) and Queens Not-to-exceed cost that the National +65.1▲ +18.9▲
University Belfast (Northern Ireland) Science Board authorized
are supplying additional equipment for Total project cost +55.5▲ +9.3▲
the project. Scheduled completion date (months) +42▲ +12▲
Expected duration of operations: Scopeb -5.9▼ None
50 years. Legend: ▲ = cost or schedule increase; ▼= scope reduction.
Source: NSF documents and officials. | GAO-21-417 Source: GAO analysis of NSF documents and information from NSF officials. | GAO-21-417
aIncludes an award funded by appropriations under the American Recovery and Reinvestment Act of
2009 and an award funded by NSF’s Major Research Equipment and Facilities Construction account.
bScope changes included are reductions in response to NSF’s policy on cost overruns or as part of a
cost increase.
Page 23 GAO‑21-417 National Science Foundation
NSF’S DANIEL K. INOUYE SOLAR TELESCOPE
Latest Construction Awarda Cost and Schedule Performance History
Total Project Cost, in millions, as of DKIST was delayed for pandemic and non-pandemic related reasons. In
March 2021
February 2020, there was an ice storm which caused a power outage at
the construction site, and a subsequent 2-week slow down. Then the
construction site was shut down again from March 17 to June 4, because
of the COVID-19 pandemic. The project began a phased reentry to the
site on June 4, 2020. The project requested, and NSF approved, a no
cost extension to the project’s end date from June 30, 2020 to December
31, 2020. However, the project was unable to meet this deadline
because, among other things, personnel needed for testing were located
Note: Percentages do not sum to 100
percent due to rounding. on the U.S. mainland and unable to reach Hawaii until it reopened to
travel in October 2020. To address costs and delays related to the
Increase in Construction Awarda
pandemic, NSF initially increased the not-to-exceed cost of the project to
Total project cost, in millions, as of
March 2021 include a management reserve of $9.4 million using authority delegated
to NSF by the National Science Board to adjust costs by up to $10 million.
In February 2021, the National Science Board approved an additional
increase in management reserve of $9.5 million for a total reserve of
$18.9 million. As of March 2021, NSF applied $9.4 million of the reserve
to the award and delayed end of construction to June 2021.
Remaining Project Risks and Potential for Cost or Schedule
Increases
As of December 2020, the DKIST project had $3.5 million of budget
contingency remaining—$1.1 million more than the estimated remaining
risk exposure of about $2.4 million when weighted for the risks’
probability. The project also had 1 month of schedule contingency
remaining to offset any potential delays in completing construction.
However, NSF had to extend the award due to commissioning challenges
and the pandemic which will require further use of management reserve
Source: NSF documents and officals. | GAO 21-417 within the $18.9 million currently authorized.
aIncludes an award funded by
appropriations under the American According to the project documentation, the largest remaining non-
Recovery and Reinvestment Act of 2009 pandemic risk category is project completion and closeout, which includes
and an award funded by NSF’s Major
Research Equipment and Facilities
risks related to staff retention and performance and quality issues. As of
Construction account. December 2020, this category included seven risks with about $1.4
million in probability weighted risk exposure. The project maintains a list
Remaining Contingency and Scope of scope reduction options, which, as of December 2020, included
Reduction Options approximately $21,000 in total possible project scope reduction options.
As of December 2020 with construction However, the ability of these remaining scope reduction options to reduce
98 percent complete. costs will continue to decrease as the project approaches completion.
NSF Management Reserve:
$18.9 million
Budget contingency:
$3.5 million of budget contingency
remaining ($1.1 million more than the
estimated remaining risk exposure of
about $2.4 million).
Schedule contingency:
One month.
Estimated value of remaining scope
reduction options:
$21,000
Source: NSF documents and officials. | GAO-21-417
Page 24 GAO‑21-417 National Science Foundation
VERA C. RUBIN OBSERVATORY
The National Science Foundation’s (NSF) Vera C. Rubin Observatory (Rubin), an 8.4-
meter, wide-field optical telescope, will initially be used to image the entire visible
southern sky—every 3 days for a decade—using the world’s largest digital camera (3
billion pixels). Built on a mountaintop in Chile, a location with pristine skies, the
telescope will collect data and images to chart billions of galaxies and increase
knowledge about potentially hazardous asteroids, dark matter, and energy. Rubin has
the potential to advance every field of astronomical study, from the inner solar system
to the large-scale structure of the universe. Its former name was the Large Synoptic
Survey Telescope (LSST).
Project Information Project Summary
Location: Cerro Pachón, Chile. As of December 2020, the Rubin project was 86 percent complete and in
its 7th year of construction. Since our April 2020 report, the project made
progress on the observatory’s dome and telescope mount assembly.
However, construction was halted in Chile due to the pandemic in March
and the site remained closed until late September 2020. The project is no
longer operating on schedule because of pandemic-related delays. NSF
expects the end of construction will occur in early 2024, but no formal
changes have yet been made to the project schedule. NSF increased the
not-to-exceed cost by $10 million to include a management reserve for
costs due to the COVID-19 pandemic, such as increased labor due to
delays or inefficiencies.
Construction Status of the Vera C. Rubin Observatory, as of December 2020
Percentage complete 86
Dollars in millions
Not-to-exceed cost that the National Science Board authorized 483.0
Scheduled construction completion Total project cost in latest construction award 471.2a
date, including schedule National Science Foundation (NSF) funding obligated to date 459.4
contingency:
October 2022.
Construction award: Changes in Cost, Schedule, and Scope
Dollars in millions
Cooperative support agreement with
the Association of Universities for Cumulative changes since Changes since
Research in Astronomy, Inc., consisting original construction award April 2020
of 42 U.S. institutional members and
Not-to-exceed cost that the National +10▲ +10▲
five international affiliates.
Science Board authorized
Responsible NSF directorate:
Total project costb +3.4▲ None
Mathematical and Physical Sciences.
Scopec -9.5▼ -8.1▼
Project partners:
The LSST Corporation, Department of Scheduled completion date (months)d None None
Energy.
Legend: ▲ = cost or schedule increase; ▼= scope reduction.
Expected duration of operations: Source: GAO analysis of NSF documents and information from NSF officials. | GAO-21-417
50 years.
a
Excludes fee of $1.1 million in total provided to date to stimulate efficient performance since award.
Source: NSF documents and officials. | GAO-21-417
b
This cost change was anticipated at the time of the original construction award, according to NSF
officials, in order to accommodate evolving NSF policies on budget contingency.
c
Scope changes included are reductions in response to NSF’s policy on no cost overruns at initial
award ($1.4 million) or to increase the available budget contingency ($8.1 million).
d
Schedule does not yet incorporate COVID-19 pandemic impacts.
Page 25 GAO‑21-417 National Science Foundation
NSF'S VERA C. RUBIN OBSERVATORY
Latest Construction Award Cost and Schedule Performance History
Total project cost, in millions, as of In June 2020, NSF increased the not-to-exceed cost for Rubin by $10
December 2020
million to $483 million to include a management reserve to address
pandemic-related costs. According to NSF and project documents, the
added costs are primarily due to schedule delays as well as direct costs
associated with office and construction site closures. As of March 2021,
NSF had not yet applied any of this reserve to the award, thus the total
project cost has not changed since our April 2020 report. However, NSF
and the project anticipate that additional funding beyond the $10 million
Notes: Excludes fee of $1.1 million provided to the recipient
will be needed for pandemic-related costs. Further, increasing the cost of
to stimulate efficient performance. Percentages do not sum
to 100 percent due to rounding.
the project will require authorization from the National Science Board.
Remaining Contingency and
The project will not finish by its current construction completion date of
Scope Reduction Options October 2022. Prior to the pandemic, the project had experienced delays
related to the construction of the dome enclosure and telescope mount
As of December 2020 with construction
86 percent complete.
assembly. According to NSF, significant progress was made on these
items in early 2020. However, they remain unfinished as the project
NSF Management Reserve:
halted construction in March 2020 in response to the COVID-19
$10 million
pandemic and the Chilean government-imposed quarantine throughout
Budget contingency:
the country in July 2020. Limited construction activities at the site began
$27.4 million ($2.8 million more than
the probability-weighted risk exposure
again in late September 2020. Dome construction resumed in November
of $24.6 million) and mount assembly construction resumed in January 2021. NSF
Schedule contingency: expects a re-baselining of the project in late 2021 to establish a new
4.8 months.
schedule and total project cost. According to project documents, the
Estimated value of remaining scope
pandemic will delay completion by at least 10 months. The project
reduction options: measurement baseline currently has 4.8 months of schedule contingency
$9 million. for non-pandemic-related delays, however pandemic impacts and the
Source: NSF documents and officials. | GAO-21- planned re-baselining will alter the project's schedule.
417
Remaining Project Risks and Potential for Cost or Schedule
Contributions of Project Partners
Increases
The U.S. Department of Energy (DOE),
a cosponsor of the Vera C. Rubin As of December 2020, the project had an estimated remaining risk
Observatory is responsible for exposure of $24.6 million for non-pandemic-related risks, which is $2.8
delivering the camera at a cost of $168 million less than the remaining budget contingency of $27.4 million.
million. SLAC National Accelerator
Laboratory manages a collaboration of
According to project documents, factoring in actions to mitigate these
DOE national laboratories and risks lowers the expected risk exposure to just $12.4 million. The largest
universities to develop, fabricate, and non-pandemic-related risks include late delivery or integration of the
deliver the camera. The COVID-19 dome enclosure or mount assembly and late delivery of the DOE-funded
pandemic also impacted camera camera.
construction, but as of January 2021
NSF did not expect camera In accordance with NSF policy, the project maintains a list of scope
construction would further delay the reduction options. As of December 2020, there was approximately $9
project.
Source: GAO analysis of NSF and DOE
million in total possible project scope reduction options remaining. As the
information. project moves towards completion, fewer scope reduction options will be
available.
Page 26 GAO‑21-417 National Science Foundation
REGIONAL CLASS RESEARCH VESSELS
The National Science Foundation’s (NSF) Regional Class Research Vessels (RCRV)
project will construct three 199-foot vessels to support the nation’s ability to conduct
fundamental scientific research in the coastal zone and continental shelf, including from
the ocean’s surface through the water column to the sea floor and subsea floor
environment. These vessels will provide enhanced capabilities beyond those of the
retiring vessels they will replace. The three vessels’ research locations will depend on
where the science demand is greatest, but NSF plans to operate the first vessel along the
west coast of the U.S.
Project Information Project Summary
Location: Construction site is in As of December 2020, construction of NSF’s RCRV project was 35
Louisiana.
percent complete and the project was in its 4th year of construction. Since
our April 2020 report, the RCRV project has made progress on vessel
construction and 3D modeling, but has fallen behind schedule. The
project continued construction on vessel 1 and vessel 2. They also
performed a keel laying ceremony for vessel 3 in March 2020 but delayed
start of construction until January 2021. NSF increased the not-to-exceed
cost by $10 million in December 2020 and designated a management
reserve of about $14.1 million held by NSF for COVID-19 related costs,
such as increased labor costs due to delays or inefficiencies. NSF
authorized the project to use $5.5 million of this reserve to address
COVID-19 costs to date, but NSF has not yet applied any reserve since
Scheduled construction completion the current obligations cover these increases.
date, including schedule contingency:
January 2025 for three vessels. Construction Status of the Regional Class Research Vessels, as of March 2021
Construction award: Percentage complete (based on construction of three vessels) 35
Cooperative support agreement with Dollars in millions
Oregon State University, which Not-to-exceed cost that the National Science Board authorized 375.0
contracted with Gulf Island Shipyards,
LLC. Total project cost in latest construction award 359.5
Responsible NSF directorate: National Science Foundation (NSF) funding obligated to date 343.0
Geosciences.
Project partners: Changes in Cost, Schedule, and Scope
The U.S. Navy supported initial design Dollars in millions
oversight for the vessels. Cumulative changes since Changes since
Expected duration of operations: original construction award April 2020 report
30 years. Not-to-exceed cost that the National +10▲ +10▲
Source: NSF documents and officials. | GAO-21-417 Science Board authorized
Total project cost +5.5▲ +5.5▲
Scopea None None
Scheduled completion date (months) +6▲ +6▲
Legend: ▲ = cost or schedule increase.
Source: GAO analysis of NSF documents and information from NSF officials. | GAO-21-417
a
Scope changes included are reductions in response to NSF’s policy on cost overruns or as part of a
cost increase.
Page 27 GAO‑21-417 National Science Foundation
NSF’S REGIONAL CLASS RESEARCH VESSELS
Latest Construction Award Cost and Schedule Performance History
Total project cost, in millions, as of March As of December 2020, the RCRV project had no schedule contingency
2021
remaining and NSF anticipated additional schedule delays due to the
pandemic. The project also increased its total project cost and delayed
project completion, but had not implemented any scope reductions.
The RCRV project has fallen behind schedule, in part due to shipyard
delays in developing the 3D models and technical drawings needed for
construction. As we reported in April 2020, the project recipient, Oregon
State University, had concerns regarding the project management and
Note: Percentages do not sum to 100 percent due to
subcontractors of the Gulf Island Shipyards that led to a partial work
rounding. suspension in early 2019. In November 2019, Oregon State University
Remaining Contingency and Scope implemented the contract's dispute resolution process and assumed
Reduction Options responsibility for the project's modelling. In May 2020, the project
implemented a change to its baseline to address the impacts of the
As of December 2020 with construction
of three vessels 35 percent complete. dispute. This change used $18 million of budget contingency, including a
NSF Management Reserve:
$12.2 million increase in the contract to the shipyard. The change also led
$14.1 million
to about a 9-month delay in delivery of the three vessels and exhausted
the 180 days of schedule contingency available at the time.
Budget contingency:
$24.7 million ($4.5 million more than the In addition, the project has continued to fall behind schedule due to the
probability-weighted risk exposure of COVID-19 pandemic. The shipyard lacked sufficient staff to complete
$20.2 million). RCRV construction as planned and labor inefficiencies led to delays in
Schedule contingency: technical drawings used by the shipyard. In December 2020, NSF
None. authorized about $14.1 million in management reserve to cover costs
Estimated value of remaining scope associated with the pandemic by increasing the project's not-to-exceed
reduction options: cost by $10 million and using additional funds available because the total
$4.9 million project cost in the construction award was less than the prior not-to-
Source: NSF documents and officials. | GAO-21-417
exceed cost. No additional funds have yet been obligated to the
construction award, but NSF subsequently authorized Oregon State
University to modify its contract with the shipyard by $5.5 million to
address the increased labor costs in 2020. This modification also delayed
delivery of the first two vessels by 4 months, delivery of the third vessel
by 3 months, and project completion by 6 months.
Since we reported in April 2020, vessel delivery has slipped from August
2021 to September 2022 for the first vessel; from January 2022 to March
2023 for the second vessel; and from July 2022 to August 2023 for the
third vessel.
Remaining Project Risks and Potential for Cost or Schedule
Increases
As of December 2020, the project had an estimated risk exposure of
$20.2 million and a remaining contingency of $24.7 million. Inadequate
shipyard performance remains a major risk, but has decreased due to the
project's re-planning efforts. Nonetheless, hull delivery delays remain a
risk to completing the project on time and budget.
The project has implemented no scope reductions. Fourteen scope
reduction options remain totaling $4.9 million in cost reduction. The
largest remaining option was to reduce the 18-month timeframe currently
devoted to transitioning from construction to full operations. Such a
reduction could save $1.5 million in staff costs, but would incur risks to
transitioning activities that could impact operations. Other scope reduction
options included eliminating certain pieces of equipment, such as the
inspection vehicle or cold storage space, to support longer term
operations.
Page 28 GAO‑21-417 National Science Foundation
ANTARCTIC INFRASTRUCTURE
MODERNIZATION FOR SCIENCE
The National Science Foundation’s (NSF) Antarctic Infrastructure Modernization for
Science (AIMS) project will modernize the core infrastructure of McMurdo Station in
Antarctica, the largest of three stations operated by NSF’s United States Antarctic
Program and used by multiple agencies. McMurdo Station serves as a logistics hub for
remote field sites and for the Amundsen-Scott South Pole Station. The AIMS project is
expected to make environmental and safety upgrades to McMurdo Station and redevelop
it into a more compact, operationally and energy-efficient core facility to support research.
Note: Rendering of McMurdo Station’s core facility. The planned core facility will consolidate critical buildings, such as medical facilities and
field science support.
Project Information Project Status
Location: McMurdo Station, Antarctica Construction of the AIMS project was about 24 percent complete as of
December 2020. In March 2020, work at the project site was suspended
to avoid the possibility of introducing COVID-19 to Antarctica. As of
December 2020, AIMS remains behind schedule with all construction
work at McMurdo paused. The site has been placed in a safe and stable
condition until pandemic and longer-term logistical impacts are better
understood. NSF and the awardee continue to discuss how to reprioritize
work and maintain forward momentum given the considerable
uncertainties of the global logistics chain and construction feasibility due
to COVID-19. Materials for the Vehicle Equipment and Operation Center
and Lodging facilities are still being procured and delivered to the
project’s shipping staging site in Port Hueneme, California. Prior to the
Expected construction completion
date, including schedule contingency: suspension of work, production of the stone material needed for new
January 2028 structures at the site had begun as had the demolition of retired
Construction award:
structures.
February and April 2019 modifications to
the existing Antarctic support contract Construction Status of the Antarctic Infrastructure Modernization for Science, as of
with Leidos Innovations Corporation. December 2020
Percentage complete 24
Responsible NSF directorate:
Geosciences. Dollars in millions
Project stakeholders: Not-to-exceed cost that the National Science Board authorized 410.4
Other federal agencies—such as the Total project cost in latest construction awards 410.4
National Aeronautics and Space National Science Foundation (NSF) funding obligated to date 151.7
Administration, National Oceanic and
Atmospheric Administration, and the Source: GAO analysis of NSF information. | GAO-21-417
Department of Energy.
Expected duration of operations:
35 to 50 years.
Source: GAO analysis of NSF information. | GAO-21-417
Page 29 GAO‑21-417 National Science Foundation
NSF’S ANTARCTIC INFRASTRUCTURE MODERNIZATION FOR SCIENCE
Latest Construction Award Cost and Schedule Performance History
Total project cost, in millions, as of Due to the pandemic, AIMS remains behind schedule with all construction
December 2020
work at McMurdo Station paused. With the loss of a second construction
season likely, NSF officials expect that the project will be re-baselined by
the end of 2021. In accordance with NSF policy, this could include
reductions in project scope, a return to the National Science Board for an
increased total project cost, or both. As of December 2020, the impacts of
the pandemic were not yet reflected as construction cost increases,
formal changes to its scheduled completion date, or scope reductions.
The project’s not-to-exceed cost of $410.4 million, which included $67.2
Note: .The baseline includes fees and on-site oversight costs. million in budget contingency, was set by the National Science Board in
February 2019. On April 30, 2021, NSF determined that it would not be
Independent Cost Estimate funding additional components of AIMS due to the significant impacts of
In November 2018, the U.S. Army Corps the pandemic. Current components of AIMS include the Vehicle
of Engineers completed an independent Equipment and Operations Center, the first lodging unit, and associated
cost estimate (ICE) report for the AIMS
project. According to NSF officials, the
utilities.
ICE was critical for negotiations with the
contractor as NSF utilized data within the Remaining Project Risks and Potential for Cost or Schedule
ICE, such as labor rates and cost of Increases
materials, to verify costs. Specifically, the
ICE assisted NSF in determining the As of December 2020, the AIMS project had a risk exposure of $52.1
reasonableness of the contractor’s million and $57.8 million in remaining contingency, and 15.2 months of
proposed cost estimate and schedule for schedule contingency remained available. The project had cumulatively
the project and associated risks. used $9.4 million in budget contingency. Of this, $7.3 million was used for
According to NSF officials, NSF and the
contractor resolved all recommendations
contract modifications for initial construction, with the remainder used for
from the ICE report to NSF’s satisfaction other realized risks, including additional equipment purchases and leases.
prior to setting the not-to-exceed cost.
As of December 2020, the AIMS project had $17.0 million in major risks
Remaining Contingency and Scope monitored by NSF. Given the level of uncertainty and high potential
Reduction Options impact on Antarctic operations associated with the COVID-19 pandemic,
As of December 2020 with construction a meaningful risk profile will not be available until the project is re-
about 24 percent complete. baselined. Other major previously-known risks being monitored include
Budget contingency: proposals exceeding estimates with an estimated value of $15.2 million,
and design and value engineering delays with an estimated value of $1.4
$57.8 million ($5.7 million more than the million, and their associated impacts have been overtaken by the
probability-weighted risk exposure of
$52.1 million).
pandemic. Environmental conditions at McMurdo during the 2020/2021
season prevented the construction of a safe ice pier for cargo unloading,
Schedule contingency: but this did not impact AIMS since no on-site construction was planned
15.2 months (included in the 2028 due to the pandemic. Domestic production of AIMS components was able
scheduled completion date). to continue.
Source: GAO analysis of NSF information. | GAO-21-417
In accordance with NSF policy, the project maintains a list of scope
reduction options, which as of April 2019 included approximately $34.0
million to $43.1 million in total possible project scope reduction options.
For example, the largest scope reduction option, with an estimated value
of up to $19.1 million, is to remove the new trades shop from the AIMS
scope and instead continue to use the current facility. Another option, with
an estimated value of up to $4.0 million, is to remove the gymnasium from
the emergency operations facility and instead continue to use and
maintain the existing gymnasium.
Page 30 GAO‑21-417 National Science Foundation
LARGE HADRON COLLIDER HIGH
LUMINOSITY UPGRADE
The Large Hadron Collider (LHC) is the world’s most powerful particle accelerator. The
facility’s four detectors observe new particles that are produced when high-energy
protons are accelerated and collided, providing insight into fundamental forces of nature
and the condition of the early universe. Through the National Science Foundation’s (NSF)
Large Hadron Collider High Luminosity Upgrade (HL-LHC) program, the agency will fund
a portion of a larger international effort to upgrade the facility’s accelerator and detectors.
Specifically, NSF plans to fund the design and implementation of certain parts of the
upgrades as two separate projects for the facility’s detectors, the A Toroidal LHC
Apparatus (ATLAS) and Compact Muon Solenoid (CMS) detectors. The Department of
Note: photograph above depicts the A Toroidal Large Hadron Energy (DOE) is also contributing to upgrades to the LHC’s accelerator and to the ATLAS
Collider Apparatus detector.
and CMS detectors.
Project Information Project Summary
Location: Geneva, Switzerland. As of December 2020, NSF’s HL-LHC program was in its 1st year of
construction. In February 2020, the National Science Board authorized an
award to Columbia University for the ATLAS detector not to exceed $75
million and a second award to Cornell University for the CMS detector not
to exceed $78 million. NSF awarded construction awards to the two
universities in March 2020. While the upgrades involve separate
cooperative agreements for each detector, NSF considers them one
program consisting of two distinct projects. The COVID-19 pandemic has
delayed construction of the detectors, including an estimated 7-month
delay for ATLAS from its baseline schedule, and impacted other aspects
of the LHC upgrade. Excluding impacts of the pandemic, the detectors'
Expected construction completion baseline schedule estimates end of construction in 2025, not including
date, including schedule schedule contingency, or 2026 including it.
contingency:
December 2026. Construction Status of the Large Hadron Collider High Luminosity Upgrade, as of
Construction awards: December 2020
Cooperative agreements with Columbia Dollars in millions
University (ATLAS detector) and ATLAS CMS Program Total
Cornell University (CMS detector).
Percentage complete 8.2 9.4 not applicable
Responsible NSF directorate:
Mathematical and Physical Sciences. Not-to-exceed cost that the 75 78 153
Project partners: National Science Board
authorized
European Organization for Nuclear
Research (CERN) and the Department Total project cost in latest 75 77.2 152.2
of Energy. construction awards
Expected duration of operations: National Science Foundation 22 27.9 49.9
12 years. (NSF) funding obligated to
Source: GAO analysis of NSF information. | GAO-21-417
date
Source: GAO analysis of documents and information from NSF. | GAO-21-417
Page 31 GAO‑21-417 National Science Foundation
NSF’S LARGE HADRON COLLIDER HIGH LUMINOSITY UPGRADE
Latest Construction Award Cost and Schedule Performance History
Total Project Cost, in millions, as of NSF has obligated $22 million for construction of ATLAS and $27.9
December 2020
million for CMS, as of December 2020. The project's initial awards
estimated total project costs of $75 million for ATLAS and $77.2 million
for CMS. The National Science Board's not-to-exceed cost is $75 million
for ATLAS and $78 million for CMS. In addition, according to NSF
officials, NSF invested a total of $23.7 million for the design of the
detector upgrades. Funding for the design came from NSF’s Research
and Related Activities account. Funding for construction will come from
the Major Research Equipment and Facilities Construction account.
Remote work conditions and social distancing protocols due to the
COVID-19 pandemic have led to delays to both detectors as compared to
the baseline schedule. Both projects are assessing the impacts of the
pandemic, and neither has completed changes to their baseline
schedules as the impacts of the pandemic are ongoing.
Remaining Project Risks and Potential for Cost or Schedule
Increases
Neither the ATLAS team nor the CMS team has fully assessed the impact
of COVID-19 on their projects. Both project teams are developing change
requests to their baseline to address these impacts. NSF plans to use
management reserve, not the projects' contingency, to cover expenses
related to the pandemic but has not yet established reserve amounts for
these projects. According to NSF, the estimated cumulative costs
currently range from $2 million to $3 million, due to labor inefficiencies
Remaining Contingency and
Scope Reduction Options and re-planning costs, among others. As of December 2020, the project
teams estimate schedule delays from the pandemic of 7 and 2 months for
As of December 2020.
ATLAS and CMS, respectively.
Budget contingency:
$36.9 million as follows According to project documentation, as of December 2020, the ATLAS
project team had $18.8 million in budget contingency remaining after
• $18.8 million for the ATLAS detector.
• $18.1 million for the CMS detector.
using $1.2 million of its initial $20 million contingency fund. The highest
impact risks managed by the project team include volatility of commodity
Schedule contingency:
prices, delays due to other ongoing projects at CERN, and loss of key
• 13.5 months for the ATLAS detector. personnel. The project team plans to update its risk exposure in late fiscal
• 10.2 months for the CMS detector.
year 2021.
Estimated value of scope reduction
options: As of December 2020, the CMS project had $18.1 million in budget
$15.1 million as follows contingency remaining after using $840 thousand of its initial $18.9 million
• $8.4 million for the ATLAS detector. contingency budget. In October 2020, the CMS project team estimated its
• $6.7 million for the CMS detector. risk exposure was $17.5 million. The highest impact risks managed by the
Source: GAO analysis of NSF information. | GAO-21-417 project include uncertainty regarding foreign currency exchange rates,
DOE’s Contributions to Upgrading unavailability of scientific labor, and quality problems. Risks regarding
the Large Hadron Collider COVID-19 were not included in the project team's analysis.
The estimated cost range of DOE's NSF policy also directs a project’s design to include prioritized, time-
contributions is $149 million to $181 phased options for reducing its scope during construction if needed. As of
million for the ATLAS detector and
the final design review, the project teams had identified a total of $15.1
$144 million to $183 million for the
CMS detector. DOE’s work on the million of potential scope reduction options for the projects, which are
detectors is led by the department’s subject to change throughout the design and construction of a project.
national labs, whereas NSF’s work is
led by university researchers.
Source: DOE fiscal year 2021 budget information. |
GAO-21-417
Page 32 GAO‑21-417 National Science Foundation
Appendix II: Summary of the National
Appendix II: Summary of the National Science
Foundation’s Future Major Facilities Project in
Science Foundation’s Future Major Facilities
Design
Project in Design
This appendix provides an individual summary of the National Science
Foundation (NSF) project that is in design and planned for construction as
a major facilities project: the Leadership Class Computing Facility. As of
December 2020, no construction funds had been awarded for this project
and all cost, schedule, scope, and design information was subject to
change.
The project’s summary is based on project documents and other
information that NSF officials provided and includes the following:
• an overview of the project and its purpose;
• a timeline identifying key project dates;
• project information, such as the expected date for completion of
construction; the anticipated type of awards for construction; the
responsible NSF directorate; project partners; and expected
duration of operations;
• a summary of the project’s current status;
• a summary of the project’s design and construction costs, if
available, and the budget account NSF planned to use for
construction of the project; 1 and
• information on potential project risks.
1Costs are reported in then-year dollars, which means that NSF or the recipient converted
base-year dollars by applying an inflation index. According to NSF policy, inflation is a part
of NSF’s budgeting and project planning.
Page 33 GAO‑21-417 National Science Foundation
LEADERSHIP-CLASS COMPUTING FACILITY
The National Science Foundation’s (NSF) Leadership-Class Computing Facility (LCCF)
project is intended to provide advanced computational capabilities to enable
transformative research in all areas of science and engineering that would not be
possible by theory or experiment alone. According to NSF officials, future research using
LCCF might include extremely detailed simulations ranging from biological molecules to
supernovae and analyses of very large data streams such as satellite images to create
high-resolution Earth maps.
Note: Photograph above depicts NSF’s most advanced
computing system currently in operation, known as Frontera.
Project Information Project Status
Location: Texas Advanced Computing As of December 2020, the LCCF project was in its second year of design;
Center, University of Texas at Austin consequently, all cost, schedule, scope, and design information for the
project was subject to change. In March 2019, the NSF Director approved
the project to enter the design stage as a candidate major facilities
project. NSF approved advancing the project to the preliminary design
phase in September 2020, following the recommendation of the
conceptual design review and facilities readiness panels. A preliminary
design phase award of $3.5 million was made on September 17, 2020.
The project represents the final phase of a two-phase deployment of high-
performance computing systems. The first phase—known as the Frontera
project at the Texas Advanced Computing Center at the University of
Texas at Austin—was completed in September 2019. According to NSF,
Expected construction completion at that time, Frontera was the largest high-performance computing
date, not including schedule system deployed on a U.S. academic campus. The LCCF project will
contingency:
support the design and construction of an upgrade to the Frontera system
Fiscal Year 2026.
as well as to the physical facility that will host it.
Construction award:
Planned for 2024. Design and Construction Costs
Responsible NSF directorate:
As of December 2020, NSF had not yet established the construction cost
Directorate for Computer & Information
Science & Engineering. and scope for the LCCF project. The final cost will be subject to the
Project partners:
outcomes of further design reviews, NSF approval, and eventual National
Science Board authorization. In fiscal years 2019 and 2020, NSF
None.
obligated $5.5 million from its Research and Related Activities account for
Expected duration of operations:
the design of LCCF. Based on a recommendation from the Conceptual
10 years.
Design Review Panel, NSF is also planning to fund 15 to 20 science
Source: GAO analysis of NSF information. | GAO-21-417 teams ($4.5 million to $6.0 million) in fiscal year 2021 to help define the
benchmark suite of science applications that will be used to measure the
performance improvements of the LCCF system. According to the
project’s cooperative agreement, NSF may provide $2.5 million additional
funding in fiscal year 2022 to advance the design of LCCF following
successful completion of the preliminary design review, subject to
availability of appropriations. The preliminary design review is tentatively
scheduled for October 2021.
Page 34 GAO‑21-417 National Science Foundation
NSF’S LEADERSHIP-CLASS COMPUTING FACILITY
NSF’s Support for High- Project Risks
Performance Computing Systems
As of December 2020, NSF had not yet formally identified risks for the
NSF has supported high-performance
computing capabilities for nearly 4 LCCF project because the project is early in the design stage. The project
decades. In 2007, NSF awarded team identified 25 significant risks during the conceptual design, such as
$226.6 million for the Blue Waters high- failing to meet performance specifications. However, the conceptual
performance computing system design review panel found that there were immediate, identifiable risks
through a cooperative agreement to the
University of Illinois at Urbana- that were absent or insufficiently delineated in the design and
Champaign. According to NSF, at the recommended that the project team expand their consideration of risks
time of its deployment in 2013, Blue early in the preliminary design phase.
Waters was one of the most powerful
supercomputers in the world and was According to NSF officials, one anticipated challenge for the LCCF project
one of the fastest on a university is the rapid pace of technological change in the field of high-performance
campus. Scientists and engineers
across the country used the computing
computing. The officials stated that forecasting the technology
and data power of Blue Waters to marketplace in the future can be challenging as technology can change
tackle a wide range of problems, radically because of external market forces. Conversely, the rapid pace of
including predicting the behavior of change can also be an opportunity if the LCCF project can incorporate
complex biological systems and
simulating the evolution of the cosmos.
the latest technological advances that result in the most advanced
computing capabilities. According to NSF officials, taking advantage of
Because of the rapid evolution of
computer technology, by 2019, NSF no such opportunities as late in the design or construction stage as possible
longer considered Blue Waters to be will be important for the success of the project.
the leadership computing system for
fundamental science and engineering
research. Anticipating these
technological advances, in August
2018, NSF awarded about $60 million
to the University of Texas at Austin for
the follow-on project to Blue Waters.
Frontera was intended to provide three
to five times the computing capability
and twice the storage capacity to
support the increased computational
requirements for science and
engineering research. NSF also
anticipated that Frontera would help
inform science requirements and
reduce risks for LCCF, which is
planned to provide substantially more
computational capabilities than both
Blue Waters and Frontera.
Source: GAO analysis of NSF information. | GAO-21-417
Page 35 GAO‑21-417 National Science Foundation
Appendix III: GAO Contact and Staff
Appendix III: GAO Contact and Staff
Acknowledgments
Acknowledgments
Candice N. Wright, (202) 512-6888 or WrightC@gao.gov
GAO Contact
In addition to the contact named above, Sarah Harvey (Assistant
Staff Director), Paul Kazemersky (Analyst in Charge), Molly Ryan, and Tind
Acknowledgments Shepper Ryen made key contributions to the report. Also contributing
were Cheron Brooks, Louise Fickel, and Patrick Harner.
(104515)
Page 36 GAO-21-417 National Science Foundation
The Government Accountability Office, the audit, evaluation, and investigative
GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of the
federal government for the American people. GAO examines the use of public
funds; evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO’s commitment to good government
is reflected in its core values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly
GAO Reports and released reports, testimony, and correspondence. You can also subscribe to
GAO’s email updates to receive notification of newly posted products.
Testimony
Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and
distribution and depends on the number of pages in the publication and whether
the publication is printed in color or black and white. Pricing and ordering
information is posted on GAO’s website, https://www.gao.gov/ordering.htm.
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card, MasterCard,
Visa, check, or money order. Call for additional information.
Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts.
Visit GAO on the web at https://www.gao.gov.
Contact FraudNet:
To Report Fraud,
Website: https://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in
Automated answering system: (800) 424-5454 or (202) 512-7700
Federal Programs
Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400,
Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125,
Relations Washington, DC 20548
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548
Stephen J. Sanford, Acting Managing Director, spel@gao.gov, (202) 512-4707
Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814,
External Liaison Washington, DC 20548
Please Print on Recycled Paper.
File and source
- File
- gao-21-417.pdf
- Size
- 2,240,737 bytes
- SHA-256
- 963206d785bb799f68c1f85921c886850969399ea9ffebbb4760ed2a42608cd6
- Our copy
- gao-21-417.pdf
- Original
- www.gao.gov