April 20, 2024

Exascale computing project brings engineering and science acceleration opportunities to industry

February 20, 2024 — Over its seven-year lifespan, the Department of Energy’s (DOE) Exascale Computing Project (ECP) has developed a capable high-performance computing (HPC) ecosystem, bringing together applications from critical mission, an integrated software stack and hardware technological advances to manifest and optimize Earth’s newest and most powerful supercomputers.

Frontier, the first DOE exascale system to come online, has already delivered unprecedented results. Using ECP-compatible software, researchers can run jobs on Frontier that simulate physical processes at extreme scales and complexity, allowing them to observe phenomena that were inaccessible with less powerful machines.

For industry partners and, ultimately, an entire HPC community that forms the foundation of US technological competitiveness, exascale provides the opportunity to investigate scientific and technical challenges that were previously too intensive in computational terms and expensive to address, providing a path to advance engineering far beyond the current state. the art. At SC23, held in Denver, Colorado, last November, members of ECP’s Industry and Agency Council, comprised of executives from U.S. companies, government agencies, and independent software vendors, reflected on how ECP and the move to exascale are impacting the current and planned use of HPC to accelerate component design and manufacturing, drive competitiveness, and build global technology leadership.

Digital twins and multidisciplinary simulation

Pete Bradley, principal investigator of Digital Tools and Data Science at Pratt & Whitney, highlighted the usefulness of exascale in simulating the complex combustion physics associated with current and next-generation gas turbine engines.

“We are going through a digital transformation,” Bradley said. “Pratt & Whitney is building a model-based company that connects all aspects of our product lifecycle, from customer requirements to preliminary design, through detailed designs, manufacturing and maintenance. Advanced modeling helps us build digital twins that allow us to optimize products virtually before bending a single piece of metal. This will deliver new advances to our customers in performance, fuel consumption and emissions, and shorten the time it takes to go from concept to production. Doing this work requires authoritative models at multiple levels of fidelity, from global to nanoscale. “Exascale computing can allow us to understand these phenomena that were previously out of our reach, and then we can integrate those models to deliver capabilities beyond anything the world has ever seen.”

Bradley notes that multidisciplinary simulations offer a way to visualize several physical processes simultaneously, for example fluid-structure interactions, which play a critical role in the performance and life of a component. The complexity of these design spaces requires using the tools of today and inventing those of tomorrow. Bradley discussed how machine learning combined with artificial intelligence could provide a mechanism for “physics-informed” models rather than just experience-based ones to accelerate component design and product commercialization in the future. He said there is “a long way to go” in terms of supercomputing capacity, which has interesting implications for the industry.

GPU and software validation

At oil and gas company ExxonMobil, subsurface imaging drives a large part of its upstream business, allowing scientists to visually separate different properties of rocks within the Earth. With greater computing power, the detail of those images has increased dramatically over the past 30 years.

“Computing really drives two big pieces,” said Mike Townsley, senior director of HPC at ExxonMobil. “It’s better images and faster resolution time, and we trade it off depending on how important a project is or how vital a very sharp image really is in an area.”

In 2021-2022, the company conducted an in-depth multidisciplinary study to determine what HPC capacity would be needed to double imaging performance without overcoming power and space limitations. Townsley said: “We discovered […] that there was no way we could do this without GPU.” The decision to move to GPUs; However, it was tempered by the dollar cost per performance of a system and the time and expense associated with porting code used for decades to GPU architectures.

Townsley highlighted the leadership role ECP played in accepting GPU devices at ExxonMobil: “[The] “The ECP effort demonstrated that GPU accelerators (and their software ecosystems) were productive across a wide variety of sciences, de-risking our decision.”

According to Townsley, ECP led the way by demonstrating the feasibility of GPU-based architectures for visualizing different physics problems, and this work gave the company confidence, knowing that the tools and capabilities would be available when it began porting its code. He attributed much of this increase to ECP’s support and investment for key tools and libraries such as SPACK, OpenMPI, HDF5, and zfp.

He also recognized ECP’s portability software suites, Kokkos and Raja, for developing future platform-independent tools. Ultimately, “better images lead to better results,” Townsley says, and ECP has provided a path forward to make more detailed, much higher-performance simulations of the subsurface a reality.

Key benefits beyond exascale

For GE Aerospace Research, DOE’s exascale computers are becoming a useful tool to virtually test new engine designs to achieve greater fuel efficiency and reduce CO2 emissions. “We have a history of working with the [national] propulsion turbomachinery laboratories,” says Rick Arthur, senior principal engineer for computational methods at the company. Historically, jet engine simulations were limited to what was pragmatically computable.

Arthur notes that as the HPC community has moved toward exascale and beyond, these simulations have extended beyond individual blades to entire rows, stages, and even entire systems. Arthur explains that Frontier has provided a platform to evaluate the aerodynamic and acoustic aspects of a novel open-fan engine design at full scale and under realistic flight conditions, a feat that was impossible without the computational power of exascale.

However, Arthur points out that ECP offers a key benefit in addition to access to exascale systems. “The search for exascale has been as important as exascale itself, providing an abundance of petascale,” he said. “Problems requiring full machine size are rare (and are currently prohibitively expensive to create, verify, and interpret), but that abundance allows exploratory sets to be run more freely at petascale, which is much more accessible to many in the industry.” .

Additionally, Arthur notes that the relationships, technological advances, and learnings derived from participating in ECP have helped inform the company’s internal HPC investments and preparation. When it comes to expanding the capabilities of what can be done computationally, Arthur says, “DOE has been the leader and we are grateful to follow.”

Download, modify, enable

TAE Technologies, Inc., is a private fusion company working to build a viable commercial fusion power plant as a source of abundant clean energy. Over its 25-year history, the company has developed increasingly larger and more advanced machines, and the science involved has become increasingly more complicated. “As machines grow and the physical problems we face become more complex, so do the modeling software we need and the HPC capabilities we need to address these problems,” said TAE computational scientist Roelof Groenewald.

As machines grow, more computing power is needed to model the full size of the reactor. During the panel discussion, Groenewald highlighted the benefits of modular software design as exemplified by ECP’s public WarpX code and its use in fusion reactor design. “The care that WarpX developers took in modularizing the code allows us to build a very different model for simulating next-generation fusion experiments. Because WarpX scales well, our WarpX-based simulations also scale well.”

WarpX was initially designed to simulate next-generation particle accelerators using laser plasma wake field acceleration, but the temporal and spatial scales of interest are different from those needed by the fusion community.

“One of the benefits for us, which applies to many ECP products, is that the code is completely open source. It is easy to start with the code, download it and modify it for free, which is essential for our use case,” says Groenewald. “The developers took the time to develop WarpX in an extremely modular way, where the different physics cores are essentially independent of each other and can be moved and arranged so that we can very quickly build a different model from the parts to simulate the specific length. and deadlines that interest us.”

Groenewald echoes the view that the large number of petascale aspects of exascale provides the ability to run sets of work to explore many aspects of design. Using the Perlmutter supercomputer at Lawrence Berkeley National Laboratory, the company ran its entire code in significantly less time (1 day instead of 10), illustrating how ECP products greatly help improve iteration speed in designs by accelerating solution time.


By pushing the boundaries of what is computationally possible and enabling people to do more, industry leaders are highlighting the value of ECP software and GPU acceleration for their work. The use of GPUs and future accelerators is key to performance and energy efficiency at all levels of computing systems, from the desktop to a rack system, a local HPC center or exascale. ECP brought together scientists, computing experts, vendors, and industry in the quest to make exascale a reality, and in doing so, demonstrated that better progress can be made together. Looking back at what has been done, what has been created, and what exascale will do, the future is bright. Fran Hill, chief scientist for the Department of Defense’s HPC Modernization Program, acknowledged that the work done through ECP is just the beginning of great things to come. “We cannot see this as the end of ECP, but rather as the beginning of the era of exascale.”

Source: Caryn Meissner and Rob Farber, ECP

Leave a Reply

Your email address will not be published. Required fields are marked *