Optimising Resource Utilisation in Exascale Supercomputing: Interview with the REGALE project

The REGALE project, which stands for "An open architecture to equip next-generation HPC (High-Performance Computing) applications with exascale capabilities," has successfully concluded after a three-year period from April 2021 to March 2024.

With a budget of €7,500,000 from the funding programme Horizon Europe, REGALE aimed to enhance the efficiency and effectiveness of supercomputing systems designed to operate at the exascale level.

The project brought together a diverse group of partners from Greece, Germany, France, Spain, Austria, and Italy. The collaborative effort was focused on optimising both computational resources and the processes used by developers working with these advanced systems. 

In this interview, Georgios Goumas, the project coordinator, shares insights into the project's goals, achievements, challenges, and future directions.

Can you please describe the project in your own words?

REGALE sought to address the challenges associated with managing resources in supercomputing systems, particularly those with exascale capabilities. 

Exascale computing refers to systems capable of performing at least one exaflop, or one billion billion (10^18) calculations per second. Such systems are incredibly powerful but come with high costs and substantial energy demands. Our goal was to ensure these resources are used effectively, both in terms of computational power and human effort.

We developed an open architecture and prototype systems to help manage these supercomputers more efficiently. This was crucial because modern applications are complex, involving big data, machine learning, and edge computing. Our work aimed to simplify the process of deploying and running these applications on large-scale systems, addressing the challenges of workflow management.

What have been the key objectives for the REGALE project and what progress has been made?

REGALE had two main objectives: improving the use of computational resources to enhance application performance, system throughput, and energy efficiency; and making it easier for developers to use supercomputing services. Over the past three years, we achieved these goals by creating a new open architecture that supports modular, scalable, and energy-efficient operations.

We built and tested several prototype systems based on this architecture. Additionally, we integrated advanced workflow engines into five pilot applications to improve their performance and flexibility, as well as help managing complex workflows.

Can you give some concrete examples of how the REGALE project supports European HPC users and promotes greener and more sustainable supercomputing?

REGALE supports European HPC users by simplifying the development and deployment of complex, workflow-based applications on large supercomputing systems. This is particularly beneficial for developers who are not HPC experts, as it reduces the complexity involved in using these advanced systems.

In terms of sustainability, our project focuses on reducing the energy consumption and carbon footprint of supercomputers. For example, our architecture supports operational modes that optimise power usage, helping to manage the environmental impact of high-performance machines. We also developed tools that improve power management and monitoring, contributing to more energy-efficient supercomputing practices.

What were the main challenges you encountered during the project's development?

One major challenge was navigating the extensive array of existing tools in the HPC ecosystem. Integrating these tools into a cohesive and scalable system that met our objectives required significant effort. Additionally, introducing new methods to the established HPC infrastructure was complex, as it involved extensive testing and validation on experimental platforms. Ensuring that our new approaches were both effective and safe required considerable perseverance and collaboration.

How is the development of such a project supporting the ambition of the EuroHPC JU to make Europe a world leader in supercomputing?

REGALE supports the EuroHPC JU’s ambition by promoting an open and flexible approach to supercomputing. This approach supports platform independence and avoids dependency on specific technologies. By leveraging European expertise, REGALE strengthens Europe’s role in global supercomputing and contributes to advancing more sustainable and efficient computing solutions.

What’s next for the REGALE project and the results developed under this project?

The tools and architecture developed through REGALE are being adopted by several supercomputing facilities and research institutions. Our partners, including institutions from TUM (Technische Universität München), ICCS (Institute of Communication and Computer Systems), and others, are continuing to develop and deploy these outcomes.

We anticipate further collaborations and the integration of our solutions into operational systems. Moving forward, we are exploring new opportunities to build on our achievements and continue advancing the field of HPC.