Enhancing Workflow Integration and Sustainability in HPC: Interview with the eFlows4HPC project

The eFlows4HPC project has successfully concluded after three years of development, which began in January 2021. The project aimed to deliver a workflow software stack and a set of services to integrate High-Performance Computing (HPC) simulation and modelling with big data analytics and machine learning (ML).

The project, which was coordinated by the Barcelona Supercomputing Center (BSC) in Spain, involved sixteen partners from seven countries: Spain, Germany, France, Italy, Poland, Switzerland, and Norway.

With an overall budget from Horizon Europe funding programme of €7,600,000, the eFlows4HPC project represents a significant effort in advancing HPC technologies. The project outcomes were expected to have a substantial impact on various fields that rely on HPC, data analysis, and machine learning.

We interviewed Rosa Badia from BSC, the coordinator of the eFlows4HPC project, to discuss its achievements and future implications.

Can you please describe the eFlows4HPC project in your own words?

The eFlows4HPC project developed a comprehensive platform and services that combine HPC simulations with big data analytics and machine learning. By integrating these advanced technologies, the research team aimed to improve workflow efficiency and productivity across various scientific and industrial fields, such as geosciences, climate science, and manufacturing. This platform helped to define complex workflows, enhancing their development and deployment.

What have been the key objectives for the project and what progress has been made?

In addition to developing a platform that integrates HPC with big data and machine learning, we built the eFlows4HPC software stack and the concept of High-Performance Computing Workflows as a Service (HPCWaaS). The platform facilitates the reusability of complex workflows in federated HPC infrastructure and provides methodologies and tools that enable sharing and reuse of existing workflows. This makes it easier for diverse user communities to leverage these technologies and adapting workflow templates to create new workflow instances.

We can now showcase the effectiveness of our innovations through three primary applications: manufacturing, climate modeling, and urgent computing for natural hazards. These case studies demonstrate how our advanced workflow technologies can facilitate the development of efficient HPC and data-centric solutions. To further support widespread adoption, we have released the eFlows4HPC software stack as open source, ensuring its versatility and broad applicability.

Can you give some concrete examples of how the eFlows4HPC project supports European HPC users and promotes greener and more sustainable supercomputing?

The software has been installed on EuroHPC systems at BSC and other European centres, such as the Euro-Mediterranean Center on Climate Change (CMCC) and the German Climate Computing Centre (DKRZ), making it available to a wide range of users. 

Our solutions help users develop, deploy, and execute complex workflows more efficiently. For instance, the UCIS4EQ workflow, a specialised workflow designed to enhance the simulation and analysis of seismic events, can adapt to earthquake scenarios in real-time, reducing unnecessary computations and saving energy. 

This dynamic approach involves starting computational workflows for initial earthquake scenarios and adjusting them based on real-time monitoring of the earthquake alert. Additionally, we developed a Container Image Creation (CIC) service that generates efficient, portable software environments tailored to specific HPC architectures, enhancing both portability and performance.

What were the main challenges you encountered during the project's development?

Starting the project during the COVID-19 pandemic meant adapting to remote work and virtual meetings. We organised a remote kick-off meeting and conducted the first training sessions as online seminars. 

Another challenge was the multidisciplinary nature of our consortium, which required us to find a common language to communicate effectively across different fields such as HPC and various application areas like manufacturing and climate science. 

This initial phase of adaptation was crucial for understanding the application requirements and aligning them with our technical components.

How does the eFlows4HPC project support EuroHPC's goal to make Europe a world leader in supercomputing?

eFlows4HPC provides tools and methods to manage complex workflows that combine HPC, data analytics, and artificial intelligence. This supports EuroHPC's mission by making it easier for users to leverage advanced supercomputers for more sophisticated applications.

As exascale systems become available, Europe is equipped with powerful and complex supercomputers. Our solutions help users maximise these resources by providing more complex application workflows and methodologies that support the integration of different technologies. This not only advances the capabilities of supercomputing but also educates the next generation of HPC programmers and informs a broader audience about the benefits and challenges of using EuroHPC supercomputers.

What’s next for your project and results developed under this project?

We have generated 14 key exploitable results from 47 results. Some of these results are already being used in other projects, such as DT-GEO, CAELESTIS, and InterTwin. 

Our manufacturing application results are being commercialised as SimTwins. The project results will be maintained through new competitive projects or contracts with industry. 

We also plan to continue refining our technologies and exploring new applications, ensuring that the advancements made during eFlows4HPC will have a lasting impact on the HPC community and beyond.