Interview: the ACROSS Project Pioneers HPC, Big Data, and AI Integration for Enhanced Applications

The ACROSS project, successfully concluded in February 2024 after three years. Officially known as “Advanced Convergence in High-Performance Computing and Big Data for Optimal Resource and Application Support,” the project developed an innovative platform that merges High-Performance Computing (HPC), Big Data, and Artificial Intelligence (AI) to enhance applications in aeronautics, climate research, and energy sectors.

The project was a collaborative effort involving a diverse consortium of partners from various European countries. The partnership included institutions from Italy, France, Czechia, the United Kingdom, Norway, Greece, the Netherlands, and Germany. This multinational research team brought together a wide range of expertise to create a versatile and efficient platform that can handle complex and data-intensive applications. 

In this interview, project coordinators Olivier Terzo and Alberto Scionti share insights into the accomplishments of ACROSS, the challenges faced, and future directions for the project.

Can you please describe the ACROSS project in your own words?

The ACROSS project is all about developing a cutting-edge platform that combines HPC, Big Data, and AI. Our goal is to enhance the capabilities of supercomputing systems so they can handle complex applications more efficiently. By integrating these technologies, we decided to support critical sectors like aeronautics, climate research, and energy, making it easier to process large datasets and solve intricate problems.

We focused on creating a platform that not only performs well but also seamlessly integrates Big Data and AI with traditional supercomputing. This integration is key to addressing the sophisticated challenges in our target sectors, where the need for advanced data processing and computational power is growing rapidly.

What have been the key objectives for the project and what progress has been made?

Our main objectives were to advance the integration of HPC, Big Data, and AI into a unified platform and to enhance the efficiency of applications across these domains. Over the life time of the project, we developed a robust platform that supports modular and scalable operations. We also created prototypes and tested them extensively to ensure they meet our performance and flexibility goals.

Additionally, we focused on improving application performance by incorporating advanced AI techniques and Big Data tools. This has enabled us to enhance several pilot applications, making them more efficient and adaptable. Our progress includes successful integration of these technologies into practical, real-world scenarios.

Can you give some concrete examples of how the ACROSS project supports European HPC users and promotes greener supercomputing?

Historically, supercomputers have been used to perform the most demanding simulations, covering a wide spectrum of application domains (pharmaceutical, energy, weather forecasts, engineering systems design, etc.). With AI now gaining momentum, all these application domains started to exploit the benefit of integrating AI models in their use cases, which has further increased the challenge of  using compute resources efficiently. 

To tackle these challenges, ACROSS simplified the process of deploying and managing complex applications by providing a unified platform that integrates HPC, Big Data, and AI. This is particularly beneficial for users who are not HPC experts, as it reduces the complexity involved in managing these advanced systems.

On the sustainability front, our project focuses on optimising the energy efficiency of supercomputers. By developing techniques to manage and reduce energy consumption, the environmental impact of HPC systems was reduced. This was done by implementing energy-efficient practices and tools that support greener computing solutions.

What were the main challenges encountered during the project's development?

There were some challenges connected to the development of the software stack, that we addressed. The first challenge was connected to the access to an infrastructure where all the software components could be installed and tested. 

A second challenge was connected to the unavailability of real hardware, where to test our project ideas. To this end, we approached the problem by using high-end FPGA devices and by adapting State-of-the-Art open-source projects. This allowed us to experiment more easily with spiking neural models and to compare their performance/energy ratio with traditionally artificial neural models.   

How does the ACROSS project align with the EuroHPC JU’s goal of making Europe a world leader in supercomputing?

EuroHPC-JU is supporting  the HPC community in Europe by adopting new first-class supercomputers, specifically looking at the integration of the forefront technologies (which include also Quantum Computing systems). 

In our opinion, the challenges addressed by the ACROSS project, and the innovative outcomes well support these EuroHPC-JU actions. Indeed, the next generation of supercomputers will be more and more complex, requiring flexible and modular software to orchestrate the ever-complex workflows and optimize the usage of compute resources. This is exactly the purpose of the co-designed and implemented ACROSS orchestration software stack.

Furthermore, the EuroHPC-JU is also promoting the creation of platforms that can allow users to seamlessly access resources on different machines; the ACROSS platform has been designed since the beginning to be easily integrated with such kind of federative platforms. 

What’s next for the ACROSS project and its results?

The tools and technologies developed through ACROSS are already being adopted by various supercomputing facilities and research institutions. Our partners are eager to continue building on these outcomes and integrating them into their systems.