Predicting extreme weather events with HPC and AI: the MAELSTROM project

Weather and climate significantly impact human safety, health, and economy. Better forecast predictions could help manage the effects of short-term weather events and long-term climate change. Current weather prediction models still struggle to provide reliable forecast predictions of extreme weather events that is more than a week old. At the same time, climate prediction systems often fail to accurately predict local weather pattern changes due to climate change. 

MAELSTROM, an EuroHPC JU-funded project, aims at overcoming these limitations by harnessing the combined power of Artificial Intelligence (AI) and HPC to achieve a change in forecasting precision and reliability.

Since its start in April 2021, the MAELSTROM project has brought together seven organisations including research institutes, universities, and SMEs from six European countries with expertise in weather and climate prediction. With a budget of €4,312,412.50, this collaboration has set new standards in the field and developed new solutions that could impact human safety, health and economy in the face of climate change and extreme weather events. 

EuroHPC Joint Undertaking (EuroHPC JU) interviewed Peter Dueben, Head of the Earth System Modelling Section at the European Centre for Medium Range Weather Forecasts (ECMWF), which is a partner institution in the project, to highlight MAELSTROM's key achievements:

Can you please describe the MAELSTROM project in your own words?

Sure! MAELSTROM is a project that has brought together world-leading researchers in weather and climate modeling, as well as experts in supercomputing and machine learning. Their goal is to improve how Europe's supercomputers are used to develop and run advanced machine learning applications for Earth sciences.

What have been the key objectives for the project and what progress has been made?

The key objectives of the MAELSTROM project were to develop efficient software tools for creating weather and climate prediction applications, and explore optimal hardware configurations for training and running these. Over the three years of the project, we have achieved astonishing progress. Our machine learning tools have grown exponentially in capability and scale, evolving from experimental models into production-ready applications that help improve weather forecasts today.

Can you give some concrete examples of how your project supports European HPC users and how it promotes greener and more sustainable supercomputing?

The project has developed benchmark datasets, machine learning solutions, and software for a number of machine learning applications in weather and climate science. These are now available for download and are used by the community. 

In addition, we have acquired knowledge on how to run machine learning applications efficiently on modern supercomputers, especially on EuroHPC systems, will which if share across the community, will generate a lasting legacy for the project. The project has improved the efficiency and quality of machine learning solutions, with some results already being used for operational weather predictions that benefit European society.

Finally, we are proud to say that our efforts to improve energy efficiency in supercomputing, based on feedback from developers and performance benchmarks, will help reduce the energy and computing demands of our community as it prepares to use vast resources in the coming years.

What were the main challenges you encountered during the project's development, if any?

The main challenge was the pace of developments in the domain of machine learning in the last couple of years. Some of the questions and challenges that were stated in our project’s proposal, have changed significantly over the lifetime of the project. 

Furthermore, developments around machine learning within Earth sciences have sometimes distracted us from the project goals. For instance, while pure machine-learned weather forecasts were performing really well compared to traditional models, some of the simpler machine learning tools, which only replicate certain parts of the forecasts, haven’t been as effective. Nevertheless, we could easily tackle this challenge with some minor adjustments to the project focus, and the project achievement could overall exceed the original expectations regarding readiness of tools for operational use. 

How is the development of the MAELSTROM project supporting the ambition of the EuroHPC JU to make Europe a world-leader in supercomputing?

Europe leads the world in weather and climate modeling, especially with high-resolution simulations on the largest EuroHPC supercomputers. However, recent advancements in machine learning for weather predictions have largely been driven by major tech companies from the US and Asia. 

Projects such as MAELSTROM help to keep pace with the large machine learning efforts outside of Europe. This will help Europe maintain its leadership in Earth system applications and fully leverage the power of advanced supercomputers.

What’s next for your project and results developed under this project?

Over the last three years, MAELSTROM has built a European community of knowledge around supercomputing, machine learning and weather and climate modelling. This community will keep developing machine learning applications to enhance weather and climate predictions for the benefits of society and will continue to use the software and hardware advancements made during this project. 

The legacy of MAELSTROM will be key for creating advanced AI systems like the Artificial Intelligence Forecasting System, which uses machine learning to improve weather and climate predictions, and for developing foundational machine learning models in Europe.