Multimodal foundation model for German property invoice checking

AI Technology: Natural Language Processing; Vision (image recognition, image generation, text recognition OCR, etc.).

The project company is a service provider in the insurance sector and specializes in invoice checking for property damages. 

With their current computer vision and natural language processing models, they already manage to process 30% of incoming claims automatically.

Part of the model family is a small-sized language foundation model for semi-structured documents that is specialized for processing German craft language and dealing properly with numbers. 

In an FFplus innovation study that has been granted to the project, the team wants to substantially improve this foundation model by adding multimodality and combining it with a Llama 3.1 to give it some question-answering and explanation generation capabilities. 

This will substantially increase the project's automation rate by improving the detection of not-ok invoices and by automatically generating explanations for not-ok verdicts.