Process Mining with LLMs

LLM Integration

David Obembe explores the integration of Large Language Models (LLMs) with process mining tools to enhance efficiency. He describes two approaches: the direct prompt method, which involves feeding entire tables into the LLM, and a schema-based method using SQL databases. This innovation aims to streamline data analysis and improve prompt engineering, ensuring that LLMs provide accurate insights into business processes 1.

The crux of the work was focused on prompt engineering, because so we need to ensure that the answers we're getting, you know, were what we wanted.

--- David Obembe

Obembe's thesis also highlights the potential of conversational interfaces in process mining, reducing the complexity of navigating process maps and enhancing user interaction 2.

   

Engineering Challenges

Integrating LLMs with process mining tools presents several engineering challenges, particularly in prompt consistency and performance. Obembe notes that as prompts grow larger, LLMs like GPT-4 and Claude V3 may struggle with consistency, sometimes generating incorrect answers 3. To address this, experiments were conducted simultaneously across different LLM versions to ensure reliability.

We found out that perhaps it was returning answers that were never in the prompt, and it was a little strange.

--- David Obembe

Additionally, Obembe discusses the trade-offs between precision and recall in LLM outputs, emphasizing the importance of balancing detail with accuracy in process mining applications 4.