Process Mining with LLMs

Metrics

David Obembe explains the metrics used to evaluate LLMs in process mining, focusing on precision, recall, and accuracy. He notes that precision and recall are crucial for assessing the completeness and correctness of answers, while accuracy is used for SQL queries 1. The performance of LLMs like GPT-4 and Claude V3 is compared, revealing trade-offs in precision and recall. David highlights that smaller prompts yield better results, as larger prompts can decrease performance 2.

GPT-4 would rather give you correct answers than incorrect answers, but while doing that, it misses out on maybe other correct answers.

--- David Obembe

This insight underscores the importance of balancing prompt size and context in process mining.

   

Efficiency

David contrasts the efficiency of LLMs with human analysts in process mining tasks. He acknowledges that while both can make mistakes, LLMs offer a faster time to value by quickly identifying bottlenecks and improvement opportunities 3. This speed advantage is significant, as human analysts require more time to interact with tools and models. David also discusses the potential of Retrieval Augmented Generation (RAG) to enhance LLMs' contextual understanding of business processes.

With LLMs, the time to value is shorter. It's as easy as asking, what are the bottlenecks in my process?

--- David Obembe

This development could lead to more tailored and efficient process mining solutions.