Thorsten discusses how user behavior can be unpredictable, even within a deterministic ranking system. He emphasizes the importance of understanding the effective sample size when evaluating new systems, particularly when the logged data differs significantly from the current system. The conversation highlights practical limitations in evaluation and the need for diagnostics to ensure reliable results.
