SE-Radio Episode 277: Gil Tene on Tail Latency

Response Time

Understanding response time in distributed systems is complex due to the aggregation of multiple components. Gil Tene explains that achieving low latency requires examining higher percentiles of response times, as human interactions often involve multiple database calls and services 1. He highlights the importance of considering the distribution of arrival times, especially in systems like financial services where spikes in demand can occur 2.

If what you show me involves 100 things, then you need two more nines in. You need to look two more nines deep into the thing than what you want me to do.

--- Gil Tene

This approach ensures that systems can handle peak loads without degrading user experience.

   

Queues

Queues play a crucial role in determining response times and system latency. Gil Tene describes how wait times are often the result of queuing at various stages, from the initial request to the final response 3. These queues can exist at multiple points, such as network sockets or internal system processes, impacting overall latency.

The wait time functioning is the time from when you wanted the service to happen until the service started to do its work.

--- Gil Tene

Understanding these queues helps in optimizing system performance and reducing latency.

   

Latency

Latency is defined as the time between an input event and its resulting output, encompassing both service and wait times. Gil Tene emphasizes that while response time is observable, the internal service and wait times significantly affect system behavior 4. These components are crucial for modeling and improving system performance.

Latency is a reaction time. It's a time from when one event happened until some other event that results from it happens.

--- Gil Tene

Recognizing the difference between these times aids in better system design and user experience.

   

Tail Latency

Tail latency focuses on the high percentiles of latency performance, which are more significant than average measurements. Gil Tene explains that humans are more affected by these rare, longer response times than by average cases 5. This is because even a single slow response can disrupt user experience, leading to potential revenue loss.

Humans do not perceive a common case. Humans do not forgive you because your average is good.

--- Gil Tene

Thus, addressing tail latency is crucial for maintaining a smooth and reliable user experience.