Software Engineering Radio - the podcast for professional software developers avatar

Dexa/Software Engineering Radio - the podcast for professional software developers

Learn more
Published Sep 3, 2019

SE-Radio-Episode-282-Donny-Nadolny-on-Debugging-Distributed-Systems

Donny Nadolny from PagerDuty shares expert insights into debugging distributed systems, discussing the intricacies of TCP connections, the challenges of maintaining Zookeeper clusters, and innovative techniques that emphasize persistence and cross-disciplinary collaboration for effective problem-solving.
Episode Highlights
Software Engineering Radio - the podcast for professional software developers logo

Topics covered

Popular Clips

Episode Highlights

  • Cluster Ops

    The operation of Zookeeper clusters is crucial for maintaining system consistency. explains that Zookeeper clusters are typically run with three or five nodes, with a leader elected to manage all write operations. This setup ensures high availability, as any node can fail without data loss, thanks to the replication of writes across a majority of nodes 1. However, issues can arise, such as when the cluster appears operational but is unable to perform tasks, as experienced by at PagerDuty 2.

    The processes were running, but if you try to do a read or write, then it would just time out.

    ---

    This highlights the importance of understanding the intricacies of cluster operations and the potential for hidden issues.

       

    Health Checks

    Health checks in Zookeeper can be misleading, as they may indicate the system is healthy when it is not. points out that the standard "RuOK" check only confirms that the port is open, not that the system can perform its necessary functions 3. He suggests implementing deeper health checks that perform actual operations to verify system health 4.

    It's very easy to make kind of a naive health check that just replies and says that you're healthy as long as the process is up.

    ---

    This approach can help identify issues that basic checks might miss, ensuring more reliable system performance.

       

    Debugging

    Debugging Zookeeper involves identifying and resolving complex issues that may not be immediately apparent. shares that adding more intrusive health checks helped uncover a stuck state in the cluster, allowing them to capture a stack trace and understand the problem 5. Reproducing bugs is essential for effective debugging, and Nadolny6.

    One of the key things that you want is obviously to be able to reproduce a bug.

    ---

    These strategies highlight the importance of thorough monitoring and testing in maintaining system reliability.

Related Episodes