SE-Radio Episode 270: Brian Brazil on Prometheus Monitoring

Data Loss

Prometheus's approach to handling data loss prioritizes availability over consistency. Brian Brazil explains that during network partitions, data may be lost as Prometheus opts not to backfill, avoiding additional load that could exacerbate outages. This strategy is suitable for most services, but high-sensitivity applications may require more consistent monitoring solutions 1.

If you're not okay with a small blip in data, well, we can monitor the system that's doing that and tell you how it's generally performing with all the metrics.

--- Brian Brazil

Integration with existing systems often begins with Prometheus as an advisory tool, gradually taking on more roles as organizations adapt their processes 2.

   

Data Durability

In Prometheus, data durability is managed through engineering trade-offs that favor availability. Brian Brazil highlights that Prometheus avoids consensus systems like Zookeeper to prevent dependencies that could fail during network issues. Instead, each server operates independently, ensuring continuous monitoring even if some nodes fail 3.

Consensus is great when you need the consistency, like you can imagine for billing data is probably the best example where you need everything to be perfect.

--- Brian Brazil

The querying model uses an HTTP API, allowing for flexible data retrieval and visualization, which supports Prometheus's focus on service-level metrics over instance-specific data 4.