anomaly detection
anomaly detection
Detect Outliers in Your Metrics: A Practical Guide to Grafana Machine Learning
I’ve always been the kind of person who stares at a Grafana dashboard, watching a squiggly line, and wondering, “Is that dip normal? Should I be worried?” For years, my answer was to set up a basic threshold alert. If CPU usage goes above 90%, page someone. But what about the weird, subtle stuff? The slow creep of memory leakage or the sudden, inexplicable drop in request rate that doesn’t cross any static line but just feels… off?
Prometheus Anomaly detection: Z-Score in PromQL
Monitoring HTTP request rates is one of the most basic yet essential tasks in observability. A sudden spike might indicate a traffic surge or even a DDoS attack, while a sudden drop could signal a backend failure. Static thresholds work, but they often miss subtle patterns or raise too many false alarms. A better way is to use statistical anomaly detection—specifically Z-score based alerts in Prometheus. In this post, we’ll walk through how to set up a Z-score PromQL alert to detect anomalies in HTTP request rates using only Prometheus and native PromQL.