kubernetes
kubernetes
Building a Single-Pane RED Dashboard for Microservices with OpenTelemetry and Semantic Conventions
Three months ago we had 34 microservices and 34 Grafana dashboards, each one hand-built by whoever owned that service at the time. Some had rate/error/duration panels. Some had CPU and memory but no latency. One had a pie chart of HTTP status codes that nobody had looked at in a year. When we had an incident that touched five services at 2am, the on-call engineer had to open five different dashboards, each with different label names (route vs path vs endpoint), different histogram bucket boundaries, and different naming for the same metric.
How We Cut Our Prometheus / VictoriaMetrics Storage Bill by 60% (Cardinality Audit Walkthrough)
The bill that made me open a support ticket with myself Last October our VictoriaMetrics cluster crossed 340 million active time series and our monthly infra cost for the storage tier hit $4,900. Nobody had approved that number. It just… grew, the way disk usage always grows, one Helm chart install at a time until finance asks why the “monitoring” line item is bigger than the “database” line item. I spent the better part of a week doing a cardinality audit on that cluster.
Scaling a real-time metrics pipeline 50x
Rebuilt Ubisoft's monitoring pipeline to handle 3.3M datapoints/sec, up from 66k/sec.
Monitoring aggregation platform for game studios
A Go/Kafka-based platform aggregating monitoring data across game studios' production titles.