How We Cut Our Prometheus / VictoriaMetrics Storage Bill by 60% (Cardinality Audit Walkthrough)

The bill that made me open a support ticket with myself Last October our VictoriaMetrics cluster crossed 340 million active time series and our monthly infra cost for the storage tier hit $4,900. Nobody had approved that number. It just… grew, the way disk usage always grows, one Helm chart install at a time until finance asks why the “monitoring” line item is bigger than the “database” line item. I spent the better part of a week doing a cardinality audit on that cluster.