How We Cut Our Prometheus / VictoriaMetrics Storage Bill by 60% (Cardinality Audit Walkthrough)

The bill that made me open a support ticket with myself

Last October our VictoriaMetrics cluster crossed 340 million active time series and our monthly infra cost for the storage tier hit $4,900. Nobody had approved that number. It just… grew, the way disk usage always grows, one Helm chart install at a time until finance asks why the “monitoring” line item is bigger than the “database” line item.

I spent the better part of a week doing a cardinality audit on that cluster. Not a rewrite, not a migration, just going metric by metric and asking “does anyone query this, and if so, do they need every label value?” By the end we were at 128 million series and $1,950/mo — a 60% cut — without losing a single dashboard or alert that anyone actually used.

This post is the walkthrough, including the two things I tried that didn’t work and the one metric that turned out to be 40% of our entire cardinality budget by itself.

The one-paragraph version

Storage cost in Prometheus/VictoriaMetrics scales almost linearly with active series count, not with query volume or dashboard count. Most cardinality explosions come from a small number of metrics with unbounded label values — pod names before ReplicaSet hashes get stripped, request paths with IDs in them, user emails, full URLs. Find the top 20 metrics by series count, look at which labels are driving that count, and either drop the label at scrape time, aggregate it away in recording rules, or drop the metric entirely if nobody’s using it. You’ll get 80% of the win from fixing 3-5 metrics. The remaining 20% is slower, more political work — asking teams why they need per-pod histograms for a service with 400 replicas.

A runnable starting point

Before touching anything, get a baseline. This query works on both Prometheus and VictoriaMetrics:

topk(20, count by (__name__)({__name__=~".+"}))

That tells you which metric names have the most series. It’s necessary but not sufficient — a metric can have a modest series count and still be expensive if its labels are high-cardinality strings. So pair it with VictoriaMetrics’s built-in cardinality explorer, which is genuinely one of the best things about running VM over vanilla Prometheus:

curl -s "http://victoria-metrics:8428/api/v1/status/tsdb" | jq '.data.seriesCountByMetricName[:20]'

If you’re on plain Prometheus without that endpoint, use promtool:

promtool tsdb analyze /prometheus/data --limit 20

This gave us our starting list. The top offender wasn’t even close: http_request_duration_seconds_bucket with 54 million series, roughly 16% of the entire cluster from one metric.

Step 1: find which labels are actually driving the cardinality

Knowing the metric name isn’t enough — you need to know which label is exploding it. This is the query I run against every candidate metric:

count(count by (le, path, method, pod) (http_request_duration_seconds_bucket)) 

Then I break it down label by label to isolate the culprit:

# how many distinct values does each label have?
count(count by (path) (http_request_duration_seconds_bucket))
count(count by (pod) (http_request_duration_seconds_bucket))
count(count by (le) (http_request_duration_seconds_bucket))

In our case, path came back with 340,000 distinct values. That’s the tell. Nobody writes an API with 340,000 routes — this was a Gin/Echo service that hadn’t set up route templating, so path contained the raw URL including /users/48291/orders/77103 instead of /users/:id/orders/:id. Every unique user/order pair became a brand new time series that lives forever in the TSDB until retention kicks it out.

Step 2: fix cardinality at the source, not at the query layer

My first instinct was to fix this downstream with a recording rule that pre-aggregates away path. That works for dashboards, but it doesn’t reduce ingestion cost — VictoriaMetrics still has to ingest and store all 54 million raw series before your recording rule runs on top of them. Recording rules save you query-time cost, not storage cost. This was mistake #1 and it cost me half a day before I checked the actual series count and realized nothing had changed.

The real fix has to happen either in the application (route templating) or at scrape/relabel time. We couldn’t get the app team to ship a code fix that week, so we did it with metric_relabel_configs to drop the label entirely as a stopgap:

scrape_configs:
  - job_name: 'api-service'
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        target_label: path
        replacement: 'aggregated'
        action: replace

This collapses every path value into a single label value at ingest time, before it ever hits storage. It’s blunt — you lose per-route latency breakdowns entirely until the app team ships proper route templating — and I want to be upfront that this is a real regression in observability, not a free win. We accepted it for three weeks. The proper fix landed when the app started emitting a route_template label instead of raw path, and we reverted the relabel rule.

Two weeks later, once the app fix shipped, cardinality on that one metric went from 54M to 1.8M series just from templating the routes properly. That’s the fix you actually want; the relabel drop is just to stop the bleeding.

Step 3: kill metrics nobody queries

Cardinality reduction isn’t just about high-cardinality labels — some metrics are just dead weight. We used Grafana’s Prometheus data source query inspector logs plus this trick against our query log to find metrics that hadn’t been touched by any dashboard or alert in 30 days:

# VictoriaMetrics exposes query stats if you enable -search.logSlowQueryDuration
grep -oP '(?<=metric":")[a-zA-Z_:]+' /var/log/vmselect/query.log | sort | uniq -c | sort -rn > queried_metrics.txt

# compare against the full metric list
comm -23 <(curl -s http://victoria-metrics:8428/api/v1/label/__name__/values | jq -r '.data[]' | sort) \
         <(sort queried_metrics.txt) > unused_metrics.txt

unused_metrics.txt had 340 entries. Most were leftover from decommissioned services, old cAdvisor container labels, and a Kafka exporter someone installed for a POC in 2023 that nobody removed. We dropped these entirely at the scrape config level:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'kafka_consumer_lag_.*|old_poc_.*|cadvisor_container_.*'
    action: drop

This alone was another 22 million series gone, with zero risk — if nothing has queried a metric in 30 days across dashboards, alerts, and recording rules, it’s very unlikely to be load-bearing. We kept the drop list in version control and gave it a two-week soak period with alerts on absent() before fully committing, in case something queried quarterly instead of monthly.

Step 4: bound cardinality with per-metric limits, not just cleanup

Cleanup is reactive. To stop this from happening again, VictoriaMetrics has a flag that caps ingestion per metric name — it won’t silently let a new metric blow up your series count the way http_request_duration_seconds_bucket did:

./vmstorage \
  -storageDataPath=/data \
  -search.maxUniqueTimeseries=50000000 \
  -maxLabelsPerTimeseries=30

And at the scrape/relabel level, we added a hard cap on the total series any single job can produce, using Prometheus’s native sample_limit:

scrape_configs:
  - job_name: 'api-service'
    sample_limit: 20000

If a service starts emitting a runaway label value (a new deploy that adds a user_id label, for example), the scrape fails loudly instead of silently ballooning your TSDB. We’d rather get paged for a failed scrape than discover the damage during next month’s invoice.

Decision table: what to do with a high-cardinality metric

Troubleshooting

“I dropped the label but series count didn’t go down." Relabel drops only apply to new scrapes. Existing series in the TSDB stay until they age out via retention or you explicitly delete them:

curl -X POST -g 'http://victoria-metrics:8428/api/v1/admin/tsdb/delete_series' \
  --data-urlencode 'match[]={__name__="http_request_duration_seconds_bucket", path!="aggregated"}'

Run this only after you’re sure the relabel config is live everywhere, or you’ll delete series that are still being written from a lagging config rollout.

“Recording rules didn’t reduce my storage bill." Right — see mistake #1 above. Recording rules add series, they don’t remove the source series unless you separately drop them. If your goal is cost reduction, the recording rule and the relabel drop have to ship together.

“Cardinality dropped but query latency got worse." This happened to us on one dashboard after we collapsed the path label to aggregated. Grafana panels that used to filter by path now scan the same reduced series but with wider time ranges because the panel query no longer narrows by a specific route. Check your panel queries for label filters that assumed the old cardinality existed — they’ll silently return empty results or scan more data than before.

"sample_limit is rejecting a legitimate scrape." Check what changed with:

count by (job) ({__name__=~".+", job="api-service"})

against the same query from a week ago (use offset 7d or your historical dashboard). If it’s a real, justified increase — a new microservice version with genuinely more distinct routes — raise the limit deliberately rather than removing it.

What “done” looks like

You’re done with a cardinality audit — for now, it’s never permanently done — when:

• Your top 10 metrics by series count are all things you can name and justify in one sentence each
• count({name=~”.+"})` returns a number you actually know, not just a number that’s “probably fine”
• Every metric with 1M+ series has a name in a spreadsheet somewhere with the label(s) responsible written next to it
• You have a sample_limit or equivalent guardrail on every scrape job, so the next runaway label fails loudly instead of quietly landing on next month’s invoice
• Your drop list and relabel configs are in version control, not applied by hand and forgotten We’re not chasing zero waste — some cardinality is the cost of doing business, and a service with 400 real replicas legitimately needs per-pod visibility sometimes. The bar isn’t “smallest possible TSDB,” it’s “every series someone can explain.” If you can’t explain it, you probably can’t query it either, and you’re just paying rent on data nobody will ever look at.

If you’re setting up alerting on top of a cleaned-up metrics set like this, it’s worth reading through how burn-rate alerting works once cardinality stops getting in the way — see our infrastructure monitoring guide for the full alerting math. And if the next problem you hit is your team ignoring the alerts this cleanup was supposed to make trustworthy, that’s a separate fix — see why alert fatigue happens and how to fix it.