The Hidden Cost of High-Cardinality Labels - And How to Find Yours Before Your Bill Does

The morning our Prometheus pod OOM-killed itself three times before standup

It was a Tuesday. Our prometheus-server pod had restarted three times overnight, each time taking about four minutes to replay the WAL before it could serve queries again. Nobody had touched the alerting rules. Nobody had deployed a new dashboard. The only thing that had changed was a routine deploy the day before that added a user_id label to an HTTP request counter — “just for debugging,” the PR said.

That one label turned a 12-million series TSDB into a 340-million series TSDB in about six hours. Memory usage went from 6GB to 38GB. The instance had a 32GB limit. Game over.

This is the part of Prometheus and VictoriaMetrics operation that nobody warns you about in the getting-started docs: cardinality is the thing that actually determines your bill, not the number of metrics you define, not your scrape interval, not your retention window (though those matter too). One bad label can cost you more than a hundred well-designed ones combined. This post is about finding that label before it finds your on-call rotation.

The one-paragraph version

Cardinality is the number of unique time series a metric produces, and it’s driven by the combination of every label value, not any single label alone. A counter with 5 label keys, each with 10 possible values, doesn’t have 50 series — it has up to 10^5 = 100,000. The fix is never “add more RAM.” The fix is finding which labels are unbounded (user IDs, request IDs, raw URL paths, pod IDs after every rollout) and either dropping them, aggregating them at scrape time, or moving that data to logs/traces where high cardinality is cheap and expected. Do this with prometheus_tsdb_head_series growth tracking and a cardinality audit query — both below — before you do it with a bill.

A runnable starting point: find your top cardinality offenders right now

If you run Prometheus with the TSDB admin API enabled (--web.enable-admin-api), this is the fastest path to an answer:

# Top 15 label names by number of distinct values across all series
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.labelValueCountByLabelName' \
  | jq 'to_entries | sort_by(-.value) | .[0:15]'

Sample output from the incident above:

[
  { "key": "user_id", "value": 284213 },
  { "key": "pod", "value": 4110 },
  { "key": "request_path", "value": 9982 },
  { "key": "instance", "value": 340 }
]

user_id at 284k distinct values on a single label was the whole story. Nothing else came close.

If you’re on VictoriaMetrics, the equivalent is built into the UI and is honestly better:

curl -s "http://localhost:8428/api/v1/status/tsdb?topN=15" | jq .

VictoriaMetrics also exposes a /api/v1/status/tsdb breakdown by metric name and label, which Prometheus’s native endpoint doesn’t do as cleanly — worth knowing if you’re choosing between the two for a cardinality-heavy environment.

Step 1: quantify the blast radius with PromQL, not guesswork

Before touching anything, measure how many series a specific metric is contributing versus the whole database:

# Total active series in the head block
prometheus_tsdb_head_series

# Series contributed by one metric name
count(count by (__name__)({__name__="http_requests_total"}))

That second query is misleading on its own — it just tells you the metric exists. What you actually want is the series count per metric, which requires a different approach since PromQL can’t natively “count series matching a name” without expanding them:

count({__name__="http_requests_total"})

Run that before and after a suspected offending label is added. In our case:

before: http_requests_total -> 1,840 series
after:  http_requests_total -> 312,000 series

That’s your smoking gun. I’ve made the mistake of trying to eyeball this from Grafana’s “Explore” panel — don’t. At high cardinality the query itself times out before it renders, which is its own confirmation that something’s wrong, but it wastes ten minutes you don’t have during an incident.

Step 2: rank labels by their marginal contribution, not their total count

Total distinct values per label (from step above) tells you what’s big, but not what’s exploding your specific metric. This query isolates cardinality contribution per label key within one metric:

count(count by (user_id) (http_requests_total)) 

Repeat per suspect label and compare against the metric’s total series count. If count by (user_id) returns nearly the same number as the metric’s total series count, that label is effectively unique per request — it’s not a label, it’s an identifier, and it never belonged in a metric.

Step 3: fix it at the source with relabeling, not downstream aggregation

The tempting fix is to keep the label and aggregate it away in recording rules. That works for dashboards, but it doesn’t stop the ingestion cost — the raw series still get written and stored before your recording rule ever runs. You have to drop it before it hits the TSDB.

# prometheus.yml scrape_config — drop the offending label at scrape time
scrape_configs:
  - job_name: 'checkout-service'
    metric_relabel_configs:
      - source_labels: [user_id]
        regex: '.*'
        action: labeldrop
        # applies to the label 'user_id' regardless of value
      - source_labels: [request_path]
        regex: '/api/v1/users/[0-9]+'
        target_label: request_path
        replacement: '/api/v1/users/:id'
        action: replace

That second rule is the one that actually mattered for us in the long run. request_path had ~10,000 distinct values, but almost all of them were /api/v1/users/12345-style paths that varied only by ID. Instead of dropping the label entirely (which loses useful route-level breakdowns), we normalized it with a regex so /api/v1/users/12345 and /api/v1/users/67890 collapse into one series.

Result: http_requests_total went from 312,000 series back down to 1,900. Memory usage on the Prometheus pod settled at 5.8GB, close to where it was before the bad deploy.

Step 4: catch the next one before it ships, not after

The whole point is to not do this reactively again. Two guardrails, both cheap:

A limit on ingestion itself, so a bad deploy degrades gracefully instead of OOM-killing the whole instance:

# prometheus.yml global config
global:
  # reject scrapes that would push a single target over this series count
  # (Prometheus 2.45+)
storage:
  tsdb:
    out_of_order_time_window: 10m

# per-scrape-job sample limit — this is the one that actually saved us
scrape_configs:
  - job_name: 'checkout-service'
    sample_limit: 5000

sample_limit doesn’t prevent bad labels from being written, but it does cause the entire scrape to fail loudly with a clear error in the Prometheus targets page instead of silently ballooning memory. I’d rather have a service go dark in monitoring for ten minutes with a screaming alert than a slow OOM death spiral at 3am.

An alert on series growth rate, so you get paged on the trend, not the outage:

groups:
  - name: cardinality-guardrails
    rules:
      - alert: TSDBSeriesGrowthAnomalous
        expr: |
          (
            prometheus_tsdb_head_series
            -
            prometheus_tsdb_head_series offset 1h
          ) > 50000          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "TSDB head series grew by more than 50k in the last hour"
          description: "Current: {{ $value }} new series. Check recent deploys and metric_relabel_configs for new unbounded labels."

This is the kind of alert I wish existed before our incident. It’s not glamorous, and it will occasionally fire on legitimate cluster scale-up events — we tuned the threshold twice before it stopped false-paging during normal autoscaling. That’s a fair tradeoff for catching a runaway label in 10 minutes instead of 10 hours. For more on tuning alert thresholds without drowning your team in noise, see the alert fatigue post — the same threshold-tuning discipline applies here.

Decision table: what to do with a high-cardinality label once you’ve found one

Troubleshooting: three failure modes I’ve actually hit

“My relabel_configs are correct but cardinality isn’t dropping." Check whether you edited relabel_configs instead of metric_relabel_configs. The former applies before scraping (affects target discovery), the latter applies to samples after they’re scraped. This is the single most common config mistake I’ve seen — including my own, twice.

“sample_limit is rejecting scrapes I didn’t expect." sample_limit counts total samples in a scrape, not per-metric. If one metric on a target explodes, the whole target’s scrape fails, including metrics that were fine. Check up{job="..."} and the Prometheus targets page for the specific error — it’ll say sample_limit exceeded. This is a blunt instrument; it protects the whole instance at the cost of losing all metrics from that target during the bad window.

“Cardinality dropped but query latency didn’t improve." You likely fixed ingestion but not historical data — the exploded series are still sitting in old blocks until they age out per your retention window. prometheus_tsdb_head_series will look great immediately; disk-based query performance on a 15-day range won’t improve until those old blocks are compacted away. If you need the fix retroactive, you have to delete the series explicitly:

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__="http_requests_total",user_id=~".+"}'
curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

Do this on a non-production instance first. delete_series is one of those APIs that works exactly as advertised and that’s exactly why it’s dangerous — there’s no undo.

What “done” looks like

You’re done, for now, when:

• prometheus_tsdb_head_series growth per hour stays under a known baseline (ours is 5,000/hr under normal deploy cadence) with an alert on anomalous growth
• No metric has a label whose cardinality scales with request volume instead of service topology — verified by running the step 2 query against your top 10 metrics by series count
• sample_limit is set on every scrape job, sized to roughly 2x normal volume, so a bad label degrades one target instead of the whole instance
• You can answer “which label is our most expensive one” in under two minutes using the TSDB status endpoint, not a 20-minute Grafana query that might time out

That last one is the real test. If finding your worst label takes longer than fixing it, you’re going to find it during an incident instead of during a code review, and one of those is much more expensive than the other. For a broader view of where else infrastructure monitoring budgets quietly leak — retention, scrape intervals, remote-write duplication — the monitoring infrastructure guide covers the adjacent cost levers this post doesn’t.

FAQ

What counts as “high cardinality” in Prometheus? There’s no universal number, but as a working threshold: a single metric with more than 10,000-50,000 active series is worth investigating, and anything in the millions on a single-instance Prometheus will cause real memory pressure. The Prometheus docs on cardinality recommend keeping label values bounded and predictable — user IDs and raw URLs fail that test immediately.

Does VictoriaMetrics handle high cardinality better than Prometheus? It handles the storage layer more efficiently (better compression, less memory per series due to its MergeTree-inspired storage engine) and its cardinality diagnostics UI is more usable out of the box. But it doesn’t eliminate the underlying problem — an unbounded label still generates unbounded series, and unbounded series still cost money and query latency regardless of backend.

Can recording rules fix cardinality problems? No, and this is the mistake I made first. Recording rules run after ingestion, so the exploded raw series still get written to disk and consume memory before your rule ever aggregates them. Recording rules reduce query-time cost, not ingestion or storage cost. You have to drop or relabel at scrape time.

How do I find which team or service added a bad label? git blame on the scrape config or exporter code is the honest answer, but if you don’t have that visibility, correlate the prometheus_tsdb_head_series growth spike timestamp against your deployment log (ArgoCD, Flux, or CI/CD history) — whatever changed in that window is your suspect.

Will dropping a label break dashboards that use it? Sometimes, and you should grep your Grafana dashboard JSON and alerting rules for the label name before dropping it in production. We broke one dashboard panel that grouped checkout errors by user_id for fraud investigation — we had to move that specific use case to log-based queries in Loki instead, which was the right home for it anyway.

Is cardinality a cost problem even on managed/cloud Prometheus services? More so, usually — hosted TSDB-as-a-service pricing is frequently billed per active series or per sample ingested, so an unbounded label shows up directly on your invoice within a billing cycle instead of as a slow memory creep you might not notice for weeks.