Building a Single-Pane RED Dashboard for Microservices with OpenTelemetry and Semantic Conventions

Three months ago we had 34 microservices and 34 Grafana dashboards, each one hand-built by whoever owned that service at the time. Some had rate/error/duration panels. Some had CPU and memory but no latency. One had a pie chart of HTTP status codes that nobody had looked at in a year. When we had an incident that touched five services at 2am, the on-call engineer had to open five different dashboards, each with different label names (route vs path vs endpoint), different histogram bucket boundaries, and different naming for the same metric. It took 20 minutes just to confirm which services were actually degraded, before anyone started fixing anything.

The fix wasn’t “build a better dashboard.” It was standardizing what every service emits, so one dashboard works for all of them. That’s what OpenTelemetry semantic conventions are for, and this post covers exactly how we wired it up — instrumentation, the Prometheus/OTel Collector pipeline, and the actual RED dashboard JSON.

The one-paragraph version

RED (Rate, Errors, Duration) only works as a single-pane dashboard if every service emits metrics with the same names and label keys. OpenTelemetry’s semantic conventions define standard names for HTTP/gRPC server metrics (http.server.request.duration, rpc.server.duration) and standard attribute keys (http.route, http.response.status_code, service.name). If you instrument with the OTel SDK and don’t rename things downstream, you get consistent metrics for free, and a single Grafana dashboard with a service variable replaces per-service dashboards entirely. The catch: you have to actually enforce the convention org-wide, because one team hand-rolling myapp_requests_total with a different label schema breaks the whole thing.

A runnable starting point

Here’s the PromQL that becomes the backbone of the whole dashboard, assuming OTel semantic conventions and the standard OTel Collector Prometheus exporter:

# Rate — requests per second, per service
sum by (service_name) (
  rate(http_server_request_duration_seconds_count[5m])
)

# Errors — error rate as a percentage of requests
sum by (service_name) (
  rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])
)
/
sum by (service_name) (
  rate(http_server_request_duration_seconds_count[5m])
)

# Duration — p99 latency
histogram_quantile(0.99,
  sum by (service_name, le) (
    rate(http_server_request_duration_seconds_bucket[5m])
  )
)

Run these against any OTel-instrumented service and you’ll get results without touching the query per service — that’s the entire point. Note the underscore conversion: OTel’s dotted attribute names (http.response.status_code) become underscored label names (http_response_status_code) after going through the Prometheus exporter, which trips people up the first time.

Step 1: Instrument with the OTel SDK, not custom metrics

The temptation is always to add a quick custom counter (myservice_requests_total{path="/api/users"}) because it’s five minutes of work. Don’t. Use the OTel SDK’s auto-instrumentation for your HTTP/gRPC framework so metric names and attributes come from the spec, not from whoever wrote that line of code.

Python example with FastAPI:

from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "checkout-service"})
reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="otel-collector:4317", insecure=True)
)
provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)  # emits http.server.request.duration automatically

That instrumentation call emits http.server.request.duration with http.route, http.response.status_code, and http.request.method attributes automatically, per the OTel HTTP semantic conventions. You didn’t name anything. That’s the win — nobody can misspell route as path because they never typed it.

The gap: auto-instrumentation only covers what the framework hooks into. Custom business logic (e.g., “how long did the fraud check take”) still needs a manual span/metric, and that’s where teams quietly reintroduce inconsistent naming. We caught two services doing this three weeks after rollout — a linter would have caught it sooner than a human review did.

Step 2: Collector pipeline — Prometheus format, one consistent scrape config

The OTel Collector receives OTLP from every service and exposes a Prometheus-scrapeable endpoint. This is the piece that actually normalizes everything into one queryable namespace.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch: {}
  resource:
    attributes:
      - key: service.name
        action: upsert
        from_attribute: service.name

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    resource_to_telemetry_conversion:
      enabled: true   # keeps service.name as a label, not just resource metadata

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [prometheus]

Then Prometheus scrapes the collector like any other target:

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

resource_to_telemetry_conversion: enabled: true matters — without it, service.name stays buried in resource metadata and never shows up as a label you can sum by (service_name) on, which means your dashboard variable dropdown comes back empty. I lost about 40 minutes to this exact setting on the first rollout, staring at metrics that existed in the collector’s internal telemetry but not in what Prometheus actually scraped.

Step 3: The dashboard — one JSON, service as a template variable

Grafana variable definition:

{
  "name": "service_name",
  "type": "query",
  "datasource": "Prometheus",
  "query": "label_values(http_server_request_duration_seconds_count, service_name)",
  "refresh": 2,
  "multi": true,
  "includeAll": true
}

Panel queries then filter on $service_name instead of hardcoding anything:

sum by (service_name) (
  rate(http_server_request_duration_seconds_count{service_name=~"$service_name"}[5m])
)

With includeAll and multi set, one dashboard shows rate/error/duration for every service simultaneously, or filters down to the five services involved in an incident. This is the panel set we standardized on:

We added saturation as a fourth column even though it breaks the “RED” acronym, because pure RED metrics don’t tell you why duration is climbing — sometimes it’s just a pod hitting its CPU limit, and that’s a five-second answer if saturation is already on screen instead of a second dashboard to open.

Step 4: Enforce the convention, or it rots in a month

This is the part nobody wants to do because it’s not a dashboard, it’s process. We added a CI check using a small Python script that scans OTLP export samples in staging and flags any metric name that doesn’t match the semconv registry:

import re

ALLOWED_PREFIXES = ("http.server.", "rpc.server.", "db.client.", "messaging.")

def check_metric_name(name: str) -> bool:
    return any(name.startswith(p) for p in ALLOWED_PREFIXES) or name.startswith("custom.")

# fails CI if a metric doesn't match convention and isn't explicitly namespaced as custom
violations = [m for m in exported_metric_names if not check_metric_name(m)]
if violations:
    raise SystemExit(f"Non-conventional metric names found: {violations}")

It’s crude — it won’t catch someone reusing http.server.request.duration with the wrong unit or bucket boundaries — but it caught three violations in the first month, all from services bolted on during a migration from an older StatsD-based stack. Full validation would mean checking against the actual OpenTelemetry semantic conventions schema, which is a bigger lift than we’ve done yet.

Troubleshooting

Dashboard variable dropdown is empty. Almost always resource_to_telemetry_conversion missing in the collector config, or the resource processor not upserting service.name before export. Check curl otel-collector:8889/metrics | grep service_name directly — if it’s not there, Prometheus won’t see it either.

Histogram buckets don’t line up across services, so cross-service histogram_quantile looks wrong when aggregated. This happens when one team overrides the default histogram bucket boundaries in their SDK setup for “better resolution” on their own service. histogram_quantile over sum by (le) requires identical bucket boundaries across all series being summed, or the quantile math silently produces garbage. Fix: pin bucket boundaries in a shared OTel SDK config wrapper that every service imports, don’t let individual teams set ExplicitBucketBoundaries themselves.

Error rate shows 0% during an actual outage. Usually means the failing requests are timing out before they complete a span — i.e., the client gives up and the server-side metric never records a 5xx, because the request is technically still in flight when OTel would record duration. You need a separate “in-flight requests” gauge (http.server.active_requests in the semconv spec) to catch this, since duration/error metrics alone are blind to hangs.

Two services report the same service.name accidentally. We had payments and payments-worker both set service.name=payments by copy-pasting a Terraform module. The dashboard silently merged their metrics into one line, and rate/error numbers looked fine because errors from one canceled out against volume from the other. Add a uniqueness check on service.name values as part of your service registry, not just trust in deploy configs.

What “done” looks like

You’re done when a new service can be deployed with zero changes to the RED dashboard — it just shows up in the $service_name dropdown because it emits standard OTel metrics, and an on-call engineer can go from “which services are unhealthy” to “here’s the specific endpoint and error code” in one dashboard, without switching tabs. Test it concretely: pick a random service, look up its instrumentation, and see if you can predict its metric names without opening the code. If you can, the convention held. If you have to grep the source to find out what a metric is called, it didn’t.

We’re not fully there — three legacy services still emit hand-rolled Prometheus metrics because rewriting their instrumentation wasn’t worth the sprint, so the dashboard has a small “legacy” filter branch for them. That’s a known gap, not a hidden one, and it’s on the backlog.

FAQ

Do I have to use the OTel Collector, or can services expose Prometheus metrics directly? You can skip the collector and use prometheus-client libraries directly, but then you lose the automatic semantic convention naming that OTel’s auto-instrumentation gives you, and you’re back to manually agreeing on label names across teams — which is the problem this whole approach exists to avoid.

What’s the difference between this and just using Prometheus’s own instrumentation libraries? Prometheus client libraries don’t ship a semantic convention spec — naming is entirely up to you. OTel’s semconv is a versioned, documented standard, so http.server.request.duration means the same thing whether it’s emitted by a Python, Go, or Java service.

Does this work with gRPC services too, not just HTTP? Yes — swap http.server.request.duration for rpc.server.duration, and http.route/http.response.status_code for rpc.method/rpc.grpc.status_code. The dashboard pattern is identical, just a different metric family.

How does this relate to alert fatigue? Directly — a consistent RED dashboard is what makes alert thresholds comparable across services in the first place. If you’re still fighting noisy alerts on top of this, it’s worth reading through the alert design tradeoffs in our alert fatigue writeup since threshold tuning is a separate problem from metric naming.

Should saturation metrics (CPU/memory) really be part of a “RED” dashboard? Strictly, no — RED is Rate/Errors/Duration by definition, and saturation belongs to the USE method. We added it anyway because in practice, splitting them into two dashboards just recreates the “open five tabs” problem this whole exercise was meant to solve. The broader monitoring architecture writeup covers where RED and USE metrics actually diverge if you want the full picture.

What if two teams disagree on which semconv version to adopt? Pin a version org-wide in a shared library or SDK wrapper and treat upgrades as a coordinated migration, not a per-service choice. OTel semconv attributes have changed naming across versions (e.g., http.method became http.request.method), and mixed versions in production will quietly break your sum by aggregations the same way mismatched histogram buckets do.