How to send duplicate RemoteWrite metrics

When working with metrics, you may want to send the metrics you’ve collected to 2 different prometheus instances for backup. Despite that a feature in Prometheus allows you send the metrics received to another prometheus, it’s not recommended to do that when working with high amounts of metrics.

Indeed, Prometheus consumes a lot of resources to do the remotewrite and this may reduce its capability to perform well. But other solution exists : lightweight, simple and works perfectly.

Vmagent is a lightweight agent that has 1 role : receive and retransmit metrics. Easy to use, I’ve been working with it since 3 years and I’m pretty confident to say that it works like a charm.

Prequisites

Let’s suppose that we have a main datasource called datasource1. Then 2 destinations called prometheus1 and prometheus2.

                        ------> prometheus1
datasource1-->vmagent--|
                        ------> prometheus2

Note: in this tutorial, we will use victoriametrics as prometheus server.

Configure Vmagent

let’s configure a vmagent that receive the metrics and duplicate the metrics to 2 datasources.

version: "3.5"

services:
  datasource1:
    container_name: datasource1
    image: quay.io/freshtracks.io/avalanche
    command:
      - --metric-count=10
      - --series-count=10
      - --series-interval=30
      - --const-label=whoami=omarghader.github.io
      - --remote-url=http://vmagent:8429/api/v1/write
    restart: always

  vmagent:
    container_name: vmagent
    image: victoriametrics/vmagent
    ports:
      - "8429:8429" # Adjust the port if needed
    command:
      - -remoteWrite.url=http://victoriametrics1:8428/api/v1/write
      - -remoteWrite.url=http://victoriametrics2:8428/api/v1/write

  victoriametrics1:
    container_name: victoriametrics1
    image: victoriametrics/victoria-metrics
    ports:
      - 8428:8428
    command:
      - "--httpListenAddr=:8428"

  victoriametrics2:
    container_name: victoriametrics2
    image: victoriametrics/victoria-metrics
    ports:
      - 8438:8428
    command:
      - "--httpListenAddr=:8428"

  grafana:
    container_name: grafana
    image: grafana/grafana
    ports:
      - 3000:3000

Run the stack

Run the following command

$ docker compose up -d

How to test ?

1- Grafana

You can use grafana added in the docker compose already.

Add the 2 datasources to grafana :

• http://victoriametrics1:8428
• http://victoriametrics2:8429

Now go to explore, search for {whoami="omarghader.github.io"}. You can see some results.

2- API (curl)

You can simply check that it’s working by running a curl to the api:

$ curl 'http://localhost:8428/prometheus/api/v1/query?query=\{whoami="omarghader.github.io"\}'

Result:

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"avalanche_metric_mmmmm_0_0","series_id":"0","whoami":"omarghader.github.io"},"value":[1695932862,"13"]},......}

Conclusion

Monitoring is a very important piece for every application. Sometimes working with metrics can be really hard. But hopefully, VictoriaMetrics are one of the few in the market to offer very lightweight efficient solutions.

If you love this tutorial please leave a comment, and if you need my help to setup monitoring for your company, you can hire me.