How to Set Up Prometheus Remote Write to Push Metrics to VictoriaMetrics

I’ve been running my own monitoring stack for years, and if there’s one thing I’ve learned, it’s that Prometheus alone can feel like trying to store all your holiday decorations in a single shoebox. It works fine at first, but soon you’re dealing with capacity issues and wondering where that one crucial metric from three months ago disappeared to.

That’s why I recently set up Prometheus remote write to push metrics to VictoriaMetrics for centralized, long-retention storage. VictoriaMetrics acts like a massive, organized storage unit for all your metrics - it’s scalable, efficient, and surprisingly simple to set up. In this guide, I’ll walk you through the exact steps I used to get this working.

Why I Chose VictoriaMetrics for Long-Term Metric Storage

Before we dive into the technical details, let me share why I moved from Prometheus’s default storage to VictoriaMetrics:

• Cost-effective long-term retention: VictoriaMetrics uses about 7x less storage space than Prometheus for the same data
• High availability: Built-in replication means I don’t lose metrics if a node goes down
• SQL-like query language: VMQL is compatible with PromQL but adds useful extensions
• Single binary deployment: The all-in-one version is ridiculously easy to get running

The best part? Prometheus’s remote write feature makes this integration nearly seamless. You’re not replacing Prometheus - you’re enhancing it with enterprise-grade storage.

What You’ll Need

Before we start, make sure you have:

• A running Prometheus instance (I’m using v2.45.0)
• VictoriaMetrics installed (I used the single binary v1.95.0)
• Network connectivity between your Prometheus and VictoriaMetrics instances
• Basic familiarity with YAML configuration files
• About 30 minutes of your time

Step 1: Verify Your VictoriaMetrics Installation

First, let’s make sure VictoriaMetrics is running and accessible. I’m assuming you have it running on the same network, but this works equally well across cloud providers.

Start VictoriaMetrics if it’s not already running:

# Basic startup command for VictoriaMetrics single binary
./victoria-metrics-prod -retentionPeriod=12

The -retentionPeriod=12 flag sets retention to 12 months - adjust this based on your storage capacity and requirements.

Verify it’s working by curling the health endpoint:

curl http://your-victoriametrics-ip:8428/health

You should see a simple OK response. If you get connection refused, check your firewall rules and ensure VictoriaMetrics is actually running.

Step 2: Configure Prometheus Remote Write

This is where the magic happens. We’ll modify Prometheus’s configuration to push all metrics to VictoriaMetrics via remote write.

Open your prometheus.yml configuration file and add the remote_write section:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Your existing scrape configs remain unchanged
scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
  # Add your other jobs here

# Remote write configuration for VictoriaMetrics
remote_write:
  - url: http://your-victoriametrics-ip:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      capacity: 10000
      max_shards: 30
    retry_on_http_429: true

Let me break down the key parameters:

• url: The VictoriaMetrics endpoint that accepts Prometheus metrics
• max_samples_per_send: How many samples to send in one batch (I found 10,000 works well)
• capacity: Queue size for samples waiting to be sent
• max_shards: Number of parallel connections to VictoriaMetrics
• retry_on_http_429: Automatically retry if VictoriaMetrics is temporarily overloaded

Step 3: Reload Prometheus Configuration

After saving the configuration file, you need to reload Prometheus. I prefer using the API reload endpoint rather than restarting the entire service:

# Reload configuration
curl -X POST http://localhost:9090/-/reload

Check Prometheus’s status page at http://your-prometheus:9090/status to verify the configuration loaded without errors. Look for the “Remote Write” section - it should show your VictoriaMetrics endpoint as active.

Step 4: Verify Metrics Are Flowing to VictoriaMetrics

Now for the moment of truth - let’s confirm metrics are actually being written to VictoriaMetrics.

First, let’s check what metrics VictoriaMetrics has received:

# List available metrics in VictoriaMetrics
curl -g 'http://your-victoriametrics-ip:8428/api/v1/label/__name__/values'

You should see a list of metric names that match what Prometheus is scraping. If the list is empty, wait a minute and try again - it might take a moment for the first batch to be sent.

You can also query a specific metric directly from VictoriaMetrics:

# Query a metric directly from VictoriaMetrics
curl -g 'http://your-victoriametrics-ip:8428/api/v1/query?query=up'

Compare the results with the same query in Prometheus to ensure they match.

Step 5: Configure Grafana to Use VictoriaMetrics as a Data Source

If you’re like me, you’re probably using Grafana for visualization. Let’s add VictoriaMetrics as a data source.

In Grafana, go to Configuration → Data Sources → Add data source and select Prometheus. Then configure it as follows:

• Name: VictoriaMetrics (or whatever you prefer)
• URL: http://your-victoriametrics-ip:8428
• Type: Prometheus

Keep the other settings as default and click “Save & Test.” You should see a green success message.

Now you can create dashboards that query VictoriaMetrics directly, taking the load off your Prometheus instance for historical queries.

Troubleshooting Common Issues

I ran into a few hurdles during my setup - here’s how I solved them:

Problem: Prometheus logs show “connection refused” errors for remote write. Solution: Check your VictoriaMetrics URL and port. Ensure firewalls allow traffic between the instances.

Problem: Metrics appear in VictoriaMetrics but with delays. Solution: Adjust the queue_config parameters. Increasing max_shards can help with throughput.

Problem: High memory usage in Prometheus after enabling remote write. Solution: Reduce max_shards and capacity values. Monitor Prometheus memory usage and adjust accordingly.

Problem: VictoriaMetrics returns 429 “too many requests” errors. Solution: Enable retry_on_http_429: true and consider increasing VictoriaMetrics resources.

Advanced Configuration: Authentication and TLS

If you’re running this in production (or just want to be extra secure), you’ll want to add authentication and TLS.

For VictoriaMetrics, you can enable HTTP basic authentication:

remote_write:
  - url: http://your-victoriametrics-ip:8428/api/v1/write
    basic_auth:
      username: your_username
      password: your_password

For TLS, simply use HTTPS in the URL and ensure your certificates are properly configured.

Next Steps and Enhancements

Now that you have basic remote write working, here are some ways to level up your setup:

• Set up VictoriaMetrics clustering for high availability
• Configure recording rules in VictoriaMetrics to precompute expensive queries
• Implement metric relabeling to filter which metrics get sent to VictoriaMetrics
• Set up alerting directly from VictoriaMetrics using vmalert

The beauty of this setup is that it scales with your needs. I started with a single VictoriaMetrics instance and later expanded to a clustered setup as my metric volume grew.

Frequently Asked Questions

Q: Will remote write impact Prometheus performance? A: There’s minimal impact if you tune the queue parameters correctly. Prometheus is designed to handle remote write efficiently.

Q: Can I use remote write with Prometheus Operator? A: Absolutely! You’ll add the remote write configuration to your Prometheus CRD - the principles are the same.

Q: What’s the difference between remote write and remote read? A: Remote write sends metrics from Prometheus to VictoriaMetrics. Remote read allows Prometheus to query VictoriaMetrics for historical data - you can configure both!

Q: How do I migrate existing historical data? A: Use vmctl, VictoriaMetrics' data migration tool, to copy existing Prometheus data to VictoriaMetrics.

Setting up Prometheus remote write to push metrics to VictoriaMetrics has been one of the most impactful improvements to my monitoring stack. It’s like upgrading from a closet to a properly organized warehouse for all your metrics. The setup is straightforward, and the long-term benefits for scalability and retention are absolutely worth the effort.

If you’re looking for other monitoring improvements, you might enjoy my posts on setting up Grafana alerting with custom messages or building a comprehensive home monitoring system.

Have questions or run into issues? Drop a comment below - I’m happy to help troubleshoot based on my own experiences!