Building a Monitoring Stack with Prometheus, VictoriaMetrics, and Grafana

Ever found yourself drowning in server logs, wondering why your application crashed again at 3 AM? I’ve been there—more times than I’d like to admit. That’s why I built a robust monitoring stack with Prometheus, VictoriaMetrics, and Grafana to keep an eye on my systems. In this guide, I’ll walk you through setting up this powerful trio to collect, store, and visualize metrics at scale—without losing your sanity.

Why This Stack?

Prometheus is the de facto standard for metric collection, VictoriaMetrics handles long-term storage like a champ, and Grafana turns raw data into beautiful, actionable dashboards. Together, they’re like the Avengers of observability—just with fewer capes and more YAML.

What You’ll Need

• A Linux server (Ubuntu 22.04 LTS used here)
• Docker and Docker Compose (for containerized setup)
• Basic terminal familiarity (no sudo rm -rf / jokes, please)
• 10 minutes of patience (or coffee)

Step 1: Setting Up Prometheus

Prometheus will scrape and temporarily store metrics. Here’s how to deploy it with Docker:

1. Create a prometheus.yml config file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

2. Run Prometheus with Docker:

docker run -d \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  --name prometheus \
  prom/prometheus

Pro Tip: Test if it’s working by visiting http://your-server-ip:9090. If you see the Prometheus UI, you’re golden.

Step 2: Adding VictoriaMetrics for Long-Term Storage

Prometheus is great, but it’s not designed for long-term retention. Enter VictoriaMetrics—a time-series database that’s faster and more resource-efficient than Prometheus’ built-in storage.

Deploy VictoriaMetrics

1. Run VictoriaMetrics in single-node mode (for simplicity):

docker run -d \
  -p 8428:8428 \
  --name victoriametrics \
  victoriametrics/victoria-metrics

2. Configure Prometheus to remote-write to VictoriaMetrics by updating prometheus.yml:

remote_write:
  - url: http://victoriametrics:8428/api/v1/write

Troubleshooting: If Prometheus can’t connect to VictoriaMetrics, check the logs with docker logs victoriametrics. Firewall issues? Double-check ports!

Step 3: Visualizing Data with Grafana

Now for the fun part—making your metrics look pretty.

1. Launch Grafana with Docker:

docker run -d \
  -p 3000:3000 \
  --name grafana \
  grafana/grafana-enterprise

2. Log in to Grafana (http://your-server-ip:3000) with the default credentials (admin/admin—change this ASAP).

3. Add VictoriaMetrics as a data source:

   • Go to Configuration > Data Sources.
   • Select VictoriaMetrics, set the URL to http://victoriametrics:8428, and save.

4. Import a dashboard (try this CPU monitoring example).

Gotcha: If Grafana shows “No data,” ensure Prometheus is scraping targets correctly (http://prometheus:9090/targets).

Step 4: Scaling and Optimizing

• High Availability: Run VictoriaMetrics in cluster mode for redundancy.
• Alerting: Use Grafana’s alerting or Prometheus Alertmanager for notifications.
• Backups: Schedule regular backups of VictoriaMetrics data (/var/lib/victoria-metrics-data).

FAQ

Q: Can I use this stack for IoT devices?

A: Absolutely! Use Prometheus’ pushgateway for devices that can’t be scraped.

Q: Why VictoriaMetrics over Thanos or M3DB?

A: VictoriaMetrics is simpler to configure and uses fewer resources—ideal for small-to-medium setups.

Q: How do I secure Grafana?

A: Enable HTTPS, use strong passwords, and restrict access with a reverse proxy (like Nginx).

Wrapping Up

With this stack, you’ll never be blindsided by a midnight outage again (well, mostly). For more observability fun, check out my guide on Grafana alerting.

Now go forth and monitor like a pro! 🚀