Scaling a real-time metrics pipeline 50x

June 1, 2023 · Go, Kafka, Kubernetes, Terraform, Helm

The challenge

Ubisoft’s monitoring pipeline was throttled at 66,000 datapoints per second โ€” nowhere near enough for the scale of live game telemetry across the studio’s titles.

What I built

Redesigned the ingestion and aggregation architecture deployed on Kubernetes with Terraform/Helm for reproducible infrastructure. Detected the bottleneck ongoing and fixing it to increase the scalability of the pipeline.

Result

Throughput scaled to 3.3M datapoints/sec โ€” a 50x improvement โ€” without a proportional increase in infrastructure cost, and became the backbone for the studio’s monitoring roadmap going forward.