The Moment the Jaeger Tracer Exhausted Itself and What We Switched To
The article discusses the challenges faced by Veltrix during the 2025 Black Mesa event when their treasure-hunt engine struggled under heavy load. The team initially attempted to optimize performance by switching data formats but encountered issues with memory alignment. Ultimately, they transitioned to a Rust-based rule engine, significantly improving latency and eliminating previous errors.
- ▪Veltrix served 380,000 concurrent players during the 2025 Black Mesa event, revealing hidden performance issues.
- ▪Initial attempts to optimize performance using Protocol Buffers and Flatbuffers failed due to garbage collection and alignment errors.
- ▪Switching to a Rust-based rule engine reduced latency significantly, with p95 dropping to 41 ms and eliminating memory corruption issues.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3942594) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } pretty ncube Posted on May 26 The Moment the Jaeger Tracer Exhausted Itself and What We Switched To #webdev #programming #rust #performance The Problem We Were Actually Solving Our treasure-hunt engine at Veltrix was not exploding; it was quietly drowning. During the 2025 Black Mesa event, we served 380 000 concurrent players searching 1.4 million geo-cached items across ten shards.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
