400 Million Tokens Burned Overnight
A significant error led to the consumption of 400 million tokens in a single day due to a malfunctioning onboarding process. An orchestrator agent repeatedly sent onboarding messages, causing an infinite loop of new agent sessions. The issue went unnoticed for 15 hours, as the system appeared to function normally despite the excessive token usage.
- ▪262 million input tokens were consumed in a single day, nearly triple the usual amount.
- ▪The orchestrator agent sent a welcome message every 60-90 seconds, leading to repeated session spawning.
- ▪The problem stemmed from a combination of network discovery issues and a lack of deduplication in the message forwarding process.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3911043) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } nerudek_vibecoder Posted on May 25 400 Million Tokens Burned Overnight #ai #deepseek #debugging #postmortem 400 Million Tokens Burned Overnight 5,080 API requests. Everything looked normal. My Heart Stopped At 8:03 AM Sunday, May 24, 2026. I opened the API dashboard and my stomach dropped. 262 million input tokens consumed in a single day.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
