Category: Events
The article discusses performance issues encountered in a complex application involving Apache Kafka and Presto. The initial design led to hot partitions and latency problems, prompting a reevaluation of the architecture. After implementing changes to partitioning and query optimization, significant improvements in query costs and latency were achieved.
- ▪The application faced performance issues due to high query costs and hitting Redshift's maximum concurrent query limit.
- ▪Initial Kafka topic partitioning by user ID caused hot partitions, leading to uneven cluster utilization and latency issues.
- ▪After changing the partitioning strategy to quest type and optimizing queries, there was a 30% reduction in query costs and a 25% decrease in query latency.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3942568) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } ruth mhlanga Posted on May 22 Category: Events #webdev #programming #dataengineering #python The Problem We Were Actually Solving Behind the scenes, this application relies on a complex interplay of Apache Kafka topics, Redshift views, and Presto queries to fetch user information, generate quest narratives, and push notifications.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
