llama.cpp b9455 Finally Caught vLLM: 70t/s on 2x3090 Qwen 27B UQ8
A Reddit user has reported impressive performance metrics for the llama.cpp build b9455, achieving 70 tokens per second on dual RTX 3090 GPUs. This marks a significant improvement over the previous vLLM, which dominated multi-GPU inference but had slower speeds. The new build's tensor parallelism allows for more efficient processing, making it a viable option for users previously reliant on vLLM.
- ▪The llama.cpp build b9455 achieved 70 tokens per second on a dual RTX 3090 setup.
- ▪Prior to this, vLLM was the leading choice for multi-GPU inference with speeds of 70+ tokens per second.
- ▪The new tensor parallelism feature in llama.cpp allows for simultaneous processing across GPUs, improving overall performance.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3963941) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } Storm Engine Technology. Posted on Jun 3 llama.cpp b9455 Finally Caught vLLM: 70t/s on 2x3090 Qwen 27B UQ8 #ai #llm #opensource #llamacpp A Reddit user on r/LocalLLaMA just dropped some impressive numbers for llama.cpp build b9455, and they're worth paying attention to if you're running multi-GPU setups.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
