Is there a way to mitigate performance as context grows?
·
0 reactions
·
0 comments
·
8 views
In my local LLM setup I get from 30 to 80 t/s generation at the beginning, but it drops quite a lot as context grows. I use llama.cpp/Vulkan with an MI50 and a V100, is there some command line flags that can improve this issue? Or some good practice other than restart the chat after some time?
Original article
LocalLlama
Anonymous · no account needed
Discussion
0 commentsMore from LocalLlama
daily ritual at this point…
LocalLlama·14
convert : add support for Nemotron Nano 3 Omni by danbev · Pull Request #22481 · ggml-org/llama.cpp
LocalLlama·5
Mistral Medium Is On The Way
LocalLlama·14
Something from Mistral (Vibe) tomorrow
LocalLlama·6
Deepseek Vision Coming
LocalLlama·7
First direct side by side MoE vs Dense comparison.
LocalLlama·8