Are you quanting your memory?
·
0 reactions
·
0 comments
·
2 views
Title. Curious about how people are generally dealing with the kv cache. BF16? Q8? Q4? Turboquant or some other secret sauce? I run bf16 everything hoping that I'd get less hallucinations and because that's what the g4 and q3.6 are natively trained on anyways. But very interested to hear if people are having good results running q8 or q4 or if anyone has good results using turbo3/4 or similar.
Original article
LocalLlama
Anonymous · no account needed
Discussion
0 commentsMore from LocalLlama
Qwen3.6-27B-NVFP4 - images
LocalLlama·3
Hybrid on-device inference on Android: llama.cpp + LiteRT + NPU/GPU routing
LocalLlama·4
MiniMax M2.7 AWQ-4bit on 2x Spark vs 2x RTX 6000 96GB - performance and energy efficiency
LocalLlama·3
Qwen3.6-27B at 72 tok/s on RTX 3090 on Windows using native vLLM (no WSL, no Docker), portable launcher and installer
LocalLlama·5
A Dark-Money Campaign Is Paying Influencers to Frame Chinese AI as a Threat
LocalLlama·5
"LLM is created so engineer don't have to write a report", anyway found out ONLYOFFICE can connect to OpenAI compatible, using Qwen 3.6 to do elaboration.
LocalLlama·5