Kv cache quantization: ignorance, or malice?

via

LocalLlama

I run Qwen-3.6 27B FP8 on vllm for long-horizon agentic coding harness workloads with high context window and concurrent sub-agents. On two 3090s that aren’t used for anything else, it seems reasonable to expect a good balance between speed and reliability. I want to bring up a particular point of contention regarding this optimization process. I have extensive software engineering background but am relatively new to this so feel free to correct me if I’m not on the right track. It seems like co

More from LocalLlama

Bruh

LocalLlama·3

Qwen3.6-27B-NVFP4 - images

LocalLlama·5

Hybrid on-device inference on Android: llama.cpp + LiteRT + NPU/GPU routing

LocalLlama·6

MiniMax M2.7 AWQ-4bit on 2x Spark vs 2x RTX 6000 96GB - performance and energy efficiency

LocalLlama·4

Qwen3.6-27B at 72 tok/s on RTX 3090 on Windows using native vLLM (no WSL, no Docker), portable launcher and installer

LocalLlama·6

A Dark-Money Campaign Is Paying Influencers to Frame Chinese AI as a Threat

LocalLlama·6