Running llama.cpp on Snapdragon Hexagon NPU seems promising

via

LocalLlama

I have an Oneplus 12 with Snapdragon 8 Gen 3. I followed the above README to cross-compile llama.cpp on Ubuntu and then copy to the Termux directory on the phone. It seems like llama.cpp's Hexagon backend is highly supported by Qualcomm with many PRs made by Qualcomm employees. I am getting 8t/s pp and 4.5t/s tg with gemma-3-12b-it-qat-Q4_0 and 20t/s pp 12.5t/s tg with gemma-3-4b-it-qat-Q4_0. Speed is about the same as using the SD8G3's CPU but it is not hot at all and the tg speed is good enoug

More from LocalLlama

Bruh

LocalLlama·3

Qwen3.6-27B-NVFP4 - images

LocalLlama·5

Hybrid on-device inference on Android: llama.cpp + LiteRT + NPU/GPU routing

LocalLlama·6

MiniMax M2.7 AWQ-4bit on 2x Spark vs 2x RTX 6000 96GB - performance and energy efficiency

LocalLlama·4

Qwen3.6-27B at 72 tok/s on RTX 3090 on Windows using native vLLM (no WSL, no Docker), portable launcher and installer

LocalLlama·6

A Dark-Money Campaign Is Paying Influencers to Frame Chinese AI as a Threat

LocalLlama·6