Every phone vendor sells you a TOPS number for the neural engine. We ran real models on a real device to find out what that buys, and the honest answer is: half of what you assume, in the half you were not thinking about.
All numbers here are from a Galaxy S25 Ultra (SM-S938B, Snapdragon 8 Elite), running llama.cpp with the ggml-hexagon backend on the Hexagon NPU. Same phone, same models, same day. No estimates.
Two halves of an answer
When a language model replies to you it does two very different jobs.
Prefill is reading. The model ingests your prompt and the conversation behind it, all at once, as a big matrix multiplication. This is what accelerators are built for.
Decode is writing. The model emits one token, then uses it to emit the next, and so on. Every single token requires walking the entire weight set through memory.
They have almost nothing in common at the hardware level, which is why one number cannot describe both.
What we measured
Qwen 2B, Q4_0, on the same handset:
| Backend | Prefill | Decode |
|---|---|---|
| CPU, 6 threads | 48 tok/s | 22 tok/s |
| GPU (Adreno, OpenCL) | 357 tok/s | 21.7 tok/s |
| NPU (Hexagon, ggml-hexagon) | 700 tok/s | 18.8 tok/s |
Prefill on the NPU is 14.6× the CPU. That is a genuinely enormous win and it is the reason long prompts and document context are usable on a phone at all.
Decode is a tie. Worse than a tie: the NPU is slightly slower than the plain CPU.
Why decode refuses to get faster
Decode is memory-bound, not compute-bound. For each token the machine has to read the weights out of memory, and a Q4 2B model is roughly 1.2 GB of them. At ~20 tok/s that is about 24 GB/s of sustained reads — which is a bandwidth-shaped number, not a compute-shaped one.
Our 4B measurement says the same thing from the other end: 13.87 tok/s × 2.58 GB ≈ 36 GB/s. The chip is not waiting for maths. It is waiting for memory.
No accelerator fixes this, because the accelerator is downstream of the bottleneck.
So what does make it faster?
Weight-set size. We have two data points and they agree to within 5%:
- 0.8B model, on the CPU: 64–67 tok/s decode.
- 4B model, on the NPU: 13.86 tok/s decode.
The weight ratio is 5.09×. The decode ratio is 4.85×. That is what bandwidth-bound scaling looks like — and note which way round it lands: a small model on the CPU beats a large model on the NPU by nearly five times.
The backend is not the lever. The size of the brain you loaded is the lever.
What we did about it
We stopped choosing the model for you. Every app here lets you pick, shows you the download size and tells you honestly whether that model will fit your phone comfortably. That is not a settings-screen nicety, it is the single largest speed control available on the device, and hiding it behind a “we picked the best one for you” would have cost users a 5× difference they were entitled to make for themselves.
We also stopped claiming NPU acceleration we cannot verify. When an app loads a model it checks whether the NPU actually took it, by measuring prefill against a floor for that model. If it comes back under, the app says CPU. We would rather show you a slower true label than a faster false one.
The uncomfortable footnote
These are single-device runs, not a benchmark suite. They move a few percent with heat: our 4B prefill wandered between 263.5 and 239.5 tok/s across sessions on the same phone. Anyone quoting on-device inference numbers to three decimal places without mentioning thermals is selling something.