Case study · llama.cpp · Vulkan · 19 to 20 October 2026
A 3-second pause, taken apart
My voice assistant paused about 3 seconds before each answer. The model was fast, so the time was going somewhere else. These are the eight measurements that found it.
Part of Work · What I'm doing
My voice assistant runs on my laptop, and it waited about 3 seconds before each answer. The model itself was fast, so I measured one step at a time until I found the delay: saving the model's working state was slow on this laptop's integrated GPU. One extra condition in llama.cpp brought the first spoken word down to 0.7 s.
Scroll through the eight steps below and watch the time budget change with each one.
Terms used here
- llama.cpp
- An open-source engine that runs language models on ordinary computers.
- Token
- A piece of text a model reads or writes, often part of a word. Prompts and speeds are counted in tokens.
- State snapshot
- A saved copy of the model's running state, so a conversation can resume without re-reading everything.
- Hybrid model
- A model in which part of the network keeps a running state instead of the usual attention cache. It cannot step that state back without a snapshot.
- Integrated GPU
- A graphics processor on the same chip as the CPU, sharing its memory.
- Cached memory
- Memory the CPU keeps fast copies of. Reading uncached memory is slow: here 114 MB per second.
- Staging buffer
- A temporary copy in fast memory. The GPU copies the data there first, then the CPU reads it.
- Vulkan
- A programming interface for GPUs. The engine uses it to run the model on the laptop's Radeon graphics.
- Temperature 0
- A setting that makes the model always pick its most likely next word, so two runs can be compared word for word.
The model is not the problem
llama-bench read 16 prompt tokens in 0.11 s. The server took 1.4 to 2.5 s for the same work. The model is a hybrid, Qwen3.6-35B-A3B: part of it keeps a running state instead of a normal attention cache.
Ruled out on a test server
Batch size, context size and slot count, each changed on a separate server on a spare port so production stayed untouched. None moved the delay.
Two snapshots, 0.6 s each
To resume a conversation after its text changes, the server saves snapshots of the model's running state: 63 MB each, two per request. Those two saves were the whole delay.
Faster voice, broken agent
With snapshots off, voice answered in 0.5 s. But the agent re-formats its previous reply slightly, and a hybrid model cannot step its state back without a snapshot, so it re-read its 13,700-token prompt on each turn. My first tests used only voice-style conversations. A full test of all the services caught it, and I reverted the change.
No change
A newer engine build and four Vulkan settings gave the same timings. The cost was in the code.
Skipping GPU waits saved nothing
Each saved tensor goes through a synchronous GPU barrier and wait. Skipping the redundant ones took a snapshot from 0.65 s to 0.61 s. Discarded.
114 MB per second
The waits took 14 ms in total. The CPU copy took 7,738 ms for 880 MB. On this integrated GPU the memory is visible to the CPU but not cached by it, so each read crawls.
Read directly only from cached memory
Otherwise let the GPU copy into a cached staging buffer first, then read that.
// ggml_vk_buffer_read_2d(): direct host read only when it is fast if (src->memory_property_flags & vk::MemoryPropertyFlagBits::eHostVisible && src->device->uma && src->memory_property_flags & vk::MemoryPropertyFlagBits::eHostCached) {
The fix
at measured speed
Verification
Same answers, word for word
- The same scripted six-turn conversation at temperature 0 ran on the stock and fixed builds, in two modes: history resent verbatim, and history altered so the server must restore a snapshot.
- Replies were identical word for word in both modes; snapshot restores 4 of 4 in each build.
- The exact deployment, one library swapped, was rehearsed on a copy of the engine folder first, with a backup and a one-step undo.
Outcome. After deploying, I found that the Vulkan backend's maintainer had merged the same fix two weeks earlier (llama.cpp #30049), after a benchmark on the same GPU. My engine build predated it, so I reached the same fix independently. I now check a file's recent history before patching it.
Results
| Measured | Before | After | Main cause |
|---|---|---|---|
| Voice, time to first word | ≈ 3 s | ≈ 0.7 s | Engine fix, voice-only sampling setting |
| Agent, each later turn | 3.5 s | 1.1 to 1.4 s | Engine fix, smaller prompt |
| Agent, new conversation | 2.5 s | 1.2 s | Engine fix |
| Agent, task with code tool | 43.5 s | 23.0 s | Engine fix |
| Agent, task with web search | 11.7 s | 7.1 s | Engine fix |
| One state snapshot | 0.61 s | 0.03 s | Cached read path |
| Agent prompt per request | 13,718 tok | 11,560 tok | Removed two unused tools |
The first turn after the model loads is unchanged: the agent still reads its 11,600-token instructions once, about 21 to 24 s.
Method
How I worked
The goals were faster voice and agent replies, with nothing lost and no cloud inference. Each experiment ran on a test server kept apart from the running system, and the deployment was rehearsed on a copy first, with a backup and a one-step undo. When a change broke something, I reverted it.
llama.cpp (C++, Vulkan) · Visual Studio 2022 Build Tools · Vulkan SDK · llama-bench · Python harnesses · Docker · WSL