Case study · inference performance · DeepSeek V4 Flash on colibri · 21 October 2026
One token through 284 billion parameters
A 284-billion-parameter model runs on my laptop at about half a word per second, while its drive and its cores sit mostly idle. Follow one token through the model's 43 layers and switch between the engine builds I measured to see where the time goes.
Part of Work · What's next
colibri is an open engine that runs mixture-of-experts models too big for memory by reading their experts from the SSD. On my 64 GB laptop it ran DeepSeek V4 Flash, a 167 GB model, at about half a word per second, with the drive and the cores mostly idle. I measured where the time goes. Most of it was threads falling asleep: a build where they spin while waiting was about 50% faster, and AVX-512 added another 9%.
Scroll through the story to switch between the engine builds, and watch the twelve thread lanes in the panel.
Terms used here
- Parameters
- The numbers a model learns. 284 billion of them make this model 167 GB on disk.
- Mixture of experts (MoE)
- A model split into many small sub-networks, the experts. For each token a small router picks a few, here 6 of 256 at each of 43 layers, so most of the model sits idle.
- Token
- A piece of text a model reads or writes, often part of a word. Prompts and speeds are counted in tokens.
- Offloading
- Keeping part of a model outside memory, here on the SSD, and loading each piece when it is needed.
- Quantisation
- Storing a model's numbers with fewer bits, here 4, so it needs less memory and less reading.
- Thread
- One line of work on a processor core. The panel shows twelve, one per lane.
- OpenMP barrier
- A point where each thread waits until all have arrived. colibri crosses about 500 of them per token.
- Spin or sleep
- Two ways to wait at a barrier. Spinning keeps checking, so it resumes quickly and uses the CPU. Sleeping frees the CPU and is slow to wake.
- AVX-512 and AVX2
- Instruction sets that let one CPU instruction work on many numbers at once. AVX-512 handles twice as many as AVX2.
Six experts out of 256, 43 times
DeepSeek V4 Flash is a mixture of experts. At each of its 43 layers a small router picks 6 of 256 experts for the current token, so a token touches 258 experts: 3.45 GB of 4-bit weights. 64 GB of RAM cannot hold 167 GB, so colibri keeps the rest of the model in RAM and streams experts from the SSD. Popular experts stay cached. In my runs 37% of the experts were already in RAM (amber), and the rest came from the drive (blue).
The drive is not the limit
My storage benchmark reads expert-sized blocks at random, bypassing the OS cache, at 4.8 GB/s. While writing, the engine reads only 0.72 GB/s. At full speed the drive alone would allow about 1.5 to 2 tokens per second, three to four times what the engine reaches.
Cores busy a third of the time
Each router must finish before the engine knows which experts to load, so a token is a chain of 43 dependent load-then-compute steps. More threads made it faster, so compute is on the critical path, yet the cores were busy only about a third of the time. Watch the twelve thread lanes in the panel: most of each layer is waiting.
AVX-512 never runs
The Windows release is compiled for AVX2 with no runtime dispatch. In the shipped engine I counted 0 instructions on AVX-512 registers and 6,224 on AVX2 ones, so its AVX-512 4-bit kernel never runs. My AVX-512 kernel does a token's 258 experts in 0.08 s on 12 cores. Rebuilt for Zen 5, the engine was 9% faster.
500 wake-ups per token
A second build with a different compiler and OpenMP runtime, same instruction set, was 50 to 55% faster. In GCC's OpenMP on Windows, threads waiting at a barrier go to sleep straight away; there is no spin loop, and the settings that ask for one have no effect there. colibri crosses about 500 barriers per token. LLVM's runtime spins instead: CPU use rises from 9% to 46%, and the waits disappear.
Told to sleep, the fast build slows down
One variable at a time: the fast build with OMP_WAIT_POLICY=passive fell back to the shipped engine's speed and CPU use. Asking the shipped engine to spin changed nothing. So the whole gap comes from wake-ups. All runs wrote the same 24 tokens (same hash).
Let the drive set the pace
This one is a projection. Guess the next layer's experts from the current hidden state, load them while this layer computes, and cut the barriers to one per layer. Then the drive would set the pace, at three to four times today's speed. These are my next experiments.
Drag along the thread lanes to scrub a layer by hand · tap the layer strip to jump · tap an expert to see where it lives.
The race
real time
The prompt: "Explain in three sentences why a mixture-of-experts model needs less compute per token than a dense model of the same size." Each run stopped at 24 tokens, all with the same output. Token boundaries are approximate. The last lane is a projection.
Try it
the arithmetic, live
An upper bound: it assumes loads and compute overlap perfectly. The measured line is the shipped engine on mains power (0.49 tok/s).
Results
| Engine build, one variable changed | Decode tok/s | CPU | What it shows |
|---|---|---|---|
| Shipped (GCC, libgomp, AVX2) | 0.197 to 0.227 (4 runs) | 9% | The baseline |
| Shipped, told to spin | 0.221, 0.225 | 9% | The setting has no effect on Windows |
| clang, libomp, AVX2 | 0.321 to 0.326 (4 runs) | 46% | Threads spin: about 50% faster |
| clang, libomp, AVX2, told to sleep | 0.215, 0.212 | 10% | Back to the baseline: the waits are the gap |
| clang, libomp, AVX-512 (Zen 5) | 0.349, 0.357 | 46% | AVX-512 adds about 9% |
These 14 runs were on battery, so compare them with each other; on mains the shipped engine does 0.49 tok/s. Each run first restored the same expert-usage snapshot, found 37% of its experts in RAM and produced the same text. The next step is to rerun them on mains power.
Method
How it was measured
- A model of the workload from the model's config: bytes per token, expert sizes, what the cache saves.
- A storage benchmark in C++20 that reads expert-sized blocks at random, bypassing the OS cache. Clean under ThreadSanitizer and AddressSanitizer.
- An AVX-512 kernel for 4-bit weights that decodes 16 weights per permute instruction, checked against a scalar version on each run.
- Profiled runs of the engine, and two engine builds from the same source, so one variable changes at a time.
In the panel, the order of the steps inside a layer comes from colibri's source. Their lengths are a model, scaled so each token takes the measured time. Which experts light up is random, since the routing itself was not recorded.
C++20 · AVX-512 · OpenMP (libgomp, libomp) · clang, llvm-mingw · GCC · PowerShell · Python