Frontier · inference engine · OLMoE-1B-7B · 24 October 2026
An expert-offloading engine, from scratch
A mixture-of-experts model with its experts on the SSD and a fixed-size cache in memory, in a program small enough to check byte by byte. Then I measured what the speed depends on.
Part of Work · What's next
After measuring colibri from the outside, I wrote a small engine of the same kind, moe-stream, so I could control each part. It keeps a mixture-of-experts model's experts on the SSD and a fixed amount of them in memory. Before trusting any speed, I checked that it writes exactly the same text as the reference implementation.
Guessing the next layer's experts early makes it 58% faster, and with 8 of the 13 GB in memory it reaches 57% of full speed. The chart below has a switch to turn that guessing (prefetch) on and off.
Terms used here
- Mixture of experts (MoE)
- A model split into many small sub-networks, the experts. A router picks a few for each token: here 8 of 64 at each of 16 layers.
- Token
- A piece of text a model reads or writes, often part of a word. Speed is counted in tokens per second.
- Offloading
- Keeping part of a model outside memory, here on the SSD, and loading each piece when it is needed.
- Cache
- The experts kept in memory. A hit means the expert was already there; a miss means reading it from the drive.
- LRU and LFU
- Two rules for which expert to drop when the cache is full: the least recently used, or the least frequently used.
- Prefetch
- Starting to read an expert before it is asked for, from a guess, so the read overlaps with computing.
- Reference implementation
- Hugging Face transformers' version of the same model, used as the ground truth for correctness.
- bfloat16 and float32
- Two ways to store numbers: 16 bits (fast, less precise) and 32 bits (precise enough to compare two programs bit for bit).
Speed against memory
OLMoE-1B-7B · 13 GB of experts
Tokens per second against the memory given to the expert cache. The dashed line is all experts in memory.
Findings
What the numbers say
- The ceiling is compute. With all experts in memory the CPU writes 8.6 tokens per second. Anything below that is time spent waiting for the drive.
- Prefetching helps at all cache sizes. Running the next layer's router on the current state guesses its experts early, and 82% of the guesses are then used. With no cache at all, that alone is 58% faster.
- The obvious rule fails on small caches. A token walks through 128 experts in a fixed order. When fewer than that fit, least-recently-used always drops the one needed next, so 1 GB of LRU got no hits at all. Least-frequently-used keeps the popular experts and got 26%.
- A saved "hot set" does not transfer. The most-used experts, measured on a story prompt, did worse than LFU at all sizes on a technical question, because routing depends on the text. colibri relies on such a saved file, so its benefit depends on how close the next prompt is to past use.
- The drive has room to spare. Reads peaked at 3.1 GB/s with prefetch, against 4.8 GB/s measured for this SSD. More reads in flight is the next thing to try.
Correct first
Checked against the reference before any speed counts
A fast engine that writes different text has a bug. In 32-bit numbers, my engine and Hugging Face transformers give the same scores for each next word (largest difference 0.0000) and the same 48 words in a row. In 16-bit numbers the two drift apart at word 19. That is rounding order: the same sums added in a different order. All 27 conditions produced exactly the same text, so caching and prefetching never change the answer.
Method
How it was built and measured
- The model. OLMoE-1B-7B (AllenAI, fully open): 16 layers, 64 experts each, 8 per token; 6.9B parameters, 1.3B used per token.
- The split. Each of the 1,024 experts in its own 12.6 MB file, laid out so one read brings everything it needs.
- Reads past the file cache. Windows keeps recently read files in memory, which would turn "reads from disk" into memory copies on a 64 GB laptop. All reads use
FILE_FLAG_NO_BUFFERING, so the SSD itself is measured. - The runs. 32 generated tokens per condition, the same prompt, and statistics counted over generated tokens only. The static hot set was calibrated on a different prompt. Each row records whether the laptop was on mains power before and after it, and all 27 were. A first pass that switched from battery to mains mid-run was discarded.
Python · PyTorch · Hugging Face · Windows I/O · OLMoE