Frontier · the map · interactive
Four ways to run a model
Three ways a model can write an answer, raced on the same Sudoku board, and one way to run a model bigger than the computer's memory. The animations move at the measured pace.
Part of Work · What's next
New kinds of models try to cut one of two costs. Some change how the answer is written, so it takes fewer passes through the model. Others change where the model lives, so a model too big for memory still runs. I tested both kinds on my laptop, and this page puts them side by side.
Press play to race three ways of writing the same Sudoku board. Further down, drag the memory slider to run a model off the drive.
Terms used here
- Word by word
- How today's chat models write. One pass through the model per piece of text, each waiting for the last. Reliable, and slow when the answer is long.
- Diffusion
- Fills in the whole answer over a set number of rounds. Fewer rounds are faster, until the words stop fitting together.
- Next-concept modelling (NCM)
- Sketches a compact plan of the answer and refines it level by level, each level written at once.
- Mixture of experts
- A model made of many small parts, the experts, that uses only a few of them per word. The rest can stay on the drive, which is the idea behind colibri and moe-stream.
1 · How the answer is written
one tick = one pass through the model
The same board, three ways
A hard board from Stanford's test set. Dim digits are given; the rest must be written. Each tick is one pass through a model, which is what costs time.
One digit per pass, in reading order, each waiting for the one before. 81 digits and 8 line breaks: 89 passes.
Several cells are committed per round. Cells committed in the same round cannot see each other: amber marks a cell fixed beside a neighbour in its row, column or box that it could not take into account.
Each pass writes a whole level of codes at once: 1 code, then 2, 4, 6 … 89. The board is decoded from all 12.
Press play. Each tick is one pass through the model. A pass takes a different time in each model; the measured times are in the table further down.
What the coarse levels look like
I expected the coarse levels to look like a blurred board that slowly sharpens. They don't. I decoded this board from the model's codes after each level, and the cells right out of 81 went 0, 0, 1, 2, 1, 2, 0, 4, 6, 5, 5, then 81 after the last level. A random guess gets about 9. Eight test boards showed the same pattern. The early levels are a plan in the model's own code, and the board only appears once the plan is complete. That is the "concept" in next-concept modelling.
2 · Where the model lives
OLMoE-1B-7B · 1,024 experts · 13 GB
A big model in small memory
A mixture-of-experts model has many small parts, the experts, and each word uses only 8 of the 64 in each layer. So the experts can live on the drive, with the most used ones kept in memory. This is the idea behind colibri. Drag the memory slider and watch a word pass through the 16 layers.
Output at the measured speed, in real time
Which experts light up is simulated, with a typical mix of popular and rare ones. The speed and the streaming text are measured for that memory size, with moe-stream on this laptop.
At full scale, the same idea is colibri running a 284-billion-parameter model on this 64 GB laptop. It answered all 10 grade-school maths problems I gave it, as did Claude and a 35B model that fits in memory. The 35B model was 85 times faster: 4 seconds an answer against 6 minutes.
To see where that time goes I wrote a small engine, moe-stream, and measured it. With no memory for experts it writes about one word per second. Guessing the next layer's experts early (prefetch) makes it 58% faster, and holding 8 of the 13 GB in memory brings it to 57% of full speed. The text came out the same in all 27 conditions.
Side by side
measured on this laptop
What each one buys, and what it costs
| Approach | What it changes | What I measured | The catch |
|---|---|---|---|
| Word by word (chat models) | Nothing: the baseline. One pass per token. | Qwen2.5 7B: a correct, fluent answer in 3.8 s. Claude Opus 5.5: 46 of 50 hard boards, 53 s each. | Long answers and long reasoning cost one pass per token. |
| Diffusion (Dream 7B, LLaDA 8B) | Fills the whole answer over a set number of rounds. | 32 rounds: word salad in 11 s. Fluent took about one round per word: 64 to 124 s. | Few rounds means words chosen together that cannot see each other. |
| Next-concept (NCM, 25M) | A plan in 12 levels, each level written at once. | 85.2% of 500 hard boards, 0.47 s each on the laptop CPU. My 3.3M training run: 77% of cells, no full board yet. | A specialist: trained for one kind of answer. |
| Experts off the drive (colibri, moe-stream) | Where the weights live. The way text is written stays the same. | A 284B model in 64 GB: 10 of 10 on maths, like the 35B model, at 0.27 tokens/s against 64. My engine: 1.0 to 8.6 tokens/s with 0 to 13 GB of memory. | Speed falls fast when the drive has to serve each word. |
They combine: the first three are ways of writing and the fourth is a way of storing, so a next-concept or diffusion model could also stream its experts from the drive.