Nonograms benchmark
Shared nonograms across six tiers, from the generator behind Nonograms on Crest Arcade, so you can play the same kind of puzzle the models play. One answer, no hints. Compare solve rates, the rows and columns they got right, how sure they said they were, and the cost and time of a puzzle.
No model has finished a Nonograms puzzle yet. Queued: Qwen3.5 9B, Qwen3.8 27B, Qwen3.8 Flash Next, Claude Fable 5.1 and 11 more.
MethodologyProtocol, scoring & limitations
The puzzles
Every puzzle comes from the generator behind Nonograms on Crest Arcade, and every one has exactly one solution. Five tiers grow from 5 by 5 to 25 by 25 and need more rounds of line logic as they grow. The sixth, Beyond, holds 15 by 15 puzzles where line logic stalls with at least 40% of the cells still open, so only contradiction or search finishes them.
| Tier | Grid | What it needs |
|---|---|---|
| 5 by 5 | 5×5 | Line logic, a round or two |
| 10 by 10 | 10×10 | Line logic, about 3 rounds |
| 15 by 15 | 15×15 | Line logic, about 5 rounds |
| 20 by 20 | 20×20 | Line logic, about 8 rounds |
| 25 by 25 | 25×25 | Line logic, about 11 rounds |
| Beyond | 15×15 | Line logic stalls; needs contradiction or search |
Every model gets the same puzzles in the same order from set v1. A model ranks once it has answered the first 6 of every tier. After that the bench keeps going deeper, up to 100 per tier, so n grows and the intervals narrow.
An answer
The model gets every row and column clue once, as text, and answers with the whole grid (# filled, . empty, ? for a cell it couldn't work out) and a confidence from 0 to 100. It can think as long as its settings allow. There are no hints and no second try at a wrong grid. The one retry is for a reply with no readable grid of the right size, which is a formatting slip, not an answer.
We check every cell against the solution and every row and column against its clue. An answer is solved, wrong in some cells, left with cells unknown, or has no grid. Lines right counts the rows and columns whose clue the answer satisfies exactly, so a near miss still shows.
Score
The score is the solve rate in each tier, averaged so every tier counts the same, from 0 to 100. Its 95% interval adds up each tier's Agresti-Coull uncertainty. With six puzzles a tier, two models a few points apart are not reliably different; the interval narrows as more puzzles are answered.
Confidence
We ask for a confidence on every answer and score it with the Brier score: the squared gap between what it said and what happened, averaged. 0 is perfect. Always saying 50% scores 0.25. Sure but wrong counts unsolved answers it called 90% certain or more.
Cost and time
Same as the chess and sudoku benchmarks: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent: what the tokens of every request in the puzzle, retries included, would cost at the lab's list price. Local models have no list price, so they show n/a, not zero. Time is the wall-clock wait for the replies, including any queue on our side, so compare it within a route rather than across them. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.