Mines benchmark
No-guess minesweeper boards in four sizes, one stateless request per turn. The boards come from the engine behind Mines on Crest Arcade, so you can play the same game the models play. Compare win rates, how much of each board they cleared, the flags they got wrong, and the cost and time of a board.
No model has finished a Mines board yet. Queued: Qwen3.5 9B, Qwen3.8 27B, Qwen3.8 Flash Next, Claude Fable 5.1 and 11 more.
MethodologyProtocol, scoring & limitations
The boards
Every board comes from the engine behind Mines on Crest Arcade and is no-guess: from the opening, some covered cell can always be proven safe from the numbers and the mine count. Every model gets the same boards in the same order, with the same cell already opened.
| Tier | Board | Mines | Density | Turns a perfect player needs (median) | Turn cap |
|---|---|---|---|---|---|
| Beginner | 9×9 | 10 | 12.3% | 3.0 | 16 |
| Intermediate | 16×16 | 40 | 15.6% | 6.5 | 32 |
| Expert | 30×16 | 99 | 20.6% | 19.0 | 64 |
| Master | 30×24 | 200 | 27.8% | 33.0 | 120 |
Set v1 holds 100 boards per tier. A model ranks once it has played the first 6 of every tier. After that the bench keeps playing further boards in the same order for every model, so n grows and the intervals narrow.
A turn
Every turn is a fresh request: the rules, the mine count, the whole visible board with numbered rows and columns, the turn number and one line on what the last reply did. Nothing else carries over. The model answers with up to 40 actions, each one reveal, flag or unflag a cell. They are applied in order and stop at a mine. Opening every safe cell wins; flags are never required.
An action that changes nothing, on a cell already open or already flagged, is wasted. Revealing a flagged cell, flagging an open one, a cell off the board or a line we can't read is illegal. Both are reported back the next turn. The one retry in a turn is for a reply with no readable actions, which is a formatting slip, not a move.
A game also ends unfinished at the tier's turn cap, about twice what a perfect player needs, or after 5 replies in a row that change nothing.
Score
The score is the win rate in each tier, averaged so every tier counts the same, from 0 to 100. Its 95% interval adds up each tier's Agresti-Coull uncertainty, so a perfect or an empty tier still carries some. Cleared is partial credit: the share of safe cells opened beyond the opening. Wrong flags counts flags left on safe cells, mines the model was sure of that were not there.
Cost and time
Same as the chess and sudoku benchmarks: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent: what the tokens of every request in the board, retries included, would cost at the lab's list price. Local models have no list price, so they show n/a, not zero. Time is the wall-clock wait for the replies, including any queue on our side, so compare it within a route rather than across them. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.
What it doesn't show
Every request shows the whole board, so the test is deduction and bookkeeping, not memory. Big boards are long games: a model that is careful but slow can run out of turns, and that counts as unfinished, not as a loss to a mine.