BENCH / LOGIC & RELIABILITY

Sudoku benchmark

Name: Glimmer Index Bench: Sudoku
Creator: Glimmer Index

Shared grids across seven difficulty tiers, from Cozy Sudoku's engine. One answer, no hints. Compare solve rates, confidence, cost and response time.

15 model settings486 answers261 solvedUpdated 2026-10-28

Methodology Download CSV ↓

Sudoku leaderboard

#	Model	Score	Solved per tier (of 6)CozGenSteTriDevExpBey	Cells right	Broke the rules	Sure but wrong	Calibration	API cost / grid	Output tokens / grid	Sec / grid
1	Claude Opus 5.5reasoning medium	100	6666666	100.0%	0	0 of 0	0.000	$0.244	11,827	101
2	Claude Fable 5.1reasoning medium	93	6664566	97.6%	2	2 of 3	0.047	$0.894	16,999	173
3	GPT-6 Solreasoning medium	90	6666464	93.7%	2	2 of 4	0.048	$0.149	12,925	142
4	Claude Sonnet 5.5reasoning medium	88	5556466	99.5%	1	5 of 5	0.111	$0.138	13,462	91
5	GPT-6.1 Solreasoning medium	83	6645545	94.1%	7	5 of 7	0.119	$0.112	8,723	195
6	GPT-6 Astrareasoning medium	76	5454464	94.0%	9	10 of 10	0.238	$0.453	6,987	226
7	Qwen3.8 Flash-Nextlocal, thinking low · open weights	48	4422440	45.8%	0	0 of 22	0.000	n/a	65,074	3,442
8	Claude Haiku 5.5reasoning medium	31	3113221	93.1%	8	29 of 29	0.643	$0.015	30,157	111
9	GPT-6.1 Terrareasoning medium	7	3000000	10.2%	2	2 of 39	0.048	$0.015	1,849	26
10	GPT-6 Lunareasoning medium	0	0000000	8.6%	3	3 of 42	0.075	$0.0024	1,779	24
11	gpt-oss 20Blocal, thinking low · open weights	297 of 42	0110000	53.8%	0	2 of 5	0.400	n/a	29,731	3,050
12	Qwen3.8 27Blocal, no thinking · open weights	014 of 42	0000000	29.7%	3	7 of 14	0.619	n/a	11,808	1,356
13	Ministral 3 8Blocal, no thinking · open weights	015 of 42	0000000	13.7%	0	15 of 15	1.000	n/a	108	20
14	Gemma 3 4Blocal, no thinking · open weights	015 of 42	0000000	9.6%	0	15 of 15	0.913	n/a	172	20
15	Qwen3.5 9Blocal, no thinking · open weights	015 of 42	0000000	8.5%	0	0 of 15	0.000	n/a	2,342	220

Scroll for all metrics. Score is the solve rate averaged over the seven tiers, each tier weighted the same, from 0 to 100. A model still working through the set shows how many of the 42 grids it has answered, and ranks after the models that have finished. Click a column to sort.

Cost frontierAPI-equivalent cost per grid. Local models have no list price and are omitted. Bars show 95% intervals.

Closed weightsOpen weightsPareto frontier

Time frontierObserved response time, including reasoning and queue wait. Routes and hardware differ.

Closed weightsOpen weightsPareto frontier

The Pareto frontier shows where no other model scores at least as well for less cost or time, with one strict improvement. Partial sets are included and may cover different puzzles. Use completed sets for direct comparisons.

Individual answers and confidence

Confidence and solve rateBelow the line: stated confidence exceeds the observed solve rate.

Closed weightsOpen weightsConfidence matches results

Output tokens by difficulty

Average output tokens per grid, reasoning included. Token counts include reasoning when reported by the provider.

Model	Cozy	Gentle	Steady	Tricky	Devious	Expert	Beyond
Claude Opus 5.5	4,521	8,296	12,067	16,782	8,398	11,761	20,962
Claude Fable 5.1	5,128	10,788	16,626	11,143	9,188	23,410	42,711
GPT-6 Sol	4,229	9,606	11,675	15,702	7,839	12,594	28,828
Claude Sonnet 5.5	5,884	10,288	14,392	17,670	8,807	14,058	23,134
GPT-6.1 Sol	3,938	6,007	8,432	8,794	6,595	9,598	17,701
GPT-6 Astra	2,768	4,133	10,358	5,546	8,886	5,917	11,303
Qwen3.8 Flash-Next	34,346	50,644	81,530	80,861	50,950	65,709	91,478
Claude Haiku 5.5	15,853	22,690	34,434	33,600	20,784	38,906	44,833
GPT-6.1 Terra	3,649	1,948	1,912	1,182	2,004	1,447	799
GPT-6 Luna	1,826	1,485	1,971	1,748	1,836	2,203	1,381
gpt-oss 20B	8,212	18,046	16,661	64,841	15,941	64,615	19,800
Qwen3.8 27B	15,600	11,450	10,994	10,524	11,394	10,067	12,623
Ministral 3 8B	108	108	108	108	108	108	108
Gemma 3 4B	148	256	176	204	139	154	139
Qwen3.5 9B	2,071	3,000	3,000	3,000	1,553	2,302	1,606

Every grid

One square per answer, filled when it solved the grid. Columns run from the easiest tier to the hardest. Click a square to see the answer.

Model	coz	2	3	4	5	6	gen	2	3	4	5	6	ste	2	3	4	5	6	tri	2	3	4	5	6	dev	2	3	4	5	6	exp	2	3	4	5	6	bey	2	3	4	5	6
Claude Opus 5.5
Claude Fable 5.1
GPT-6 Sol
Claude Sonnet 5.5
GPT-6.1 Sol
GPT-6 Astra
Qwen3.8 Flash-Next
Claude Haiku 5.5
GPT-6.1 Terra
GPT-6 Luna
gpt-oss 20B
Qwen3.8 27B
Ministral 3 8B
Gemma 3 4B
Qwen3.5 9B

MethodologyProtocol, scoring & limitations

The grids

Every grid comes from Cozy Sudoku's generator, and every one is checked to have exactly one solution. The first six tiers are the game's own difficulty bands. A grid's band is set by the hardest technique it needs, not by how many digits are given. The seventh tier, Beyond, holds grids that none of those techniques finishes. We generated 400 of them and kept the ones that need the most trial and error.

Tier	Hardest technique needed	Givens
Cozy	Singles only	31 to 42
Gentle	Scanning	25 to 30
Steady	Locked candidates	27 to 32
Tricky	Hidden pairs and triples	24 to 27
Devious	Wings and fish	26 to 41
Expert	Deep wings and fish	23 to 28
Beyond	No human technique finishes it	21 to 25

Every model gets the same 6 grids per tier from set v1. The grids are generated from fixed seeds, not taken from a puzzle book, so a model can't have memorised them. The set is published with the results, so when it's been around long enough to be trained on, we'll move to a new one and label it.

An answer

The model gets the grid once, as nine rows, and answers with the whole solution and a confidence from 0 to 100. It can think as long as its settings allow. There are no hints and no second try at a wrong answer. The one retry is for a reply with no readable grid in it, which is a formatting slip, not an answer.

We check every cell against the solution. An answer is either solved, wrong in some cells, left with cells empty, or one that changed a given digit. Cells right counts every blank cell filled correctly, so a near miss still shows up. Broke the rules counts complete grids that repeat a digit somewhere: the model said it was done without checking. On a model's own page, cell accuracy is split by the technique each cell needed.

Score

The score is the solve rate in each tier, averaged so every tier counts the same, from 0 to 100. Its 95% interval adds up each tier's uncertainty. With 6 grids a tier, two models a few points apart are not reliably different.

Confidence

We ask for a confidence on every answer and score it with the Brier score: the squared gap between what it said and what happened, averaged. 0 is perfect. Always saying 50% scores 0.25. Sure but wrong counts unsolved answers it called 90% certain or more.

Cost and time

Same as the chess benchmark: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent, what the tokens would cost at the lab's list price. Local models show n/a. Time is the wall-clock wait for the answer. A request that fails on our side is never counted against the model.

Data: JSON · CSVUse with credit to Glimmer Index.