Starjar benchmark
Starjar, the cosmic merge game, played one drop at a time with the jar paused while the model thinks. The bench replays the game's own physics, so the jars are the ones you can play. Compare points against the reference bot, how often the jar overflowed, forfeits, and the cost and time of a drop.
No model has finished a Starjar game yet. Queued: Qwen3.5 9B, Qwen3.8 27B, Qwen3.8 Flash Next, Claude Fable 5.1 and 11 more.
MethodologyProtocol, scoring & limitations
The jars
Starjar is a merge game in the style of Suika: drop a piece into the jar, and two pieces of a kind that touch fuse into the next kind up. A piece resting with its centre above the rim for 2.4 seconds ends the game. The bench runs Starjar's own simulation, pinned to commit b2ac8ec, so a later balance change in the game cannot move anyone's result. Every game stores its drops, and the same seed and drops reproduce the same jar.
| Tier | Start | Drops | Bot's points (average) |
|---|---|---|---|
| Daily opening | The Daily Jar of a fixed date, empty jar | 60 | 529 |
| Fresh jar | A seeded jar, empty | 150 | 1,764 |
| Midgame | The reference bot's jar after 200 drops | 100 | 1,246 |
Every model gets the same jars in the same order from set v1. A model ranks once it has played the first 2 of every tier. After that the bench keeps playing further jars, up to 60 per tier, so n grows and the intervals narrow.
A drop
Play is in paused time. Every drop is a fresh request with the whole jar: its size and rim, every piece in it with its kind, size and position, the piece in hand and where it can be let go, the score and how close the pile is to the rim. The model answers with an x position. The simulation then runs until the jar is at rest, up to 8 simulated seconds, and the next request goes out. The clock never runs while the model thinks, so a slow answer costs nothing, and it cannot drop onto a merge that is still growing the way a fast player can. The next piece is never shown, as in the game.
An x outside the legal range is moved to the nearest edge and counted as clamped. The one retry is for a reply with no readable x. A second unreadable reply forfeits the game, and the points made so far stand.
Score
The score is 100 times the points the model made divided by the points the reference bot made on the same jars under the same protocol, summed over every game it played. 100 is the bot. The 95% interval is the ratio estimator's, by the delta method. A random dropper and a lookahead player are shown for scale. In the Midgame tier only the points made after the bot's 200-drop opening count. Survived is the share of games that reached the drop cap without the jar overflowing.
Cost and time
Same as the chess and sudoku benchmarks: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent: what the tokens of every request in the game, retries included, would cost at the lab's list price. Local models have no list price, so they show n/a, not zero. Time is the wall-clock wait for the replies, including any queue on our side, so compare it within a route rather than across them. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.
What it doesn't show
Paused time removes reaction speed on purpose, so these scores compare with each other and with the reference players, not with a person playing the web game against the clock.