BENCH / OPEN WORLD FROM PIXELS

Minecraft benchmark

Minecraft: Java Edition, played from the screen alone. Each model starts a new survival world on Normal with nothing, and is told one thing: get a diamond. It sees a screenshot, answers with keys, mouse moves and clicks, and the world stays frozen while it thinks. Every model plays the same seeded worlds from the same spawn block. The score is how far up the ladder from nothing to a diamond it gets, read from the game's own records. Not affiliated with Mojang or Microsoft.

7 model settings2 worlds11,100 turns22 in progressUpdated 2026-11-21

Minecraft leaderboard

Select a column to sort. Scroll sideways for all metrics →

#ModelScore95% intervalDiamondsDiaBestWooden pickaxeStone pickaxeIron pickaxeDeaths$ / turnWorldsAPI cost / worldSec / turn
1
GPT-6 Lunareasoning medium
382 of 30Cobblestone50%0%0%16.0$0.00042$0.9376.7

Score is the mean of each finished world's highest step, scaled so a diamond is 100. The interval is a normal approximation and appears once a model has finished two worlds. A model ranks once it has played the first 3 worlds of every tier; until then it shows how many of those it has played and ranks after the models that have. Past that the bench keeps playing further worlds in the same order for every model, and every finished one counts, so n grows and the intervals narrow.

Cost frontierAPI-equivalent cost per world. Local models have no list price and are omitted. Bars show 95% intervals.
Closed weightsOpen weightsPareto frontier
204060$0.100$0.300$1.00$3.00API-equivalent USD / world (log scale)ScoreGPT-6 Luna
Time frontierObserved response time, including reasoning and queue wait. Routes and hardware differ.
Closed weightsOpen weightsPareto frontier
2040606.06.57.07.5Seconds / turnScoreGPT-6 Luna

The Pareto frontier shows where no other model scores at least as well for less cost or time, with one strict improvement. Partial sets are included and may cover different items. Use completed sets for direct comparisons.

By tier

The count after each result is n, the worlds finished in that tier.

ModelSurvival worlds
GPT-6 Lunareasoning medium0 of 2score 38
Every world

Every world

One square per world, filled when the model got a diamond. Columns run tier by tier, in the order every model meets them.

ModelDia2345678910
GPT-6 Luna
MethodologyProtocol, scoring & limitations

The task

Minecraft: Java Edition 1.21.4, a new survival world on Normal difficulty, nothing in the inventory, and the instruction to get a diamond. The prompt describes the controls and the action format, and nothing about how to play: no recipes, no tool order, no ore depths. Every model plays the same 10 worlds (fixed seeds, one spawn block for everyone) in the same order, and ranks once it has finished the first 3.

A turn

The model gets one 960x540 screenshot of the game window, the notes it wrote last turn (up to 1,500 characters, its only memory) and its last 5 actions. It answers with up to 8 actions covering at most 10 seconds of game time: keys and mouse buttons held for a duration, camera turns in degrees, single key presses, clicks at screen pixels while a menu is open, and the scroll wheel. Chat and commands are not available, and the player is never an operator. The world runs only while the actions play out and is frozen (the game's own tick freeze) while the model thinks, so a slow model loses no game time. One retry follows an answer with no readable action.

A world

A world ends when the player holds a diamond or after 2,000 turns. It is played in segments of 100 turns, each from the last one's saved world, so a failure on our side (a host, a connection, a timeout) only replays from the last save and never counts against the model.

Scoring

The ladder is wood, crafting table, wooden pickaxe, cobblestone, stone pickaxe, iron ingot, iron pickaxe, diamond. Each step is read from the game's own records, never from the screenshots: the advancements the server announces, the player's inventory after every turn, and the saved advancements and statistics at each save. A world scores the highest step it reached, scaled so a diamond is 100.

Where it runs

The vanilla client and a vanilla dedicated server run on our own DGX Spark, rendered in software on a virtual screen, with keyboard and mouse input through the X server. Not affiliated with Mojang or Microsoft.

Cost and time

Same as the chess and sudoku benchmarks: Claude models run through Claude Code, OpenAI models through the Codex CLI and Google models through the Antigravity CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent: what the tokens of every request in the world, retries included, would cost at the lab's list price. Local models have no list price, so they show n/a, not zero. Time is the wall-clock wait for the replies, including any queue on our side, so compare it within a route rather than across them. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.

Data: JSON · CSVUse with credit to Glimmer Index.

In progress