Nations Royale benchmark
Nations Royale, the world-conquest strategy game, played against its 47 computer commanders with the war paused every five seconds for orders. The bench runs the game's own simulation, so the wars are ones you can play. Compare wars won, the share of the world held, the orders the game refused, the game's own players on the same wars, and the cost and time of a war.
No model has finished a Nations Royale war yet. Queued: Qwen3.5 9B, Qwen3.8 27B, Qwen3.8 Flash Next, Claude Fable 5.1 and 11 more.
MethodologyProtocol, scoring & limitations
The wars
Nations Royale is a real-time strategy game: 48 nations on a world map of provinces, each starting from its capital. The model plays one nation and the game's own computer commanders play the other 47. The bench runs the game's own simulation, pinned to commit e426bd2, so a later balance change in the game cannot move anyone's result. Every war stores its whole order stream, and the same war key and orders replay the same war.
| Tier | The wars | Won by the reference player | Won by the median persona |
|---|---|---|---|
| Easy | Easy world, a nation players pick most | 60 of 60 | 59 of 60 |
| Medium | Medium world (eased on the world map), the same wars | 51 of 60 | 35 of 60 |
| Hard | Hard world (eased on the world map), the same wars | 40 of 60 | 23 of 60 |
| Extreme | Extreme world, one of the twelve nations the roster rates hardest: the ceiling | 11 of 60 | 12 of 60 |
Every model fights the same wars in the same order from set v1: war k of a tier is the same war key, nation and map for everyone, and Easy, Medium and Hard are the same wars at rising difficulty. A model ranks once it has fought the first 3 of every tier. After that the bench keeps fighting further wars, up to 60 per tier, so n grows and the intervals narrow. The wins in the table count all of a tier's wars.
A decision
Play is in paused time. Every 5 seconds of game time the war stops, as the page's pause button stops it, and the model gets a fresh request: the rules and orders from the game's own manual, then the board as text. That is its provinces with garrisons and caps, every province bordering them and what it costs to take, enemy columns marching on them, its own columns, every rival's size, its gold and what it can buy, and what happened since the last decision, including which of its orders were refused and why. Nothing a player could not see on the page: rival gold, the commanders' aggression and their grace clocks stay hidden.
The model answers with up to 8 orders in the page's own verbs: SEND a share of one or more garrisons at a province, BUILD to develop one, RALLY to aim a war camp or muster. An empty answer waits. The war then runs on for 5 seconds without it. Players give about one order every 5.5 seconds, so the clock matches a typical person's pace, and 8 orders a decision is 96 a minute, the top of the range the game was tuned for.
An order the game refuses (not its province, no route, too few troops, not enough gold) is not retried. It is counted and reported back on the next decision, the way the page tells a player. The one retry is for a reply with no readable orders tag; a second one waits.
A war ends won (70% of the world held for 15 seconds, the game's seal), lost (no ground left, or the capital fell and was not retaken), or timed out at 15 minutes of game time, 180 decisions.
Score
A won war scores 1. A war not won scores half its progress toward the seal, 0.5 times the share held over 0.7, so any win beats any war that was not won, a war that times out holding 35% scores 0.25, and a lost war, which holds nothing, scores 0. The score is the average war score in each tier, averaged so every tier counts the same, times 100. Its 95% interval adds up each tier's Agresti-Coull-adjusted uncertainty.
The reference players
The game's own players fought every war of the set on the same pinned commit, under the same caps. Reference player. The game's own reference player: greedy, a pass every 0.4 s, no order limit. Reference on the model's clock. The same player held to the model's clock: one pass every 5 s, at most 8 orders. Sharp persona. Fitted to the heaviest builders among live players. Median persona. Fitted to the midpoint of live players on the easy tier. Tourist persona. Fitted to live players who never develop a province.
The reference player is far faster than any model at a 5-second clock; the same player on the model's clock shows what the clock alone costs. The personas place a model among people.
Cost and time
Same as the chess and sudoku benchmarks: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent: what the tokens of every request in the war, retries included, would cost at the lab's list price. Local models have no list price, so they show n/a, not zero. Time is the wall-clock wait for the replies, including any queue on our side, so compare it within a route rather than across them. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.
What it doesn't show
Paused time removes reaction speed on purpose. A model sees the board only every 5 seconds and cannot answer a single column the moment it appears, while the commanders act on every tick; a model that thinks for minutes per decision pays nothing for it in game time. The board is text, so reading the map is not tested. These scores compare with each other and with the game's players above, not with a person playing the web game in real time.