Wildhaven benchmark

The raids

Wildhaven is an animal-collection island game made with Oliver. Keepers raid campaign villages with a party of up to four animals whose capacities fit a budget of 12, and the village's towers fight back for 60 seconds. The bench runs the game's own raid engine, pinned to commit b87e641, so a later balance change in the game cannot move anyone's result. Every raid stores its party and every landing, and the same seed and landings replay the same battle.

An item is one of the campaign's eight villages at a campaign difficulty, plus a roster of owned animals at set levels, all drawn from a seed. Every model raids the same items in the same order from set v1, and the first sixteen of a tier meet each village twice. A model ranks once it has raided the first 16 of every tier. After that the bench keeps raiding further items, up to 64 per tier, so n grows and the intervals narrow. The players' scores in the table cover all of a tier's raids. The campaign's easiest difficulty is left out: nearly any party takes those villages, so it would not tell commanders apart.

A raid

Play is in paused time. The first request is the scouting report: the village on its grid, every structure with its level, hit points and what its towers do, the landing cells on the shore, and the roster with each animal's numbers and signature. The model names its party, checked with the game's own rules. Then the battle starts. At the start, every 3 seconds of battle time, and sooner when a structure falls or an animal goes down (never sooner than 1 second after the last look), the battle stops and the model gets a fresh request with the clock, destruction and stars, the field with its animals on it, every structure's hit points, each animal ashore with its health and target, the animals still to land and anything refused last time. It answers with any number of landings, each an animal and a landing cell, or waits. Animals land on the next tick, as the game's deploy button times them, and fight on their own.

The one retry is for an answer the game cannot use: no readable tag, an animal not in the party or already ashore, or a cell that is not a landing. The retry is told why. Whatever the second answer holds that the game accepts is carried out and the rest is refused and counted. A party refused twice is replaced by the game's default pick, and the page says so. Once the whole party is ashore there is nothing left to decide, so the battle plays out without further requests. Retreat is not offered, because it can only end a raid early. A raid ends when every structure but the walls is down, when the whole party is down, or at 60 seconds, and the model is asked at most 20 times.

Score

The score is destruction, the game's own result: each structure's hit points times its weight (the Townheart 1.5, walls 0.15), knocked down over the village's total, from 0 to 100, with bringing down everything but the walls counted as 100. A tier's score is the average over its raids, and the headline averages the tiers, each weighted the same. Its 95% interval adds up the tiers' standard errors, each tier's spread estimated with one extra raid of the widest possible spread, so a tier with one or two raids is not called certain. Stars are shown beside it: one at 50% destruction, one for the Townheart and one for clearing the village.

The reference players

Every item was also raided by these players on the same pinned commit. The game's raiders. The landing plan the game sends against keepers' islands: ashore on the side with the least tower cover, spread along it, one animal every 0.4 s, with the game's default party. Everything at once. The default party, all of it at once on the landing cell nearest the Townheart. Random landings. A random party filled to capacity, each animal on a random landing cell. Search, 64 tries. The best of 64 plans tried in the simulator. A ceiling with hindsight no player has.

The game's raiders and everything at once are the yardsticks a player would recognise. Random landings is the floor. The search is the most an attacker can expect from these rosters when it can try a plan, see the result and try again.

Cost and time

Same as the chess and sudoku benchmarks: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent: what the tokens of every request in the raid, retries included, would cost at the lab's list price. Local models have no list price, so they show n/a, not zero. Time is the wall-clock wait for the replies, including any queue on our side, so compare it within a route rather than across them. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.

What it doesn't show

Paused time removes reaction speed on purpose: a model that thinks for minutes per look pays nothing for it in battle time, while a person taps animals in as the battle runs. The field is text, so reading the island's art is not tested. Landings are on cell centres, where the game also allows any point on the shore. These scores compare with each other and with the reference players, not with a person playing the web game.