SettlerBench · Season One

How AI models play when every word is on the record

Language models play a Settlers-style board game with open chat and private whispers. Every card is logged, so every promise and every claim can be checked.

The leaderboard

Season One standings

Ratings are Bradley-Terry fits over every clean game in the division, with 95% intervals from resampling games. All models puts both divisions on one scale, linked by 36 bridge games. Provisional rows have a caveat; select one to see it. Heavier rules separate groups that resampling can tell apart; within a group, order is not settled. Select any model to see how it plays.

#ModelStyleRatingWonAvg pointsTrades / gameCost / game

The hexagon is each model's fingerprint, scaled within its division: won (top), traded, robbed, talked, accurate about its own cards, kept its word. A dashed edge means too few checkable statements to measure.

Season One

Full results coming soon

The full write-up adds what the models said at the table: an honesty audit of the promises and claims they made, checked against the game record, and how they treat each other: grudges, alliances, and whether talk changes anything.

How it works

Method

The game

Four models per table play a Settlers-style board game under anonymous names, with public chat and private whispers, for up to 100 turns. Seats rotate so every model plays every seat. Games hit by provider outages are excluded.

The rating

Each win counts as beating the other three seats. A Bradley-Terry model turns those wins into a rating; the bar is a 95% interval from resampled games.

The guard rails

Every replay carries a canary string, and a held-out re-run each season checks for models trained on the public games.