Real games, or nothing
Each model builds a complete, playable HTML5 game from one prompt. If it does not run, it scores zero.
Can AI build a game?
An open benchmark: 580 builds by 9 models across 10 retro games.
Paper and data public since April 2026
The short version
- What it is
- A test of whether AI models can build a working game from one prompt, with no human help.
- Who for
- Anyone choosing an AI model for code, or using AI to judge AI.
- What I did
- I designed it, ran it, audited the data and wrote the paper. All of it is open.
- What happened
- 580 builds by 9 models across 10 games. In round two, 3 judges from different model families scored each build. The top two models finished 0.04 points apart.
Benchmark V2, published April 2026. Consensus of three AI judges (Sonnet, Gemini Pro and GPT-5.4), scores out of 10.
- What I’d change
- Lock the judges and the rubric before scoring. Rounds one and two used different judges, so their scores do not compare.
Two minutes? Read the five lines, then what it means for product work.
Start with the games. You can play every build.
Start with the method, then what it does not show.
Ten small retro game screens
An illustration of the kind of games the models built. The real builds are in the library.
Why this benchmark exists
AI coding ability is hard to judge honestly.
Public tests such as HumanEval and SWE-bench can be gamed, because models train on the test sets. Reviews of the “I tried the new model” kind are noisy.
The deeper problem is the judges. Ask one AI to score another, and it tends to score high. In round one, Gemini Flash raised the consensus score by about 2.4 points for every builder. I only saw it by checking against two other judges.
Round one (V1), published with the paper in April 2026. Flash’s scores compared with the other two judges.
So the question was simple. Can today’s models build something that runs, with no help? And can we measure it in a way that is hard to fake?
Four choices that make it hard to game
The rules: no human help during a build, and no retries.
A planner and a builder
One model writes the spec and another builds it. Round two crossed five planning set-ups with eight builders: 40 builds per game. The set-ups were Opus, GPT-5.4, Gemini Pro, GLM-5, and no spec at all.
Three judges from different families
Sonnet, Gemini Pro and GPT-5.4 score each build on their own. When they disagree, the gap is the signal. One Tetris build got 4.4, 6.0 and 9.8.
A real browser for every build
Playwright opens every game in a headless browser. It catches script errors and dead controls before any judge sees them. In round two, 2 of 400 builds did not finish.
Benchmark V2, published April 2026.
Easy, medium, hard
One game from each part of the difficulty curve. You can play the builds.
-
Easy
7.81 / 10Snake
It is in a thousand tutorials. Every builder shipped a playable game, and scores sat close together.
Play the Snake builds -
Medium
7.63 / 10Breakout
Bouncing-ball physics. Strong builders coped. Smaller ones slipped on the edge cases.
Play the Breakout builds -
Hard
5.50 / 10Pac-Man
Four ghosts that each behave differently, in a maze. Models that did well on Snake dropped more than two points.
See the Pac-Man scores
Benchmark V2, published April 2026. Three-judge consensus out of 10, across all 40 builds of each game.
Nine lessons from 580 builds
In the order I learned them.
Decide what to measure before you start
Rounds one and two used different judges, so their scores do not compare. Lock the rubric, the judges and the weights before you score anything.
Context changes who wins
In round one, builders saw earlier game code in the same session. In round two, each build had a fresh session. That one change moved Opus from third to first.
Familiar tasks flatter the models
Snake and Donkey Kong sat about 2.6 points apart on average, with the same models and specs. No choice of model closed that gap.
Specs can hurt
With no spec, Haiku built Snake at 7.93 out of 10. Given a detailed plan from Opus, it fell to 1.43. Most builders did worse with specs. Only GPT-5.4 clearly gained.
Cheap, fast judges agree with everything
Gemini Flash was the fastest, cheapest judge in round one. It rated almost everything 9 or 10, and scored broken builds above working ones.
AI cannot grade AI alone
AI judges read code. They see mechanics, not play. Many builds they rated 8 or more cannot be won. A 30-second playtest would show it.
Test the pair, not the model
The best builder alone is not always best in a pipeline. Some planner and builder pairs worked well together, and some did not.
One shot only works for well-known tasks
Famous games such as Snake and Pong come out well first time. Obscure ones, such as Donkey Kong with its barrels, do not, whatever the spec or price.
Orchestration is the hidden cost
The builder and judge calls cost about $364. The agent framework around them, dispatching, polling, summarising and retrying, cost about $753. That is $1.83 of coordination for every $1 of real work. A framework that resends the whole conversation each turn grows this cost with the square of its length.
Benchmark V1 and V2, published April 2026. Scores out of 10. Lesson nine is the pipeline’s own bill: about 60% coordination, 35% builders, 5% judges. Each lesson has its data on the lessons dashboard.
What it does not show
The honest edges, said plainly.
- AI judges read code, not play. A human re-rating had covered 104 of the 580 builds when the paper came out in April 2026.
- One shot only. It says nothing about agents that test and fix their own work.
- Round one and round two scores do not compare. The judges changed.
- Familiar games flatter every model. A good Snake score says little about new problems.
- The judges disagree more than a leaderboard suggests. One Tetris build scored anywhere from 4.4 to 9.8.
- One run per build, with no retries. Another run could land differently.
Benchmark V2 and the human re-rating count, as published in April 2026.
What this means for product work
The same questions come up whenever a team ships an AI feature.
Spec quality is part of the product
If a detailed spec can sink a builder, prompt design is product work. The same goes for retrieval prompts, tool descriptions and system prompts.
AI judges need a second opinion
Use at least two judges from different model families. Check they agree on the order. Re-run when you change the panel.
Test the pairs you will ship
Agent pipelines are planner and builder pairs. Picking the best of each on its own can lose to a pair that works well together.
People set the bar
AI judges grade code; people grade games. Keep a person checking the outputs that matter.
The data is open
Check my working. Everything below is public.
Choosing a model, or judging AI with AI?
Tell me what you are testing. I read every message.