Gauntlet Kit: a loop that keeps building until it wins

TL;DR

• It turns "build me X" into a loop: build, get judged by an independent critic against a real quality bar, rebuild, until it wins.
• Four ideas do the work: a concrete bar you can open or measure, builder ≠ critic, blind A/B with labels stripped and order randomized, and no round limit.
• The article gives a prompt; this gives a runnable protocol where loop state lives on disk, survives compaction, and the gate fails without a verdict plus evidence artifact.
• Zero dependencies, Node 18+; Playwright only if you want automatic screenshots.

What it is and who it is for

gauntlet-kit packages the Gauntlet Loop for any coding agent — Claude Code, Codex, Cursor, Copilot, Gemini CLI, or anything that reads AGENTS.md. The method comes from Matt Shumer, who built a 55,000-line, AAA-looking first-person shooter from a single prompt by having agents compare their work against real Call of Duty screenshots and keep going until they won; Ben Tossell wrote up the method.

It is for the person whose agent produces something that works and looks finished in one round, and who has no way to tell whether it is actually good. The README states the whole interface in two lines: point your harness at the repo, say what you want built.

The four ideas

1. A concrete bar. Not "make it great" — an artifact you can open, run or measure. Adjectives let an agent declare victory on round one.
2. Builder is not the critic. A builder remembers why it made each choice and will defend it; verdicts come from a fresh context that never saw the reasoning.
3. Blind A/B. Strip the labels and randomize the order, or the critic scores "ours" generously and the loop ends early.
4. No round limit. The bar decides when you are done, not a schedule.

What this repository adds

The article gives a prompt; this gives a protocol.

How it runs

Phase 0  lock a bar        →  concrete, inspectable, external, above your level
Phase 1  decompose         →  4–12 independently judgeable pieces
Phase 2  the gauntlet      →  per piece: builder → capture → blind A/B → fresh critic → gap → repeat
Phase 3  smoothing         →  make many agents' work feel like one product
Phase 4  stop              →  won, diminishing returns, or budget. Never "it looks good now"

Running it takes three commands and a way to watch:

node scripts/gauntlet.mjs status     # every piece, round, current gap
node scripts/gauntlet.mjs progress   # writes progress/live.html
node scripts/gauntlet.mjs gate       # exits 1 unless every piece has evidence

The full protocol is gauntlet/LOOP.md and the hard constraints are gauntlet/RULES.md. Both are written for an agent to execute rather than for you to read, though they are short. The ledger — pieces, rounds, verdicts, evidence — lives in gauntlet/state/, and agents/ holds the prompt cards for decomposer, builder, critic, blind judge and smoother.

The README's advice on watching is one line: watch from a distance, because interrupting a builder mid-round costs you the round.

Honest limits

The README lists five, and they are the useful part:

• A bad bar produces a confident bad result. The bar is the one thing worth your attention.
• It is expensive. Many agents, many rounds, real tokens: compute traded for quality.
• Some things cannot be barred. If the output cannot be inspected, compared or measured, the method has nothing to grip.
• Critics can still be wrong. They are calibrated to be harsh and to default to the reference, which costs extra rounds — deliberately, because the opposite failure ends the loop early.
• done is not won. A piece that stalls at diminishing returns is reported as stalled, with its residual gap.

When to use it, when not to

Use it when the hard part is taste rather than correctness — a landing page, a game feel, a CLI's ergonomics — and you have a real reference to compare against. The bar can be a competitor's page, a screenshot, a spec or a measurement, but it has to be something a critic can look at.

Do not use it when the output cannot be inspected or measured, and do not use it to skip Phase 0: a vague bar is the failure mode that makes a confident bad result. And read status before believing a win — done is not won.

Sources

1. gauntlet-kit — repository — the four ideas, protocol phases, layout, honest limits quoted above.
2. README.md — commands, layout, credit and licence (MIT).

Project page: Gauntlet Kit

It is an independent, unaffiliated packaging of the method.

FAQ

Does it need an API key or a service? No. The README says zero dependencies and Node 18+; Playwright is only needed for automatic screenshots.

What stops the loop? The bar, not a round limit — the README lists won, diminishing returns, or budget as the three stops, and never "it looks good now".

Why is a tie counted as a loss? By design, so a generous critic cannot end the loop early on a comparison that was actually even.

Where does the loop state live? On disk in gauntlet/state/, so it survives compaction and resume, and the gate command fails when a piece has no verdict plus evidence.