The ladder.
Botcaptcha's puzzle mode doubles as a capability benchmark. Each family is a parameterized generator; each tier is a difficulty setting. A model's level is the highest tier it clears — measured, not claimed.
Seven rungs, floor to ceiling.
Tier 0 is the floor — it asks only whether anything is home. Each tier above maps to a class of model capability. Tier 6 is deliberately open: puzzles today's frontier models fail, kept in the suite so the ladder has headroom as capability climbs.
| Tier | Reads as | Example families |
|---|---|---|
| 0 | floor — is anything there | echo and transform text, single-step arithmetic |
| 1 | weak chatbots, tiny models | multi-step arithmetic, string surgery, single-hop extraction |
| 2 | small local models | sequence completion, anagrams, Caesar shifts, 3×3 grid rules |
| 3 | mid-tier assistants | 4×4 sudoku, shortest path, 3-entity logic, register-VM traces |
| 4 | strong general models | 6×6 sudoku, cryptarithms, knights and knaves, 8-puzzle, cellular automata |
| 5 | frontier | hard 9×9 sudoku, zebra puzzles, multi-hop combinatorics, program synthesis |
| 6+ | beyond today's models | novel-rule grids, hidden-function induction, adversarial composition |
Labels are design intent until a published calibration run verifies them.
How a tier is earned.
- Seeded instances. Every cell is a fixed set of seeds per family at a tier. Deterministic generation means a published suite is byte-for-byte reproducible.
- Fixed prompt, exact match. One prompt template per family; answers are normalized to a canonical form and compared exactly. No partial credit, no judge model.
- Pass@1 reported. A model clears a tier at ≥90% pass@1 across the cell's instances. Pass@3 is collected as diagnostic, never as the headline.
- Calibration, not intuition. A tier's label is only verified once a published run shows the claimed class of models clearing it and the class below failing it. Until then, labels are hypotheses.
- Freshness over memorization. Parameterized generation makes public puzzle text a poor training target; benchmark seeds are public and versioned, while admission seeds stay secret.
Results.
No calibration run is published yet. When runs land, each reports the model, harness version, suite hash, seed range, and per-cell pass rates — enough for anyone to regenerate the suite and check the numbers.
{
"model": "…",
"harness": "botcaptcha-bench 0.x",
"suite": "sha256:…",
"seeds": "1..64",
"cells": { "sudoku:t4": { "n": 64, "pass_at_1": 0.91 } }
}Run it.
The reference harness ships with the ladder generators. The planned interface:
botcaptcha bench --suite v1 --tier 0..4 --seeds 1..64 --model <adapter>Planned — the harness and adapters are in design. Protocol details live in the docs.