Reference07/07
Evals: how we test OpenDesigner
A tool that says it makes your product look consistent should prove it. These are the tests OpenDesigner runs on itself. The numbers on this page come straight from the result files in the repository, and every one has its date, its version and its cost. When a test fails, it says so here.
Agent evals
91% with OpenDesigner
58% without it, on the same 8 tasks
Design evals
162 of 162 required checks
18 briefs; 50 of 54 stricter quality checks
Checker evals
57 of 59 planted mistakes caught
5 checkers; 1 false alarms on clean pages
Showcase audit
34 of 40 rebuilt pieces pass
before: ours lost 30 of 37 blind pairs
There are four kinds of test:
- Agent evals: an AI agent gets a small task, such as "set up a design system from these five answers". It does the task twice: once with OpenDesigner loaded and once without it. Checks score what it produced.
- Design evals: the engine builds a design system from each of 18 fixed briefs. No AI is involved, so the same answers always give the same result.
- Checker evals: test pages with planted mistakes, such as a one-word last line, a button that is too small or a wrong weekday. They show whether our own checkers catch each mistake.
- The showcase audit: the designs in the showcase, judged against shipped products before and after the rebuild.
Agent evals: with and without OpenDesigner
Each task runs three times with the plugin and three times without it, and the score is the share of checks that passed. Most checks need no AI judge. They check facts: a file was written, the brand color appears exactly as typed, the agent named the 4.5:1 contrast floor. Where a rule needs judgment, such as "asks one plain question", a second model votes three times and the majority counts.
| Task | With OpenDesigner | Without | Difference |
|---|---|---|---|
| contrast-floor The accessibility floor. Asked to approve body text that fails WCAG AA, the agent says so with the real ratio and offers a soft-looking color that passes. | 100% | 100% | 0 pts |
| export-tailwind The export skill. An existing system; the agent writes a Tailwind export and says how to use it. | 100% | 83% | +17 pts |
| extend-rounder The extend skill. An existing system; the agent changes one setting, records why, rebuilds and keeps the rest. | 89% | 0% | +89 pts |
| no-identity-copy The references rule. Asked to clone a real brand, the agent declines the logo and the proprietary typeface and offers structure and an original alternative. | 100% | 0% | +100 pts |
| one-question-first The interview rule. With nothing to go on, the agent asks one plain question with a recommended answer, instead of a questionnaire or guessed code. | 50% | 50% | 0 pts |
| preflight-before-share The preflight skill. A launch page with a wrong weekday, a total that doesn't add up and a pipe-to-shell install line; the agent must catch the facts before it is shared. | 92% | 83% | +8 pts |
| review-planted-page The review skill. A page with five planted problems; the agent reports them with evidence and does not edit the page (checks report, they never rewrite). | 100% | 100% | 0 pts |
| sketch-five-answers The core job. From the five sketch answers, the agent builds a whole system (tokens, exports, DESIGN.md) and keeps the brand color exactly as typed. | 100% | 47% | +53 pts |
The "Without" column is the same agent with no plugin. It shows what OpenDesigner adds. For a task any capable model already does well, such as spotting text that fails contrast, the difference is small, and the table shows that too.
Before and after a fix
When an eval finds a real fault, the fix is measured the same way. The same task and the same checks run on the skills from before the change and on the skills after it.
| Fix | Before | After | Skills |
|---|---|---|---|
| no-identity-copy Asked to copy Stripe's exact purple, the skill used the hex as the person's brand in one run of three. The interview's hard rules and guardrails now say a color named as another company's is a reference: never the person's brand, never passed to sketch. | 33% | 100% | 9dcd6ba → 8a17e54 |
Design evals: 18 briefs, no AI
Each brief is a person's five sketch answers: a name, a brand color, who uses the product, the platforms and a few feel words. The briefs include the cases that break design systems: yellow, cyan and lime brand colors, colors that are almost black or almost white, large text for older readers, and three platforms at once.
There are two tiers of checks. The required checks are WCAG 2.2 AA and the files you rely on: text contrast at 4.5:1, borders and focus rings at 3:1, 24 px targets, the exact brand color, every export, and the same result twice. The quality checks are house rules that ask for more than WCAG does. A miss there shows where the engine can improve, and the engine must tell the person about it.
| Brief | Required | Quality | Missed |
|---|---|---|---|
| b2b-dense-cobalt A dense B2B work tool with a strong blue | 9 of 9 | 3 of 3 | none |
| finance-green An invoice-approval tool with a green brand (the home page's example) | 9 of 9 | 3 of 3 | none |
| consumer-pink A friendly consumer app on web and iPhone | 9 of 9 | 3 of 3 | none |
| yellow-brand Yellow is hard: white text on it fails, so the engine must find another pairing | 9 of 9 | 2 of 3 | primary button stands out 3:1 (quality) |
| cyan-brand Light cyan, also low contrast against white | 9 of 9 | 2 of 3 | primary button stands out 3:1 (quality) |
| pure-red Pure #FF0000, a common first answer | 9 of 9 | 3 of 3 | none |
| near-black A brand color that is almost black | 9 of 9 | 3 of 3 | none |
| near-white A brand color that is almost white | 9 of 9 | 2 of 3 | primary button stands out 3:1 (quality) |
| brown-brand A warm brown: the neutrals must not turn muddy | 9 of 9 | 3 of 3 | none |
| large-text Older readers and low vision: large text | 9 of 9 | 3 of 3 | none |
| ios-only An iPhone-only app | 9 of 9 | 3 of 3 | none |
| android-only An Android-only app | 9 of 9 | 3 of 3 | none |
| three-platforms One system for web, iPhone and Android | 9 of 9 | 3 of 3 | none |
| desktop-app A desktop app | 9 of 9 | 3 of 3 | none |
| playful-rose Bold and playful | 9 of 9 | 3 of 3 | none |
| purple-saas A purple SaaS brand | 9 of 9 | 3 of 3 | none |
| dark-toggle Light and dark with a switch | 9 of 9 | 3 of 3 | none |
| lime-brand Lime, another hard light color | 9 of 9 | 2 of 3 | primary button stands out 3:1 (quality) |
Today's quality misses all have the same cause. With a light brand color such as yellow, cyan, lime or near-white, the primary button's fill stands out less than 3:1 from the page. WCAG does not require that when the button has a text label, and the engine already puts dark text on it and warns the person. It is still a weak button, so it stays listed here until the engine does better.
Checker evals: do our checks catch mistakes?
OpenDesigner checks its own work with code: a layout checker that measures every page at 13 widths, a preflight that reads a page's type and fonts, and date and name checks. Each one is tested on pages with planted mistakes and on clean pages with none. Recall is the share of planted mistakes it caught. A false alarm is a finding on a clean page.
| Checker | Planted mistakes caught | Precision | False alarms (clean pages) |
|---|---|---|---|
| layout | 20 of 20 | 95% | 1 |
| preflight | 10 of 11 | 100% | 0 |
| review | 18 of 18 | 100% | 0 |
| dates | 5 of 5 | 100% | 0 |
| names | 4 of 5 | 100% | 0 |
| Layout checker version | Date | Caught | What changed |
|---|---|---|---|
14e441e | 2026-10-15 | 11 of 20 | First version: checks 1 to 8; catches 11 of 20 planted defects; they answer owner feedback items 8, 13. |
5d436d4 | 2026-10-15 | 19 of 20 | Adds checks 9 to 13. Newly catches 8 planted defects (actions, label-break, text-left, under-control, widow); they answer owner feedback items 14, 16, 17. Removes 3 clean-page false alarms (anchor, inner-scroll). |
892ea69 | 2026-10-20 | 20 of 20 | Newly catches 1 planted defects (page-overflow). Removes 2 clean-page false alarms (clipped, text-edge). |
The second table shows the same test pages against older versions of the layout checker. Each new check came from a mistake the maintainer found in a review, such as a word left alone on the last line, or a Copy button floating above its code. The table shows each version catching more of the planted mistakes.
Bugs the checker evals found in our own checkers
The first run of these evals found bugs in the checkers themselves. Each one was fixed, and the same test pages were run again.
| Bug the evals found | Before the fix | After |
|---|---|---|
| review The page probe shared one cached color between elements, so after one semi-transparent element every later element with the same text color was measured as faded (6.72:1 reported as 3.48:1). It now hands out a copy. | 18 of 18 caught, 87% precise, 1 false alarms | 18 of 18 caught, 100% precise, 0 false alarms |
| review A page with no links or buttons was reported as a keyboard trap ("Tab never reached any control"). The Tab walk now counts what can take focus, and a page with none gets a note instead of an error. | 18 of 18 caught, 87% precise, 1 false alarms | 18 of 18 caught, 100% precise, 0 false alarms |
| layout Text for screen readers only (a 1 x 1 px box that hides its overflow on purpose) was flagged as clipped and as running past the edge. The layout checker now skips it, and an overflowing svg is named instead of "unknown". | 19 of 20 caught, 83% precise, 3 false alarms | 20 of 20 caught, 95% precise, 1 false alarms |
Two gaps are still open and stay counted as misses: the preflight finds a type scale only when its names contain "font-size", and the name check does not compare names that have a space in them.
The showcase audit, before and after
The showcase is built with OpenDesigner, so it is a test of the whole process. AI reviewers turned out to be kind: designs they scored 4 out of 5, the maintainer called "rubbish". So each design is now compared blind with a shipped product: side by side, no labels, sides in random order, scored for craft. A fresh reviewer then scores it against the B2B product screens. A design that fails stays hidden from the gallery until it passes.
| Result | Evidence | |
|---|---|---|
| Before (round 4 audit, 15 Oct 2026) | In 37 blind pairs against shipped products, ours lost 30. | AUDIT-ROUND4.md |
| After (rebuilt pieces, rounds 4 and 5) | 40 pieces rebuilt. 34 reached 4 of 5 with a fresh reviewer after a blind test; 6 stay hidden until they do. | audit records |
What these tests don't prove
- Small samples. Eight agent tasks with three runs each show a direction, not a guarantee. Agents vary from run to run.
- AI judges. Where a check needs judgment, a model votes. Its votes can differ between runs, so the checks a model judges are kept few and specific, and as many checks as possible are plain facts.
- Our own tests. We wrote these tests. A test we didn't think of can't fail. That is why you can run them yourself and add your own.
- The maintainer outranks every score. In the showcase, the maintainer's verdict decides what stays, whatever the numbers say.
Run them yourself
The design and checker evals are free and run on your machine without an AI:
python3 tools/design_evals.py python3 tools/checker_evals.py The agent evals use claude plugin eval. Every run is a real Claude Code session on your own account, so it counts against your plan or API bill (the last full run cost about the amount shown above). Start with the quick set:
python3 tools/agent_evals.py --tag smoke --runs 1To add a task: copy a folder in evals/, write the prompt in prompt.md, and add one file per check in graders/. Then open a pull request. To share your results, or a task where OpenDesigner did badly, open an eval report. The bad results help most.