Method first
AgentBench
Does giving a coding agent Dowel’s agent files and MCP server change what it builds? The same prompt runs in two copies of the same project, one with them and one without, and both diffs are scored by programs, not by eye.
Results
No runs have been published yet.
The harness and the tasks are below, and anyone can run them. Results appear here only when a run — every result, transcript and diff, failures included — is committed under results/published/. There are no numbers on this page until then.
Two conditions, one difference
It compares conditions for one agent; it does not rank agents. That measures what Dowel controls — whether its agent support helps — and avoids the disputes and terms-of-service problems a vendor ranking brings.
Every workspace is built from one commit: the local scaffolder, CLI, MCP server and registry, never what is on npm, with npm dependencies pinned to the versions the components are tested with. The agent runs outside the repository, with the runner’s own user settings and MCP servers excluded.
| Condition | The project has |
|---|---|
| with-dowel | Dowel initialised, the task's components installed, the files dowel agents writes (AGENTS.md, .dowel/, a Claude skill, a Cursor rule) and an MCP config for the server. |
| without | Dowel initialised and the same components installed. No agent files, no MCP config — preparation fails if any are present. |
What is measured
Each check is a program, so two people scoring the same diff get the same numbers. A check that did not run is recorded as skipped and left out of n, never counted as a pass.
| Metric | How | Better |
|---|---|---|
| Route created | The page file the task asks for exists after the run. | higher |
| Typecheck passes | tsc --noEmit exits 0 in the workspace. Only measured when dependencies were installed. | higher |
| Type errors | Errors tsc --noEmit reports. | lower |
| Audit findings | Findings from dowel audit over the changed files: palette and arbitrary colours, inline colours, off-scale sizes, physical directions, and native elements where an installed component replaces them. | lower |
| Invented components | Imports from the Dowel UI or blocks alias naming something the registry does not have. | lower |
| Uninstalled imports | Imports of registry components that exist but are not installed, so would not resolve. | lower |
| Expected components used | Share of the task's expected components imported, directly or through a block or component built from them. | higher |
| Accessibility lint errors | Errors from the jsx-a11y rules this repository lints its own components with, over the changed .tsx files. | lower |
| Files changed | Files added, modified or deleted since the baseline. Context, not a score. | lower |
Not measured
- Visual quality. Whether a page looks right is a judgement, and a number standing in for one is the figure nobody could reproduce. It is left out until there is a way to measure it that two people would agree on.
- Time to completion, for the same reason. The agent’s own time and cost are recorded for whoever pays, not compared.
next buildand axe on the rendered page. Planned, not in this first harness; it checks types and the static accessibility rules instead.- Whether the page works. Recall against a short list is a proxy for using the right components. The diffs and transcripts are kept for a person to read.
The tasks
Ten prompts, typed as a person would, identical in both conditions. Each project also has components installed that are easy to confuse with the right one — dialog beside alert-dialog, progress beside meter — because choosing between them is where agents go wrong. The expected list holds only choices the component guidance settles.
Account settings
src/app/settings/page.tsx“Build an account settings page at /settings where people can change their display name and email address, choose which emails they get (product updates, security alerts, weekly summary), and save their changes.”
Admin users table with bulk actions
src/app/admin/users/page.tsx“Create an admin page at /admin/users that lists users with their role and status, where an admin can select several users at once and either deactivate them or change their role in one go.”
- Expected
- data-tablebutton
- Installed
- data-table, table, checkbox, dropdown-menu, select, button, badge, avatar, alert-dialog, dialog
AI chat
src/app/assistant/page.tsx“Add an AI assistant chat page at /assistant where people type a question, see the reply appear as it streams in, and can stop a reply that is still being generated. Fake the model's replies for now.”
- Expected
- ai-conversationai-messageai-prompt-input
- Installed
- ai-conversation, ai-message, ai-prompt-input, ai-response, button, card, textarea, empty-state
Analytics dashboard
src/app/analytics/page.tsx“Make an analytics dashboard at /analytics with headline numbers for revenue, active users and churn compared with last month, a way to switch between the last 7, 30 and 90 days, and a table of the top pages by views.”
- Expected
- metric-deltatable
- Installed
- card, metric-delta, select, toggle-group, tabs, table, button, badge
Billing: usage and invoices
src/app/billing/page.tsx“Create a billing page at /billing that shows the current plan, how much of this month's API request quota and storage have been used, and a list of past invoices with a download link for each.”
Confirm before delete
src/app/projects/page.tsx“Make a projects page at /projects that lists a few projects and lets people delete one. Deleting cannot be undone, so ask them to confirm first.”
- Expected
- alert-dialogbutton
- Installed
- button, card, alert-dialog, dialog, table, toast
CRM contacts table with filters
src/app/contacts/page.tsx“Build a contacts page at /contacts for our CRM: a table of contacts showing name, company, email, owner and status, with a search box and filters for status and owner.”
- Expected
- data-tableinput
- Installed
- data-table, table, input, select, combobox, badge, button, dropdown-menu, avatar, checkbox
Login page
src/app/login/page.tsx“Add a login page at /login with email and password fields, a "remember me" option, a link to reset a forgotten password, and a sign-in button.”
Multi-step onboarding form
src/app/onboarding/page.tsx“Build a three-step onboarding flow at /onboarding: first the person's name and role, then their team's name and size, then which features they want to try. Show which step they are on and let them go back to a previous step.”
Data table with search and pagination
src/app/orders/page.tsx“Add an orders page at /orders with a table of orders (order number, customer, date, total, status) that can be searched by customer name and paged through 20 orders at a time.”
- Expected
- data-tableinput
- Installed
- data-table, table, pagination, input, button, select, badge
Run it yourself
The harness, the tasks and the method are in packages/agentbench. A run writes every result, transcript and diff; publishing one means committing all of it, failures included.
pnpm install && pnpm buildcd packages/agentbenchpnpm bench run --agent noop --tasks login # the pipeline, for freepnpm bench run --agent claude-code --install --repeat 3 \ --yes-i-understand-this-costs-money # billed to your accountpnpm bench report <runId>What could make a result wrong
- Small samples. Ten tasks, and as many runs per cell as were paid for. Read n before the mean, and the per-task rows before the totals.
- Agents are not deterministic. The same prompt gives different diffs. Repeats are the only defence.
- Task selection. The tasks were written by the people who wrote the components, about screens the library covers. That favours the with-dowel condition on recall by construction.
- The expected lists are a judgement, kept short and in the open to be argued with.
- Static checks only. Nothing is rendered; a page can pass all of this and be broken in a browser.