copperbench
A benchmark for language-model agents that edit KiCad projects. The tasks use real open-hardware boards, the results are checked with kicad-cli and a fixed list of assertions, and each pass rate is published together with what it cost to get. Grading is done offline and never involves a model.
Strict pass rate and cost per passing task for each model on the simple tier (1 task). The table is built from the result records on every deploy. A record is included when its suite version, task manifest hash, fixture hash and kicad-cli major version match the checked-out suite. Records that don't match are listed further down under "Not comparable" and are left out of these figures.
| # | Model | Route | Strict pass % | $ / pass | Passes | Repeats | Records | copperhead | Date |
|---|---|---|---|---|---|---|---|---|---|
| No model has been run on the simple tier yet. The 16 harness records below show that the tasks tell a no-op apart from a correct edit. | |||||||||
Strict pass rate and cost per passing task for each model on the medium tier (1 task). The table is built from the result records on every deploy. A record is included when its suite version, task manifest hash, fixture hash and kicad-cli major version match the checked-out suite. Records that don't match are listed further down under "Not comparable" and are left out of these figures.
| # | Model | Route | Strict pass % | $ / pass | Passes | Repeats | Records | copperhead | Date |
|---|---|---|---|---|---|---|---|---|---|
| No model has been run on the medium tier yet. The 16 harness records below show that the tasks tell a no-op apart from a correct edit. | |||||||||
Strict pass rate and cost per passing task for each model on the hard tier (0 tasks). The table is built from the result records on every deploy. A record is included when its suite version, task manifest hash, fixture hash and kicad-cli major version match the checked-out suite. Records that don't match are listed further down under "Not comparable" and are left out of these figures.
| # | Model | Route | Strict pass % | $ / pass | Passes | Repeats | Records | copperhead | Date |
|---|---|---|---|---|---|---|---|---|---|
| No model has been run on the hard tier yet. The 16 harness records below show that the tasks tell a no-op apart from a correct edit. | |||||||||
Two numbers are reported per model. Strict pass rate is the fraction of scoreable runs in which every required assertion passed.Cost per passing task is the model's total spend in USD divided by its number of passes. A run is unscoreable when a required assertion couldn't be evaluated at all (ERC with no kicad-cli installed, for example); those runs are excluded from the rate and shown after the plus sign in the Passes column.
- Open-weights model
- Costed route: price known from the pricing table, exact model version pinned
- Uncosted route: no price available, cost recorded as null
Simple
1 taskSingle-sheet boards. One edit, with one other place it has to be carried through to.
Tier definitionMedium
1 taskEdits that have to respect a stated constraint, and requests that should be refused with a reason that points at the file.
Tier definitionHard
0 tasksMulti-sheet hierarchical designs, where a change on one sheet has to be followed through on the others.
Tier definitionVariants
0 tasksThe same task on a deterministically altered copy of its fixture. A model that does much better on the original than on the variant has probably seen the original before.
Memorization variantsDiscrimination check
Each task is run twice without any model before it is used. The no-op run makes no changes and has to fail, otherwise the assertions aren't checking anything. The reference solution run applies a known-correct edit and has to pass, otherwise the task can't be solved as written. If a required assertion couldn't be evaluated at all (ERC with no kicad-cli installed, say), the run is marked unscoreable instead. Where a run didn't come out as expected, the cell shows which assertion stopped it.
| Task | Tier | Expected | No-op run | Reference solution | Date |
|---|---|---|---|---|---|
| do-rename-net | simple | edit | fail new-net-exists | pass | 2026-09-07 |
| do-budget-refusal-pullup | medium | refusal | fail refused | pass | 2026-09-07 |
Not comparable
These records were produced under a different suite version, task manifest, fixture hash or kicad-cli major version, so they can't be compared with the rows above. They are kept for reference and don't contribute to any figure on this page.
Status
- doneStandard: The task format, the JSON Schemas, the assertion vocabulary, the fixture policy and how cost is accounted for.STANDARD.md →
- doneFixtures: Real open-hardware boards under permissive licenses, pinned to an upstream commit and content-hashed, with a baseline ERC and DRC report recorded for each.FIXTURES.md →
- doneWorked tasks: One edit task with a diff-size bound sized for its fixture, and one task where the right answer is to refuse and cite the constraint that makes the request unsafe.tasks/ →
- doneValidator, sandbox, scorer, records: Everything except the model call works and is tested offline: schema validation, hash checks, the three evidence readers, verdicts, append-only result records and re-scoring of existing records.scripts/
- nextAgent mode: Not implemented yet; it needs a provider credential. The scorer already handles a missing kicad-cli by marking the affected assertions unevaluable rather than passing them.
- nextAggregate report and LEADERBOARD.md: pass@k with spread across repeats, correct and false refusal rates, a list of failures to look at, and the generated markdown leaderboard with a check that the committed file matches a regeneration.
- pendingFirst smoke matrix: A first run of direct-API models, three repeats each. This page and the paper's tables will be generated from those records.
Fixtures
The boards the tasks run on. Each is copied unchanged from a permissively licensed open-hardware project at a specific upstream commit, keeps its original LICENSE file, and is credited in NOTICE. Its content hash is recorded so a run can be checked against the exact files it saw. Only the design files go into the sandbox; the provenance notes and the baseline ERC and DRC reports are kept outside it, where the model can't read them.
| Fixture | Upstream | Tier | Sheets | Symbols | Schematic lines | License |
|---|---|---|---|---|---|---|
| antmicro-jetson-orin-baseboard | Antmicro Jetson Orin Baseboard | hard | 10 | 669 | 364,016 | Apache-2.0 |
| antmicro-microphone-board | Antmicro PDM Microphone Board | simple | 1 | 12 | 8,489 | Apache-2.0 |
| openlighting-dmxsun-baseboard | Open Lighting Project rp2040-dmxsun (2-slot baseboard) | simple | 1 | 45 | 15,122 | Apache-2.0 |
| zhiayang-mikoto | mikoto nRF52840 module | medium | 1 | 67 | 19,991 | Apache-2.0 |