Why hardware agents need verifiable RL environments

T. Chimbault, Y. Alloul
June 17, 2026

Our long-term goal is to make hardware engineering work automatable by agents. We are starting with the missing training substrate: verifiable reinforcement-learning environments where models produce hardware artifacts and tool-backed verifiers score those artifacts under engineering constraints.

This memo is not about a benchmark. It is about the missing training substrate for hardware-capable models: environments where models produce real hardware artifacts, verifiers score those artifacts, and the resulting signal can be used for post-training.

The advantage of hardware is that reward can be anchored outside the model: in simulation, physical constraints, connectivity, manufacturability, design rules, and eventually physical measurements. The score can be used for evaluation and, more importantly, as post-training signal.

Our first completed instance is narrow: a seed-generated circuit-design environment where verifier-backed reward improves a Qwen2.5-Coder-7B policy on frozen held-out seeds. We started there because circuit design gives a compact closed loop, and we are now applying the same methodology to PCB layout and repair.

TL;DR

• Environment: seed-generated circuit-design tasks where the model writes a simulator-readable artifact and a deterministic tool-backed verifier returns reward. No LLM judge.
• Reward type: the signal is grounded in engineering tools and physical constraints: simulation, measured behavior, connectivity, manufacturability, and design rules.
• Headroom: on our L1 calibrated split, the best OpenAI single-shot run solves 15/30; verifier-selected BoN@4 reaches 27/30; harder L2/L3 tiers remain at 4/30 best single-shot.
• Learnability: Qwen2.5-Coder-7B moves from 12/30 after topology-SFT to 17/30 after verifier-GRPO on frozen L1 eval seeds; mean score moves from 0.519 to 0.683.
• Signal quality: 25/30 held-out seeds improve, 9 move from fail to pass, and the improved cases are behavior/sizing fixes in already valid simulated circuits.
• Integrity: deterministic reward redteam over 780 attack rows and active exploit search over 30 attempts produced 0 successful tested reward exploits.
• Progression: circuit design is the first completed instance because it gives the cleanest closed loop; PCB/KiCad layout and repair is the next artifact class; the same pattern extends toward RTL, firmware, CAD, manufacturing, and robotics workflows.

The problem

Frontier models are strong around software. They are much weaker when they have to reason over hardware artifacts: circuits, PCB layouts, RTL, constraints, connectivity, physical rules, and design intent.

People working on hardware and robotics describe a consistent failure mode. The model can sound fluent and propose something plausible, but it loses track of nets, layout intent, simulator behavior, timing assumptions, CAD semantics, or what would actually survive an engineering workflow. Around RTL, the same pattern shows up as code-like output that misses structural intent or interface semantics.

That is why hardware is interesting for RL. The tasks are hard enough to matter, but many artifacts can still be checked by real tools.

The environment loop

A benchmark evaluates a model response. An RL environment has to do more. It must define a repeatable world state, an action or artifact interface, a verifier, a scalar reward, and failure information that remains meaningful under optimization pressure.

For the first circuit-design environment, the loop is:

1. A seed generates a concrete circuit-design task instance.
2. The model receives a public specification and component/interface constraints.
3. The model returns a design artifact, in this case a simulator-readable circuit fragment.
4. The environment composes the fragment with a hidden fixture owned by the verifier.
5. A simulator-backed verifier checks validity, behavior, and constraints.
6. The environment returns a scalar score and failure buckets.

The model does not see reference solutions, oracle examples for eval seeds, verifier source, or reward internals. Train and eval seeds are disjoint, and the eval seeds are frozen before training.

One environment step

Here is the product surface, stripped of private internals. This is not a real eval seed or a reference solution. It is the shape of one interaction.

public task spec
  pins: VIN, VOUT, VDD, GND
  objective: satisfy measured behavior under a hidden fixture
  constraints: allowed component palette, interface rules

model output
  constrained design fragment

verifier-owned fixture
  source -> load -> device model -> analyses -> measurements

simulator + verifier
  protocol checks -> structural checks -> behavior checks

reward output
  scalar score
  pass/fail
  failure buckets

The important point is that the model is not rewarded for explaining the right idea. It is rewarded for producing an artifact that survives the verifier-owned engineering procedure.

What the model sees

For each circuit-design task, the model sees a public task specification: the allowed interface, the component palette, the external pins, the high-level design objective, and the quantities the artifact will be judged on. The model returns only the design fragment. It does not control the source, load, device model, simulator analyses, or measurement commands.

This is the basic threat model: the model should win by designing a circuit that behaves correctly under the verifier-owned fixture, not by pinning outputs, smuggling in a source, redefining a device model, or exploiting parser behavior.

From benchmarks to a post-training supply chain

The product is not a single circuit benchmark. The repeatable capability is an environment factory for hardware workflows.

A single family proves an environment. A repeatable factory proves a supply chain. The goal is to turn private hardware workflows into trainable, verifier-backed environments across circuit design, PCB layout, RTL, firmware, CAD, and eventually robotics/manufacturing.

For a new hardware workflow, the recipe is:

1. Define the artifact interface: circuit fragment, PCB patch, RTL module, firmware change, CAD edit, or another hardware artifact.
2. Generate seedable task families with train, held-out eval, and transfer splits.
3. Build oracle and sanity checks so the tasks are solvable, calibrated, and not broken puzzles.
4. Attach a deterministic tool-backed verifier that scores the artifact, not the explanation.
5. Run frontier baselines to find headroom and failure buckets.
6. Redteam the reward path against trivial hacks, degenerate artifacts, and verifier bypasses.
7. Package the environment with replay, throughput numbers, and training-ready reward outputs.

Circuit design is the first completed instance of this factory: seed-generated tasks, frozen eval splits, oracle solvability, frontier headroom, a small-model learnability proof, reward redteams, and throughput measurements. PCB layout and repair is the next artifact class, using KiCad-backed boards, constrained repair actions, DRC/connectivity checks, net identity, and local edit constraints.

The business claim is therefore not "we have one public benchmark." It is: we can turn hardware workflows into private verifiable RL environments that expose model weaknesses and produce training signal.

How the stack expands

We started with circuit design because it is the smallest hardware loop where reward can be behavioral, dense, and tool-backed: the model submits an artifact, the verifier owns the fixture, and simulation measures whether the artifact works under constraints.

The next step is PCB layout and repair. The artifact changes from a circuit fragment to a board edit, and the reward shifts from simulated circuit behavior to layout validity, connectivity, net identity, manufacturability, and local repair constraints. This is a natural move up the hardware stack: from behavior under a fixture to physical artifact repair.

The broader path is to keep moving upward only when the verifier remains grounded: RTL with simulation/formal/synthesis checks, embedded firmware with hardware-facing tests, CAD with geometric/manufacturing checks, and robotics/manufacturing workflows with physical or high-fidelity simulator feedback.

Why hardware rewards are different

Software became a good RL target because the artifact can often be checked: tests, compilers, linters, type systems, traces, fuzzers. Hardware has analogous tools, but they are spread across the design stack.

• Circuit design can be checked by simulation under a test fixture.
• PCB layout can be checked by DRC, connectivity, net identity, local edit constraints, and eventually manufacturing checks.
• RTL and digital systems can be checked by simulation, synthesis, formal tools, timing analysis, and testbenches.

Our long-term bet is that these checks can become training environments for hardware agents. The checks are not just pass/fail tests. They can expose near-misses, failure modes, and dense reward signals that are grounded in the artifact rather than in another model's judgement.

The biggest difference from many language or preference tasks is that hardware rewards can be anchored outside the model. A circuit either simulates under the verifier-owned fixture or it does not. A PCB edit either preserves net identity, clears DRC, and restores connectivity or it does not. Later domains can add timing, synthesis, signal integrity, thermal behavior, mechanical constraints, manufacturing checks, and physical tests.

That does not make the reward magically correct. Simulators are approximations, design rules are incomplete, and every verifier can be attacked. But it changes the nature of the supervision. The reward is not "another model liked the answer." It is a measurement of an artifact under an engineering procedure.

This is why hardware is strategically interesting for post-training. If the environment is built carefully, the model can be optimized against grounded feedback: behavior under simulation, constraint satisfaction, connectivity, physical layout validity, and eventually real-world measurements. That is the path from evals to training environments.

First completed instance: circuit design

Our first environment is a parametric circuit-design ladder. Each seed produces a related but distinct task instance. The model submits a simulator-readable circuit fragment. The verifier owns the source, load, device model, analyses, and measurement procedure. The submitted fragment is scored only through the verifier path.

This first family intentionally constrains the circuit class. That makes the result a clean test of verifier-backed sizing and behavior optimization, not a claim that we have solved open-ended analog synthesis.

At a high level, the verifier checks three layers:

1. Protocol validity: the artifact must use the allowed interface and cannot smuggle grader-owned cards or analysis directives.
2. Structural validity: the design must be a plausible member of the intended circuit family rather than a degenerate simulator trick.
3. Behavioral validity: the simulated response must satisfy the task targets under the verifier-owned fixture.

The reward is scalar, deterministic, replayable, and simulator-backed. No model grades another model. The reward is dense enough to distinguish near-misses after the hard validity checks pass. The same verifier path is used for evaluation, redteam, and GRPO.

Experimental controls

• Train seeds and eval seeds are disjoint.
• Eval seeds are frozen before training.
• The model never sees reference solutions for eval seeds.
• The model never sees verifier source or reward internals.
• The reported GRPO comparison uses the same frozen L1 eval30 seeds and the same verifier path as the SFT baseline.
• The learnability proof is currently one small-model run family, not a replicated multi-seed training study.

Frontier headroom

Before training a small model, we checked whether stronger models left headroom on the calibrated ladder. The result is useful because the first tier is not impossible, but it is not saturated by single-shot generation. We are not claiming that every common-emitter circuit task is hard; the claim is about this calibrated ladder and capstone.

The ladder has three public tiers in this memo: L1 is the calibrated tier used for the main held-out learnability result; L2 is a harder transfer tier; L3 is a capstone tier used to test whether the family is saturated.

Aggregated frontier baseline read: L1 is tractable but not saturated in one shot; test-time search helps; harder tiers remain open. These are eval numbers, not training results.

On the L1 calibrated eval30 split, the OpenAI baseline matrix looks like this. Naked is one-shot pass@1. Agentic is solved within a four-turn verifier-feedback budget. BoN@4 is verifier-selected best-of-four independent attempts.

Do not read this table as a general ranking of the models. These are API-label and prompting-mode results inside one environment harness. The relevant signal is that stronger models leave headroom, and that test-time verifier feedback or verifier-selected search can move some models substantially.

Agentic here means the model gets iterative verifier feedback within the same task: it proposes a design, receives structured information about which gates or metrics failed, and can revise the artifact for up to four turns. This is not training; the model weights do not change. It is a test-time interaction loop.

When agentic performance improves over naked pass@1, it is a useful sign that the verifier is not merely a final judge. The feedback contains information the model can use to move a failed artifact toward a passing one. That does not prove RL learnability by itself, but it supports the core environment hypothesis: the reward and failure signal are shaped enough to guide search, and therefore are plausible candidates for policy improvement once used in a training loop.

Learnability

The more important test was whether the reward could improve a model. We used Qwen2.5-Coder-7B, a topology-SFT warm start, and verifier-GRPO on L1 train seeds. We then evaluated on frozen held-out L1 seeds and a harder L2 transfer tier.

Topology-SFT is a warm start that teaches the model the valid output form and coarse circuit family before RL. That makes the GRPO comparison more meaningful: the RL step is not mainly learning how to print a parseable artifact, it is optimizing behavior under the verifier.

We show both checkpoint 75 and the final checkpoint so the result is not a single lucky endpoint. The main L1 claim is stable across both: 17/30 pass@1 and roughly 0.68 mean score.

The clean interpretation is: verifier-GRPO improves held-out L1 behavior after a topology-SFT warm start. Transfer to the harder L2 tier is preserved and slightly positive, but L2 is not solved.

Why the GRPO result is not just syntax learning

The useful signal is not only pass@1. On L1, after topology-SFT, the remaining failures were already valid simulated circuits rather than parser or syntax failures. GRPO shifted behavior and sizing misses into passes.

• L1 pass@1 moved from 12/30 to 17/30.
• L1 mean score moved from 0.519 to 0.683.
• 25/30 held-out L1 seeds improved.
• 9 held-out L1 seeds moved from fail to pass.
• Behavior/sizing failures dropped from 18 to 13.
• The lower tail improved: p10 score moved from 0.246 to 0.479.

Two representative held-out examples make the shift more concrete:

• Held-out example A: the SFT design scored 0.556 and failed, while passing all structural gates. Its main miss was a low-frequency behavior ratio. The GRPO checkpoint scored 0.836 and passed by moving that behavior ratio into the target region.
• Held-out example B: the SFT design scored 0.132 and failed, while passing all structural gates. It was badly sized: mid-band gain and low-frequency behavior were both off. The GRPO checkpoint scored 0.845 and passed by moving both measurements toward the target behavior region.

In both cases, the improvement was behavioral, not a fix to the output format.

Integrity and operability

The natural objection after any RL result is that the model may be learning to hack the reward. We therefore ran deterministic and active reward redteams against the same verifier path used for evaluation and GRPO.

The deterministic suite covered obvious classes such as fixture smuggling, output pinning, topology decoys, agent-owned model smuggling, directive smuggling, physics degeneracy, scrambled values, and sanitizer tricks. The active attacker saw only the public task spec and was explicitly asked to exploit the verifier rather than solve honestly.

This is not a proof that the reward is non-hackable. The narrower claim is that the current verifier survived the attacks we tested: obvious deterministic attacks and a lightweight active frontier-class exploit search.

Throughput and reproducibility

For an RL environment, scoring quality is not enough. The verifier must also be cheap enough to use in rollouts and reproducible enough to debug.

• Direct verifier path: p95 about 11.65 ms on 780 deterministic redteam rows.
• H100 GRPO proof: about 4.6k reward-scored completions per hour end-to-end.
• The same committed artifacts support the headline learnability, redteam, and throughput summaries.

This is enough for a small learnability proof and early pilot experiments. It is not yet a final lab-scale serving benchmark. The next packaging step is Docker-pinned cross-machine replay plus score-server concurrency.

Next artifact class: PCB layout and repair

Circuit design was the clean first wedge because it gave us a fast feedback loop. But the broader hardware-agent workflow includes physical artifacts: boards, layouts, netlists, constraints, components, firmware, manufacturing, and robot interaction with the physical world.

For PCB, the reward semantics change. We are not claiming full circuit behavior from a board image, and we do not yet have a PCB learnability result. The first controlled PCB environments are about layout, connectivity, manufacturability, and local repair.

The current PCB loop is:

1. Generate a clean KiCad-backed board with controlled motifs.
2. Inject a controlled local fault, such as a wrong via or broken layer transition.
3. Ask the model to emit a constrained repair action.
4. Apply the repair to the .kicad_pcb artifact.
5. Verify DRC, connectivity, wrong-via removal, net identity, and local edit constraints.

The important lesson from the first PCB rungs is that KiCad DRC alone is not enough. Some broken boards are DRC-clean but connectivity-broken. The environment needs both manufacturability checks and hidden intent/connectivity checks.

We are not using random public boards directly as benchmark targets. They are too noisy and often not clean enough to defend scientifically. We use them as realism priors for generated tasks: motifs, common layouts, component patterns, board sizes, routing density, and failure modes.

Scope and next proof

What we are claiming:

• A seed-generated circuit-design environment can expose frontier headroom.
• The same verifier path can produce useful post-training signal for a small model on held-out seeds.
• The current reward survived deterministic and lightweight active reward redteams.
• The methodology extends naturally toward PCB layout and repair, where the verifier checks artifact-level changes rather than explanations.

What we are not claiming yet:

• We have solved circuit design.
• L2 transfer is solved.
• The reward is impossible to hack.
• The current PCB tasks are already a calibrated training environment.
• This result proves broad hardware-agent generalization.

Bar for frontier training teams

The bar is not a huge demo. It is a clean sequence of evidence:

• Frontier baselines on frozen held-out seeds showing real headroom.
• Oracle and sanity checks showing tasks are solvable and not broken puzzles.
• A learnability proof where reward-driven training improves held-out behavior.
• Failure buckets showing what changed: syntax, topology, behavior, sizing, connectivity, or design intent.
• Reward redteams showing that the verifier is not trivially hackable.
• A second task family showing this is a task factory, not a one-off artifact.
• Throughput and reproducibility numbers so a lab can reason about training cost.

Roadmap

• Now: circuit-design learnability proof and KiCad-backed PCB layout/repair families.
• Next: a second circuit or PCB family, stronger active redteams, Docker-pinned replay, and larger frontier baselines.
• Later: RTL, embedded firmware, mixed-signal workflows, mechanical CAD, manufacturing checks, and robotics environments.