The Reward Was the Bug
A controlled reward-shape ablation in PCB routing

T. Chimbault, Y. Alloul
June 2026

Result first. We study a verifier-scored PCB routing task in which the task, deterministic verifier, SFT warm start, seeds, and training loop are held fixed. Under these controls, changing only the reward shape substantially changes the RL outcome.

A flat feasibility-shaped reward improves the model's ability to produce legally routed boards, but does not produce held-out under-budget solves. In contrast, a continuous optimization reward preserves reward variation after feasibility is reached and increases held-out pass@8 from 0.50 to 0.73. As a negative control, shuffling the same reward values within each group reduces held-out pass@8 to 0.07.

The result is therefore not simply that RL improves a PCB task. The more specific claim is that, in this setting, the information content of the reward determines whether a deterministic verifier can become an effective training signal.

A verifier is necessary for this setup, but it is not sufficient. A pass/fail checker can certify whether an artifact satisfies constraints, but a binary or saturated reward may provide little useful ranking information to the optimizer. In compute-intensive RL settings, this distinction matters: a poorly shaped reward can consume the same rollout and training budget as an informative one while producing little or no model improvement.

The routing task below tests this claim. The contribution is both the empirical result on PCB routing and the broader procedure used to construct a learnable RL environment from a tool-verifiable hardware workflow. In an earlier memo we applied the same pipeline to analog circuit design; PCB layout is a second domain.

TL;DR

• Finding. Holding the task, deterministic verifier, SFT warm start, seeds, and training loop fixed, reward shape controls whether the RL run produces held-out improvement.
• Main result. The continuous optimization reward increases held-out pass@8 from 0.50 to 0.73, while the shuffled-reward control falls to 0.07.
• Interpretation. The reward values alone are not sufficient; the reward must preserve which sampled solution is better.
• Limitation. Greedy pass@1 remains low, around 0.03. This is a best-of-k learnability result, not yet a greedy-solve result.
• Claim. The central requirement is not merely having a verifier, but extracting an informative optimization signal from it.

The environment

This is BGA escape routing.

A ball-grid-array package, the dense grid of solder balls under a chip, exposes 32 nets. Each net must escape from under the chip to one of its legal exit ports. Routing runs through a small number of capacity-limited channels, across two copper layers joined by vias, without collisions and under a tight global cost budget.

We chose this task deliberately.

BGA escape routing is a compact but representative instance of PCB layout: the model must satisfy local connectivity constraints while respecting shared global resource constraints.

The task separates two properties that are often conflated: reaching the ports is feasibility, and doing so under a global cost budget is optimization. This separation between feasibility and optimization is the property the experiment uses.

Every net routes from its ball to one legal exit port; a via switches copper layers at a cost; each channel has limited capacity, so piling nets into the cheapest channel overflows it and blows the budget. Reaching the ports is feasibility; doing it under the global cost budget is optimization; the reward must therefore provide useful ranking information after basic feasibility has been reached.

The environment has an exact min-cost-max-flow oracle. But the oracle is not what makes the task easy. The model never sees it. The oracle is a measurement instrument: it gives the ground-truth optimum used to calibrate difficulty and certify that feasibility and optimality are decoupled.

What the model learns is the sequential, constrained construction of a routing under a global objective, the same skill downstream EDA tasks demand, most of which do not come with an oracle.

public task spec
  32 nets, each with a few legal exit ports
  channels with capacity, two copper layers + vias
  objective: route all nets, minimize total cost, stay under budget

model output
  constrained routing patch (insert_trace / insert_via)

verifier
  connectivity -> collisions -> layer/via legality -> total cost vs budget

reward output
  scalar score, continuous in cost once all nets legally escape
  pass = all nets escaped AND total_cost <= budget
  failure buckets

The model never sees reference solutions. It never sees oracle assignments for eval seeds. It never sees verifier internals. Train and eval seeds are disjoint. Eval seeds are frozen before training.

We train Qwen2.5-Coder-7B from a SFT warm start, then run verifier-GRPO on train seeds and evaluate on 30 frozen held-out seeds.

The feasibility trap

Group-relative RL needs reward differences inside the sampled group. A verifier can be perfectly correct and still produce no learning signal if every valid solution receives the same score.

That is what feasibility rewards do. They help the model reach validity, then go flat exactly when the model becomes competent. In this task, the policy learned to route all nets, but every over-budget routing looked identical to the optimizer.

Optimization rewards avoid that collapse. Once a solution is valid, cheaper solutions score higher than more expensive ones, so the reward keeps ranking outputs after feasibility is reached. Feasibility provides the initial learning target; cost provides the ranking signal after validity is reached.

The controlled result: the reward shape decides

We now isolate the effect of reward shape in a controlled experiment.

The task's two axes are decoupled by construction. An exact min-cost-max-flow oracle computes the optimal routing cost for every seed. Each instance is built adversarially against greedy. A per-net shortest route legally escapes all 32 nets but piles into cheap channels and overflows the budget.

A regression test certifies it: a greedy router routes all 32 nets on 30/30 held-out seeds and busts the budget on 30/30.

This makes the feasibility trap measurable. The easy axis, connectivity, is reachable. The hard axis, satisfying the global cost budget, remains combinatorial. The exact oracle certifies that the two axes are separated.

Our first reward scored any fully-routed-but-over-budget solution flat, regardless of overage. It was a feasibility reward in disguise. Every over-budget solution looked identical to the optimizer.

Holding the task, verifier, budget, and warm start fixed, and changing only the reward shape, you can watch the failure happen.

Prompt groups retaining reward variance per GRPO update (of 4 sampled). The optimization reward holds all four varying on every update (mean 4.0/4); the flat feasibility reward erodes to one or two by the tail (mean 2.6/4): less signal, and noisier. Same task, same verifier; only the reward shape changed.

The flat reward was not inert. On held-out seeds, it lifted the routing axis: samples legally routing all 32 nets rose from 57.9% to 87.9%.

Although this appears to be progress, held-out solves stayed at zero. Every routed solution was over budget, every over-budget solution scored identically, and the policy stalled at the budget frontier with no gradient to cross. The model improved on the easy axis while remaining unable to make progress on the hard one, which is the feasibility trap observed directly.

Two coupled levers fix it.

Gradient existence. Make the over-budget region a smooth, monotonic function of overage. A cheaper-but-over-budget solution must score strictly higher than a more expensive one, approaching but never reaching the pass threshold. This keeps reward variance alive.

Gradient reachability. Calibrate the budget so the gradient terminates in a reachable pass. The budget is a continuous knob, set by black-box measurement of the policy under training. No model internals. No hand-wavy difficulty. Just measure the policy and tune the environment so the slope is climbable.

This is the central reward-design requirement: the reward must remain informative in the region where the policy is already feasible but not yet optimal.

The budget is set so the SFT warm start lands in exactly this band: it routes legal boards but rarely optimal ones, so reward still varies across samples and the pass threshold is reachable rather than saturated.

Calibrated to the policy, the same environment learns.

Held-out pass@8 climbs from 0.50 to 0.73 over 40 GRPO updates. The policy finds a fully legal, under-budget solution within eight samples on roughly three-quarters of never-seen instances, up from half.

Held-out pass@8 increases from 0.50 to 0.73 over 40 GRPO updates on 30 frozen held-out seeds. Because this is a single run family and the bootstrap interval for the raw SFT-to-RL lift is wide, the main evidence is the real-reward versus shuffled-reward comparison below.

Because the single-run interval is wide, the primary evidence is not the raw lift but the reward-information control. Run the identical setup, with the same warm start, seeds, and 40 updates, but shuffle the reward within each group: the reward retains the same values while losing information about which solution is better.

Under this control, the model does not merely stall; held-out performance degrades. Held-out pass@8 falls to 0.07, below the warm start it began from.

Final held-out pass@8: the real reward versus the same run with the reward shuffled within each group.

Reward-information ablation

The main ablation compares the real optimization reward against a shuffled-reward control. The task, verifier, warm start, seeds, and training loop are held fixed; only the association between samples and reward values is destroyed in the control.

This is the central evidence for the role of reward information. The improvement is not explained by the task definition, the verifier, the warm start, the seeds, or the RL loop, since these are held fixed across the real-reward and shuffled-reward conditions. The decisive difference is whether the reward preserves the ordering between better and worse candidate solutions.

The effect is distributional rather than greedy. Greedy pass@1 increases only from 0 to approximately 0.03. The learned policy is therefore not yet reliably emitting a correct solution on its first attempt; instead, it has learned to place correct solutions within its sampled distribution.

For a cheap deterministic verifier, this is already a useful agentic regime: sample candidate solutions, verify them, and retry when necessary. The next step is to distill this best-of-k behavior into greedy generation, but the learnability signal is already visible in the controlled ablation.

Integrity: the reward is ground truth, not an opinion

This only matters if the reward is real.

Here, the reward is read off the engineering artifact: connectivity, geometry, layer legality, and cost, computed by a deterministic verifier against an exact optimum.

The reward is not produced by an LLM judge, a learned reward model, or a subjective plausibility score. It is computed from the engineering artifact through deterministic checks of connectivity, geometry, layer legality, and cost.

This makes the reward suitable for RL training only to the extent that the verifier itself is faithful to the relevant engineering constraints.

To check that the fast verifier is faithful to physical design rules and not merely self-consistent, we cross-validate it against the real KiCad design-rule checker on the cases most likely to diverge. Across 30 held-out seeds × 7 probes, including via-skip and via-in-pad anti-gaming cases, its physical verdicts matched KiCad on 210/210, with zero disagreements.

We do not call KiCad inside the RL loop; it would throttle rollout throughput. The fast verifier carries the reward. KiCad is the offline golden reference.

This is a faithfulness check on divergence-prone cases, not a blanket equivalence proof. KiCad validates feasibility only, while cost-budget optimization is our layer on top. Physically clean but over-budget probes pass KiCad and fail us. That is expected.

Cost shaping applies only after the verifier confirms all 32 nets legally escape. There is no "look plausible" gradient before the real hardware constraints are satisfied.

A committed adversarial patch suite (over-routed, under-routed, mislabeled nets, out-of-window traces, too-many-vias, disconnected stubs, via-in-pad, via-skip) scores low on every attack and passes none.

Cost monotonicity is locked by regression test: cheaper-over-budget strictly outscores dearer.

This is not a proof of non-hackability. It is a verifier that survived the adversarial cases we designed against it, and a reward whose useful signal is tied to the engineering artifact.

Evidence across two hardware domains

The PCB result also fits into a broader pipeline previously applied to analog circuit design. The two domains differ in artifact type, verifier, and optimization structure: the analog task uses SPICE-based continuous simulation, while the PCB task uses deterministic geometry and cost verification against an exact min-cost-max-flow optimum.

The comparison should be interpreted carefully. The controlled reward-shape ablation was run on the PCB task. The SPICE result is a positive replication of the broader pipeline, not a second negative-control experiment.

Nevertheless, the recurrence of held-out RL improvement across two hardware domains suggests that the method may generalize beyond a single benchmark. The relevant hypothesis is not that one PCB task is sufficient, but that some tool-verifiable hardware workflows can be converted into learnable RL environments when the reward exposes a sufficiently informative optimization signal.

Scope

This is a GRPO lift over a SFT warm start, with base solves near zero, on 30 frozen held-out seeds. It is currently one run family per setting. The shuffled-reward control and bootstrap intervals are in. Multi-seed replication is the next step to tighten the single-run magnitude interval.

We are not claiming PCB design is solved.
We are not claiming the reward is impossible to hack.
We are not claiming greedy pass@1 is where it needs to be.

The precise claim is the following: we found a held-out RL learnability signal in a verifier-scored PCB routing task, showed that reward shape causally controls whether the signal exists, and reproduced the broader verifier-to-reward pipeline in a second hardware domain.

These results support a proof-of-learning claim, not an industrial PCB-routing claim. The immediate next steps are multi-seed replication, broader task coverage, cost-gap reward shaping, best-of-k data collection, and distillation of sampled success into greedy pass@1.

Reproducibility package. The frozen eval seeds, train/eval split, verifier reports, parsed patches, raw completions, KiCad probe reports, shuffled-reward control, and training metrics are available for technical review.

Verifier correctness and reward usefulness are separate

The central lesson is that verifier correctness and reward usefulness are separate properties.

A deterministic verifier can establish whether an artifact satisfies the relevant constraints, but this does not guarantee that the associated reward will support learning. If the reward is binary, flat, or saturated over most policy samples, it may provide too little ranking information for group-relative RL, even when the verifier itself is correct.

In this task, optimization-shaped rewards convert the verifier from a pass/fail evaluator into a training signal that ranks feasible but suboptimal artifacts. The main contribution is therefore not the verifier alone, but the method for extracting a usable optimization signal from a tool-verifiable workflow.

The current evidence is limited but concrete: a held-out pass@8 improvement on PCB routing, a shuffled-reward control showing the importance of reward information, and a related positive result in analog circuit design. Further multi-seed replication and broader task coverage are needed to estimate the robustness of the effect.