iterion

Billy — branch-improvement validation

2026-07-08 — re-dogfood post-improvement (P1-P4), same PR #72 target, $1.80 (run 019f41af)

• Status: validated — reliability + integration confirmed; production excellent. Improvements are RIGHT and SAFE; this run’s shape (1 pass, no MR) did not trigger the two biggest savers (see “P1/P2 not exercised live” below — an honesty note, not a regression).
• Versions: bot branch-improve-loop @ b7ea4bd78 (P1 forge_auth_probe + P4 tool_max_steps 40→20 in db812f0dc; P2 verify_probe + P3 verify_build effort in b7ea4bd78) · iterion static binary v0.31.0+b7ea4bd78 (built CGO_ENABLED=0, invoked directly so os.Executable() bind-mounts the fresh binary into the sandbox).
• Method: apples-to-apples with the pilot below. Same target: independent clone /tmp/iterion-pr72-clone reset to the RAW PR #72 tip 4b2394b94 (the pilot’s added test removed, so Billy re-finds the gap). sandbox inline (iterion-sandbox-full:edge), worktree: auto bases on the clone’s HEAD (CWD = clone), --store-dir <main>/.iterion so the run is visible in the operator’s studio while operating on the clone (setupWorktree bases on CWD’s git root but writes the worktree under --store-dir). base_ref=main, open_mr=false, --merge-into none, --max-cost-usd 12.
• Result: converged in ONE pass, 8.5 min, $1.80 (31.8k tok). Billy reviewed main…4b2394b94 (the per-node timeout: feature) across all layers and re-found the exact same gap the pilot did — the parser/AST/IR/C199 were tested but the RUNTIME enforcement path was not — and added pkg/backend/model/executor_timeout_test.go (+115: a blockingBackend that blocks on the ctx deadline + immediateBackend happy path, asserting the bounded context cuts the node off AND that a context-deadline error is not retried). Commit f5f55d5 on storage branch iterion/run/ash-pulse-starforge-1c89. verify_run re-ran go build ./... && go test ./pkg/dsl/… ./pkg/backend/model/… → green (so the added test compiles and passes). Graph flow: campaign → verify_probe → verify_build → verify_run → gate (converged) → mr_gate (open_mr=false) → done.
• Cost story vs the pilot’s $2.26 (honest breakdown): campaign ~$1.17 (18.9k tok, was $1.31/21.2k — run variance on a slightly different pass) + verify_build ~$0.68 (12.9k tok, P3 effort now medium) + verify_run (deterministic tool, 9.8s) + no finalize_mr (open_mr=false avoided the pilot’s $0.26). Net −$0.46 (~20%), of which ~$0.26 is the avoided MR finalize and ~$0.20 is campaign variance; P3’s verify_build delta was only ~$0.01 — verify.sh authoring uses little thinking, so high→medium barely moves it here (it will matter more on repos with a heavier build-capture step).
• P1/P2 not exercised live (by design of this run’s shape) — structurally proven:
  • P2 (verify_probe skips verify_build) only fires on pass 2+; this run converged in 1 pass, so verify_probe correctly ran at iteration=0 → fresh=false → routed to verify_build (the new node integrates without breaking the happy path). The skip path rests on TestVerifyProbeLoopIterationWiring (proves the loop iteration reaches the tool via the campaign→verify_probe edge with-mapping into node input — does NOT resolve inside a tool command) + the python-logic cases (fresh=true only when iteration>0 AND verify.sh valid) + iterion validate.
  • P1 (forge_auth_probe short-circuits finalize_mr when no push credential) only fires when open_mr=true; this run used open_mr=false (mr_gate → done), so the probe wasn’t reached. Its logic is deterministic + unit-covered.
• Reliability: HIGH. Same target → same high-value conclusion (add the runtime enforcement test), reproduced independently. The two new deterministic nodes (verify_probe, forge_auth_probe) slotted into the graph with zero runtime surprise.
• Engine hardening: none needed. The e2e scenario sweeps gained stubs for the two new nodes (aad7bddce, test(e2e): stub verify_probe + forge_auth_probe) so go test ./e2e/ stays green.
• Lessons for next run: to measure P2’s $0.69/pass saving live, point Billy at a ≥2-pass target (a branch with 2+ real issues, or a synthetic branch with a planted build-affecting change so pass 2 re-runs). To measure P1’s skip live, run open_mr=true with no forge_token and no host gh auth → forge_auth_probe → done. For a pure production/cost demonstration this 1-pass/no-MR run is the right shape and $1.80/8.5min is a good point; the structural savers show on longer/MR runs.

2026-07-08 — pilot on a real contributor PR (#72), converged in 1 pass (run 019f415c)

• Status: validated — high-quality single-commit improvement.
• Versions: bot branch-improve-loop (feat/agent-node-timeout tree) · iterion 1f5082f (static, F7 skills fix in the engine)
• Method: local run, sandbox: auto (iterion-sandbox-full:edge, devbox toolchain so the build/test gate has Go — the cloud runner does NOT, cf. F6). Independent clone checked out on the PR branch feat/agent-node-timeout so worktree: auto bases on the PR (NOT main — a linked git worktree bases on the shared repo HEAD, which is main; an independent clone was required). base_ref=main, max_passes=2, open_mr=true, mr_branch=iterion/billy/pr-72-pilot.
• Result: converged in ONE pass. Billy reviewed main...feat/agent-node-timeout (the per-node timeout: feature), found the parser + C199 validation were tested but the RUNTIME enforcement path (executeBackend deriving the bounded context — the feature’s behavioral heart) was not, and added pkg/backend/model/executor_timeout_test.go (+108: TestNodeTimeout_Enforced with a ctx-blocking backend + 20ms timeout, plus a happy-path immediate backend). The deterministic build/test gate (verify_run) passed green. Exactly the test a good human reviewer would add. Pushed by hand to iterion/billy/pr-72-pilot → draft PR #82 (base = the PR branch, so the diff is just Billy’s addition).
• Cost: campaign $1.31 (21.2k tok) · verify_build $0.69 (13.1k tok) · verify_run (deterministic tool, 9.7s) · finalize_mr $0.26 (6.8k tok) = ~$2.26, ~9 min.
• F7 (engine) re-validated here too: finalize_mr successfully Launching skill: forge-mr-create — the directory-form skills fix works on the local sandboxed path, not just cloud.
• Efficiency/reliability misses to fix (operator asked to bring Billy to a Willy-grade production/cost ratio):
  • finalize_mr burns budget discovering there are no credentials: with no forge_token mounted and no gh auth, the agent ran 4 probe commands before abandoning the push. A deterministic pre-check should skip finalize_mr (or short-circuit it) when no push credential is present.
  • verify_build ($0.69) re-authors the build/test script each run — candidate for caching / lower effort.
• Lessons for next run: base the worktree on the PR via an INDEPENDENT clone (a linked worktree bases on main). Provide a forge_token (or run in a gh-authed shell) if you want Billy to self-push; otherwise recover the commit from the iterion/run/<name> storage branch the engine always creates and push by hand.

2026-06-14 — re-validation on a clean clone + good dead-code judgment (run 019ec5bc)

• Status: validated. Re-ran in the C082 worktree studio (non-watchexec) on a clean iterion clone with a synthetic billy-test branch: one added file pkg/log/billy_demo.go exporting WriteMarker, which had no godoc AND swallowed the os.Create error. base_ref=main, merge_into=none.
• Result: cross-family review loop converged (review → fix → re-review → streak), Run finished, committed to storage branch iterion/run/feral-crash-duskvane-127c (final_commit 9c5a5891), not merged (merge_into=none respected).
• Value — correct judgment, not a cosmetic fix. Rather than just adding godoc
  • handling the error, Billy recognized WriteMarker as unreferenced dead code that swallows an error and removed it (refactor(log): remove unused WriteMarker demo helper, “No code in the tree calls it, so remove the dead code.”). That’s the right call — exactly what a demanding reviewer should do.
• Finding (minor, non-fatal): fix_claude (claude_code) emitted one Tool error: StructuredOutput — No such tool available: StructuredOutput before recovering and producing its output normally. The agent appears to try a StructuredOutput tool that isn’t registered in the claude_code delegate — a wasted step, same broad family as the Devy claude_code-structured-output gap. Worth wiring/​silencing, but it did NOT block convergence.
• Convergence machinery (shared with Willy) is reference-correct; this run re-confirms it + the asymptote (no oscillation) on a fresh target.

---

Status: validated end-to-end (2026-06). Scope of this report: the capability and the engineering hardening it drove. The target here is iterion’s own repository (the feat/cloud-control-plane epic), so target details are included.

Summary

Billy (the branch-improve-loop bot) was exercised against a real, large branch — a ~7000-line / 42-chunk epic — and demonstrably:

1. converges a cross-family review/fix loop on a big diff (monotonic decrease to zero blockers + a two-family approval streak), rather than oscillating or stalling;
2. finds real, high-value issues the human author missed, and authors complete fixes for them — including an ADR for a design-level change;
3. drives the full pipeline plan_chunks → alternating cross-family review (Claude ↔ GPT) → same-family fix → streak_check → prepare_commit → semantic commit_changes, and stops at the asymptote.

The exercise also hardened the engine: one significant runtime bug was root-caused and fixed while driving real runs — without it the GPT family could not review a diff this size at all.

What was validated

• The loop: plan_chunks (deterministic diff measure + chunking) → round_robin reviewer (claude-opus-4-8 ↔ openai/gpt-5.5) → family-matched fixer → streak_check (2 consecutive opposite-family approvals) → prepare_commit → commit_changes.
• Chunked review at scale: a 42-chunk diff, each reviewer reading chunks one at a time then merging into one whole-diff verdict (cross-family approval is on the whole diff, never chunk-by-chunk).
• Convergence to an asymptote, not oscillation — the bot settled into a stable approved state and committed.
• Both backends working through the whole run, including the GPT family on the local ChatGPT-forfait path.

Method

A single end-to-end run (019eb168) on feat/cloud-control-plane, ~7000 lines of diff. Reviewers/fixers: claude-opus-4-8 (Claude Code) and openai/gpt-5.5 (claw, ChatGPT-forfait). Sandboxed (per-run container), --merge-into none so the result lands on a storage branch for review. Budget raised to 8h/250$ for this run (the 2h/60$ default is too small for a diff this size); convergence took ~2h of effective run time.

Result

Converged: status finished, commit 1ffc4bc — fix(secrets,memory): enforce bot-secret binding egress and isolate bot memory by id, 745 insertions / 84 deletions across 30 files, build + tests green. Fast-forward-merged into the epic, then merged to main.

Convergence trajectory (blockers per verdict):

claude 2 → gpt 2 → claude 1 → gpt 1 → claude 1 → gpt 1
→ claude 0 (approved) → cross-family streak → commit

A slight oscillation near the end (GPT re-raised one blocker before settling) is within the accepted asymptote behaviour; the prior_pushback / previous_scanned_areas feedback kept verdicts from re-litigating resolved items in a loop.

What Billy found and fixed

Genuine issues in the reviewed epic, fixed with tests:

• cloudpublisher did not persist RepoURL/RepoSHA/BotID, breaking cloud/webhook resume and bot-bound secret resolution. Fixed across the publisher, queue/types, and the run store.
• secretguard did not intersect a bot-secret binding’s egress hosts. Fixed, with ADR 018.
• Bot memory was not isolated by bot id. Fixed (fsstore, scope).
• Binding-route validation tightened; new tests added.

Engineering hardening (the enabler)

Before the fix, gpt-5.5 on the ChatGPT forfait died on the 42-chunk review with context_length_exceeded — not a fundamental limit but a bug: the forfait’s effective context window is smaller than the model’s advertised 1.05M, so claw’s preemptive compaction (sized to the advertised window) never triggered in time, and nothing reacted to the backend’s rejection.

Fix: reactive force-compaction — on a context-window rejection the tool loop force-compacts the running history to a shrinking target (256k→128k→64k→32k, independent of the advertised window) and retries. Surfaced as an llm_retry, reusing claw’s existing pure compactor. With it, Billy ran for hours with both GPT nodes (reviewer and fixer) and zero context-overflow deaths.

Operational resilience

The run absorbed several transient infrastructure interruptions — network drop, ChatGPT-forfait cap, and an intermittent sandbox bootstrap flake — via delegate-level network retries and an auto-resume loop that relaunches from the checkpoint (no progress lost) until convergence. None were the context-overflow bug; all were absorbed without operator intervention.

This site is open source. Improve this page.