Iterion

Appearance

← Bot runs

e2e-coverage (Endy) — run bilans ​

Matrix-anchored e2e coverage completion bot (ADR-058 v2, sibling of Testy). ONE campaign agent inventories an application's features into a committed feature×coverage matrix and closes each gap with a deterministic e2e test in the repo's own harness — one test(e2e): commit per feature, the matrix row flipped in the same commit. The deterministic gate re-runs the repo's suite AND enforces the matrix contract (parse, statuses, justified exceptions, and the claims grep — an orphan claim is a red gate). Whole-app runs converge at zero uncovered rows; scoped runs converge on scope-level completion.

2026-08-06 — the audits' own backlog, closed: eight gaps → tests (443198247) ​

The three audits had left 9 honest uncovered rows. Eight are now covered by deterministic tests; each was seen red under a mutation of the feature it claims to cover, then green on revert, with no product code changed.

  • Two were security-relevant and had no test whatsoever. ITERION_DISABLE_AUTH is the kill-switch deciding whether /api/* is protected at all — welding it on (a fully unauthenticated deployment) now fails the suite; both positions are asserted as ONE contract, because pinning one alone passes for a switch welded shut. /api/me/oauth/* is the BYOK / subscription-credential path for cloud runs: the STORE is the oracle, so a payload persisted unsealed, a response echoing the token, or one account reaching another's credential each turn it red.
  • The other six: iterion server / iterion runner boot posture, iterion issue import, /api/parse|unparse|validate, and the after_create/before_run/after_run hooks firing in order — the wiring whose three Run() calls could be deleted with the whole suite staying green.
  • A parallel agent left a mutation in product code (pkg/server/effort.go, a dropped 400-guard) and reverted it a moment later; the operator caught it in git status mid-flight. Worth remembering when delegating mutation testing: the tree must be audited, not trusted.
  • A test of mine passed for the wrong reason before it passed for the right one: the OAuth routes only mount when store + sealer + auth service are all wired, so the first run answered 404 and the "anonymous callers are refused" case went green on a missing route. The assertion now demands 401 exactly.
  • Final: 309 rows / 1 uncovered — the six catalog bots that need real LLM runs, which is a budget decision, not an oversight. task check fully green (the usual flake included this pass), Playwright 24/24.

2026-08-06 — adversarial review, round 3 (converging: no new false-green, only a too-narrow domain) ​

The round-3 signal is different in kind from rounds 1-2, and it is the convergence signal: not one new false-green anywhere. Every finding is the opposite failure — the hardening had narrowed what the gate ACCEPTS.

  • Round 2 fixed a false positive and introduced two more, in the same place (2e8d7a784). Extending the test-ish-name guard to the path form rejected DESCRIPTION-style citations (charge_is_idempotent (spec/charge_spec.rb)) — the natural shape for RSpec, Jest, Cypress, pytest-BDD — even though the cited FILE already proves it is a test. And "every citation must resolve" met a naive comma split, so a parametric name (test_x[a=1, b=2]) was torn into a live citation plus a bogus fragment reported as an orphan. Net effect: the gate had narrowed to matrices whose citations look like Go test function names — this repo's shape and almost nobody else's, which is precisely the "catalog bots are repo-agnostic" bar. Both fixed, plus operator prose (N/A) no longer reading as a dead reference, and a case-insensitive test[.] no longer claiming latest.tsx / contest.py.
  • The product change of round 2 was refuted-then-confirmed by execution: clamping the auto-resume backoff to max_wait is a no-op by default (measured: ~14 min over 5 attempts, identical before and after, since the default ceiling is 8 days and the backoff caps at 5 min) and bites only when an operator explicitly set a shorter wait. The platform ceiling was verified NOT to reach this path (it binds the cloud runner instead), so nothing shortens it behind the operator's back.
  • The generated tests held again: 5 of 7 designed mutations killed, nothing critical, high or medium.
  • Third matrix sample (30 rows, 100 across three audits): zero façades. The same helper-vs-wiring miss appeared twice more (dsl.cursors, dsl.compress-field — the row cited the helper's own test while the ONE line that invokes it had no cited coverage), and two documented surfaces had no row at all: the post-mortem shell (covered, now cited) and /api/me/oauth/* — five wired endpoints, zero tests, on the BYOK/forfait credential path (now an honest uncovered row).
  • Final: 309 rows / 9 uncovered. task check green except the pre-existing TestProcessBoardCardCarriesPRLaunchContext flake; Playwright 24/24.

2026-08-06 — adversarial review, round 2 (the round-1 fixes, re-attacked) ​

The loop's own rule — what surfaces at round N+1 is usually a regression of round N — held on both surfaces that were rewritten.

  • The round-1 gate hardening had introduced a FALSE POSITIVE, and it was the worst finding of the round: the test-file regex demanded a slash on both sides of tests/ and spec/, so a root-level tests/ (Rust, pytest), spec/ (RSpec) or __tests__/ (Jest) was rejected — this gate would have refused the legitimate matrix of most non-Go repos and could never have converged there. In a gate that BLOCKS, a false positive costs as much as a hole and is harder to see. Six more false-greens went with it (a heading inside the table silently dropping every row below it; a 4-space-indented table parsed as the matrix though markdown renders it as code; a tilde fence inside a backtick fence leaking its block; the short-name guard missing on the path form; ../, absolute paths and symlinks escaping the workspace; only one citation of a list having to resolve) plus citations going unchecked entirely on non-covered rows — which is how a stale path had survived in the real matrix. All closed with 8 regression tests (83e588d5e).
  • The generated tests held: 11 of 13 designed mutations were killed, nothing critical or high. The two survivors were real and are fixed (c0d308042): the secret round-trip's "no plaintext on disk" grep missed the base64 form, so a Seal/Open pass-through — no encryption at all — passed it; and the auto-resume test detected a dropped allow-list only by exhausting a 180 s timeout, because the exponential backoff ignored the operator's max_wait (now a ceiling on any wait — same mutation fails in 0.3 s).
  • A second matrix sample (40 fresh rows, 70 across both audits) found 11 more mis-citations (9ecb92be2), same failure mode throughout: a row cites the mechanics of a helper while the wiring that invokes it goes untested. Two were status-level — dispatcher.hooks claimed all four lifecycle hooks while only before_remove is proven to fire (deleting the other three Run() calls leaves the suite green), and sandbox.kubernetes-driver sat at excluded despite 57 real unit tests. Two genuine gaps became honest rows: iterion issue import, and ITERION_DISABLE_AUTH — an authentication kill-switch with no test at all.
  • Standing number: 11 mis-citations in 70 sampled rows. The claims grep proves existence; pertinence is human. That ratio is now written at the top of the matrix.
  • Final: 306 rows / 7 uncovered — the honest state, up from a "0 uncovered" that was partly an artifact of citations nobody had re-read.

2026-08-06 — adversarial review of the whole program (4 opus agents, one per surface) ​

Not a bot run: the operator's ::rva tour over the branch, one agent per surface (gate, generated Go tests, Playwright harness, matrix), each required to prove findings by execution. Every finding below was re-verified by hand before any fix.

  • The gate did not meet its own anti-façade bar — 8 executed false-greens, all now closed (7dcf45269, one regression test each): a claim resolving to a directory (., e2e) or any non-test file; the matrix citing itself (both citation forms); a 1-char bare name grepping the whole tree; a short row (unescaped pipe) read as status="" and skipped silently, erasing the feature from the accounting; a blank line truncating the row scan and hiding every row after it; a decoy Feature+Status table before the real one; a fenced example table (this bot's own skill ships one) becoming "the matrix". Plus --var target=" " reading as scoped via target != '', waiving a whole-app run's zero-uncovered requirement — the scope predicate now lives in the gate as scoped, computed on the trimmed target. Verified no false positive: the real matrix still parses fully.
  • The matrix's own claims audit found 4 façade citations out of 30 sampled (dc7d42263): mongo-store cited the BOARD store's conformance for the RUN store's contract; dsl.node-tool ("direct shell, no LLM") cited a scenario-stub run where the stub replaces every node; triggers.subscription-registry promised "query by repo / by bot" with no test in the tree for either; bots.remaining-catalog claimed covered-live for six bots with no live test at all. Re-cited, split, and the trigger query half genuinely covered by a new store test (seen red under a dropped workspace-wide filter). The matrix now reads 293 rows / 1 uncovered — the audit's real finding, surfaced instead of averaged away.
  • A generated test was a façade on a security guard (6dd479d12): the invitation suite proved the happy path and single-use rule but never that the right person accepts — dropping the invitee-email guard left pkg/auth + pkg/server fully green, so a leaked token was a free team membership. Test added, seen red under the mutation.
  • The Playwright harness passed all five attack axes with nothing high or critical: isolation proven twice (file-fingerprint diff + inotifywait — zero writes to the operator's ~/.iterion), the browser-absent skip and three boot-failure modes all fail loud, no flakiness over 3×24 runs, and the CostPreviewChip fix verified complete against all three server response shapes.
  • Standing lesson for this bot: the claims grep proves existence, never pertinence — and the pertinence miss rate measured here was 4/30. The operator review of the matrix diff is load-bearing, and a periodic sampled audit belongs in the ritual.

2026-08-06 — V5 whole-app final: zero-uncovered convergence in the strictest mode + a self-initiated pertinence pass (run 019fd738-ccdf) ​

  • Status: VALIDATED — the first target-EMPTY run (whole-application mode: convergence requires uncovered_rows == 0 from the deterministic parse, not just the campaign's claim). Converged in one pass, 48 min, $16.8.
  • Result: 9 commits on iterion/run/astral-pulse-novazap-56da. The fresh re-scan found 5 gaps the V1 inventory had settled too generously and closed them: PAT bearer auth through the REAL middleware, /api/local/secrets (seal/rotate/delete), team invitations (issue/lookup/accept), sandbox network policy (allowlist/denylist/open composed test). Unprompted, the campaign then attacked PERTINENCE — re-citing 8 under-cited rows and repairing a row terminator — the exact weakness the V1 sample audit had flagged as the claims-grep's blind spot. Final matrix: 291 rows, all terminal — 275 covered-deterministic, 4 covered-live, 4 unit-only, 8 excluded, zero uncovered, zero orphan claims. verify.sh now includes the Playwright UI suite (the V4 harness became part of the repo's own verify).
  • Program totals (V1→V5): $119.4, ~5.5 h of runs, 5/5 converged, 31 test(e2e):/harness commits (+ matrix/doc commits), 1 real product bug found + fixed, 2 bot hardenings, 0 engine defects.
  • Lessons: whole-app mode is the right periodic audit cadence (it re-derives the inventory instead of trusting it); scoped waves are the right gap-closing cadence.
  • Status: VALIDATED — converged in ONE campaign pass (83 min, $37.6), scoped target = the 9 studio-ui rows + operator-approved harness bootstrap.
  • Result: 11 commits on iterion/run/orbital-growl-crystalbloom-2659 (26 files, 1 268 insertions, zero product-code changes): @playwright/test harness in studio/e2e/ (fixture-workspace seeding via the REST API + store, real built server on a temp store, no LLM credential), Taskfile target test:e2e:ui (skips cleanly when no browser is installed — task check stays credential-free), 9 view specs (run console, board drag-persist, launch, gallery+builder, editor parse→edit→unparse round-trip, pipelines cap, dispatcher lifecycle, secrets seal/delete, browser pane preview trigger), an order-independence hardening pass, and a docs note of the house rules. 24 Playwright tests, re-run post-merge by the operator: 24 passed in 29.8 s.
  • Real bug found by the suite: CostPreviewChip dereferenced data.nodes.length while /api/runs/preview-cost answers {"nodes": null} for a workflow with no LLM node — the whole Launch view crashed into its error boundary, so a tool+compute-only bot could not be launched from the studio at all. The campaign honoured the contract (product code untouched): it committed a deterministic KNOWN-BUG tripwire test asserting the defect, with the positive assertion ready in a comment. Fixed post-merge by the operator (fix(studio) 457374ddd) and the tripwire flipped to the positive contract.
  • Lessons: the "bootstrap the harness + cover the family" compound target works in one pass when the operator pre-arbitrates the harness decision; the KNOWN-BUG tripwire pattern (assert the defect deterministically, positive assertion in a comment) is worth folding into the e2e-coverage skill.

2026-08-06 — V3 cloud family: 3 gaps closed deterministically via existing fakes (run 019fd6ae-ca4a) ​

  • Status: VALIDATED — converged in one campaign pass (53 min, $18.5), scoped target = the 3 cloud rows, "no heavy new test deps" constraint honoured.
  • Result: 3 test(e2e): commits + 1 lint fix on iterion/run/mirage-thwack-beamspire-f903: DLQ admin surface (super-admin gate + audit trail), migrate to-cloud blob upload (walker + S3 wire, pkg/store/blob/s3_roundtrip_test.go), cross-replica Valkey state (OAuth/CSRF, board run-tokens, rate buckets). None ended excluded — the existing seams sufficed. Post-merge: e2e + server + cloud suites green.
  • Friction → bot hardening: this run's verify_build found the PREVIOUS run's verify.sh in the shared per-project scratch, pinned to a dead worktree path (it noticed and rewrote it — no harm). Hardened in the bot: the verify prompt now mandates overwrite + $PWD-relative commands (fix(e2e-coverage) 0af0da7b8).

2026-08-06 — V2 cli family: 4 gaps in one pass, first-try green gate (run 019fd687-9cc6) ​

  • Status: VALIDATED — converged in one campaign pass (35 min, $12.6), scoped target = the 4 cli rows (incl. the cli.dispatch row the V1 pertinence audit had reopened).
  • Result: 4 test(e2e): commits on iterion/run/onyx-glide-starforge-e9d1: --model/--backend node re-targeting, --auto-resume recovery loop, bench asymptote convergence curve, iterion dispatch daemon boot loop. Scoped-run convergence semantics worked exactly as designed (out-of-scope rows untouched). Post-merge e2e suite green (171 s).

2026-08-06 — V1 inventory + quick wins on iterion itself: 290-row matrix, 10 test commits, gate caught a real contract break (run 019fd613-26f8) ​

  • Status: VALIDATED — first live run of the bot, converged (gate.converged=true) after one budget-raise resume.
  • Versions: bot v0.1.0 · iterion f3de1569f (worktree branch) · unsandboxed (ITERION_SANDBOX_DEFAULT=none, matching the recent local-dogfood pattern; worktree: auto is the isolation).
  • Method: CLI run from the authoring worktree, --store-dir <main>/.iterion (studio-visible), --merge-into none, --max-cost-usd 40 --max-duration 2h, --var max_passes=3, scoped target = "inventory + reconcile the 3 partial coverage docs + close the 3-5 highest-value quick wins". 113 min wall, $33.9, 2 campaign passes + 1 resume.
  • Result: 15 commits on iterion/run/019fd613… (2 739 insertions, all additive): the 359-line docs/e2e-coverage-matrix.md (290 rows, supersession pointers added to the three partial docs), 10 test(e2e): commits (~2 100 test lines: runs questions/answer ADR-081 surface, --max-* budget overrides, secret lifecycle, --preset precedence, report, issue lifecycle, diagram, skill library, memory export/import/du, --recipe overlay), plus a matrix-row repair, a flake root-cause fix and a lint fix. Merged ff-only into the worktree branch; task check green after merge (lint 0 issues, e2e suite 197s green).
  • Value: high — the matrix is the first complete, machine-checkable inventory of iterion's feature surface (only 15 honest uncovered rows remain: cli ×3, cloud ×3, studio-ui ×9 — the studio UI has no in-tree browser harness); the 10 new tests lock down the CLI operator surface that had zero e2e.
  • The deterministic gate proved itself live: pass-1 verify_run reported matrix_ok=false on a REAL defect — an unescaped | in the sandbox.host-state row (auto|none) split the markdown row so "sandbox" landed in the Status column. The fail_log carried the exact row + reason back; pass 2 repaired it (202e2bf09). Suite-green + matrix-red is exactly the independence the two floors were designed for.
  • Anti-façade spot-checks (operator, post-merge): 2/2 mutation kills — (a) RunSecretRemove stubbed to claim success without deleting → TestSecretSetListRemoveRoundTrip FAILS; (b) --max-cost-usd override silently ignored in ir.ApplyBudgetOverrides → TestRunBudgetOverrideCapsTheRun FAILS. Both green again on revert.
  • Pertinence review (operator, 4-row sample of covered-deterministic): 2 clean, 1 defensible (dsl.worktree-field cites a pkg test that enters via .bot source text — the matrix's documented "front door per family" methodology), 1 miss: runtime.supervisor cited two brick unit tests for a composed steer claim → corrected to covered-live (the composed path is LLM-driven); cli.dispatch also narrowed (config-building covered, daemon boot loop honestly uncovered). Lesson: the claims grep catches existence, not pertinence — a sample audit of pkg-cited covered rows belongs in the operator review ritual.
  • Baseline discipline observed live: the campaign bisected a mid-run full-suite failure (TestProcessBoardCardCarriesPRLaunchContext, pkg/server) against a pre-work baseline worktree and correctly skipped it — confirmed post-run: it fails identically on the untouched main checkout (env-dependent on this host, pre-existing).
  • Engine hardening: none needed — zero engine defects surfaced.
  • Frictions / lessons for next run:
    • Duration, not cost, was the binding budget: the run hit my 2-h CLI cap at the final verify_run (90% guard) → failed_resumable → iterion resume --max-duration 3h converged in 4 min. Keep the DSL default (3 h) for V1-style runs; the banked-in-stride design made the interruption costless.
    • Pass 2 over-delivered (9 commits vs "3-5 quick wins") — high-value but the scope text should say "then STOP" explicitly if tighter runs are wanted.
    • The campaign left a detached helper worktree at /tmp/iterion-baseline (its baseline bisect); harmless but git worktree prune-worthy — a cleanup note could join the campaign contract.