Reliability audit 2026-08-31 — traceability matrix
Source: full-repo reliability inventory (docs + dogfood bilans + code) verified item-by-item against main @5278ac766 by an adversarial fan-out (7 agents), cross-checked by a cross-family plan review (codex gpt-5.6-sol, 23 findings). Each row has exactly one disposition; confirmed/partial rows map to a board card (labels epic:chantiers-2026h2, chantier:c1, source:reliability-audit-2026-08-31). The in-session work is the Silent-failure pack card; refuted rows are kept so the claim is not re-raised without new evidence.
| ID | Verdict | Sev | What is actually true today | Evidence | Card |
|---|---|---|---|---|---|
| A1 | confirmed | low | Run-console/shell/browser WS get a TCP reset at exit, not a close frame; only the /api/ws hub watches shutdown. Documented as a known limitation, cosmetic (clients reconnect). | docs/probes-and-graceful-shutdown.md:141-147; pkg/server/runs_ws.go (no shutdown/drain hook; only hub.go has one) | native:de18a16c |
| A2 | confirmed | med | NATS/S3/Valkey failure returns 200 degraded by design (avoid fleet-wide eviction on shared-backend blip); only panics reach errtrack — plain ping failures rely on external body-scraping alerting nobody ships. Deliberate tradeoff, but the alerting half is a real gap. | cmd/iterion/server.go:583 (only mongo Critical:true); docs/probes-and-graceful-shutdown.md:37-55; docs/cloud-troubleshooting.md:42,90-92 | native:f92bbd1e |
| A4 | confirmed | low | iterion_workspace_clone_duration_seconds is registered and asserted in a test but never Observed by any production code path — a dead metric that reads as 'no clones ever'. | pkg/cloud/metrics/metrics.go:108-113,178; only Observe call is metrics_test.go:23 | native:f92bbd1e |
| A5 | confirmed | med | One failed Refresh (any non-Canceled error) immediately runCancel(ErrRunInterrupted). With TTL 60s / interval 20s there is room for 1-2 retries before lease expiry. Mitigated: run ends failed_resumable and auto-resumes via redelivery, so cost is a wasted interruption, not a lost run. | pkg/runner/loop_lock.go:71-80; pkg/queue/nats/lock.go:85-96 (Refresh has no internal retry) | native:22d5ba09 |
| A6 | already_fixed | low | True historically (ModelOverrides entered v7 without bump, commit 427a9f44e) but the window is closed by time: no pre-v10 runner is accepted today, the v8 bump shipped, and the additive-intent rule is codified in both doc and code comment. Nothing actionable remains. | docs/cloud-queue-schema-rollout.md:26-29; pkg/queue/types.go:90-93 KNOWN DEBT comment; SchemaVersion now 11, MinSchemaVersion 10 | — |
| A7 | already_fixed | low | Bare-Nak/stuck-queued only exists in pre-#481 (v7-or-older) builds; #481 shipped delayed Nak + DLQ park + failed_resumable flip with v8, and the fleet is at v11. The runbook forbids Path B for the (long-past) v7→v8 cutover precisely to close this. Not a live defect; 'high' overstates today's exposure. | docs/cloud-queue-schema-rollout.md:47-67,111-117; pkg/queue/types.go:94 (v11, min 10) | — |
| A8 | confirmed | med | Reverse-direction (old-version) parked messages replay to the exact bytes the new fleet rejects — an operator replay loop. Recovery is manual (resume + DLQ delete), though bounded: runs sit failed_resumable with an actionable error, never lost. Automation gap, not a loss path. | docs/cloud-queue-schema-rollout.md:140-149 | native:16e6c5dc |
| A9 | confirmed | med | Plain helm upgrade rolls server+runner together; server-first ordering is a manual two-phase --set runner.image dance with no chart-side sequencing. Post-#481 mechanics make a wrong ordering recoverable (park + flip) rather than lossy, which caps the severity. | charts/iterion/templates/runner-deployment.yaml:59-62 (runner defaults to shared iterion.image); docs/cloud-queue-schema-rollout.md:70-85 | native:16e6c5dc |
| A10 | refuted | low | Confirming the cross-family review: the unchecked box is an operator-runtime action ('roll out per Path A') for the v7→v8 cutover, which is long past — the queue is at v11. Stale checklist prose at worst, not a live gap. | docs/cloud-queue-schema-rollout.md:165-168 vs pkg/queue/types.go:94 (SchemaVersion=11, MinSchemaVersion=10) | — |
| A11 | confirmed | med | The cap's telemetry source is claude_code's rate_limit_event exclusively; claw/pi (and codex) runs against the same subscription are not capped. Doc states it plainly ('claude_code-shaped today'). Real coverage hole, partially bounded by --max-cost-usd on metered paths. | docs/usage-caps.md:148-151; rate_limit_event parsing only in pkg/backend/delegate/claude_code_stream.go | native:ac44fe6e |
| A12 | confirmed | low | Accurate: launch refusal only when AlwaysReachesLLM; a model-free-path run starts (pod + clone in the worst case) and is stopped by the mid-run guard at the first call. Deliberate and argued (the Vigie collect-half lesson: refusing zero-LLM gather work loses unrecoverable material). | docs/usage-caps.md:166-176,195-208 | native:ac44fe6e |
| A13 | confirmed | low | Reconnecting the SAME Anthropic subscription opens a fresh meter — credentials.json carries no account_id (Codex's auth.json does). Fails open by one run; mid-run guard is the backstop. Provider-data limitation, documented. | docs/usage-caps.md:249-255 | native:ac44fe6e |
| A14 | confirmed | low | Pre-flight fails open on every uncertainty (no/unreadable ledger, nothing measured, rolled-over reading) — deliberate: the cap protects a subscription, not the other way round, and the in-run guard bounds the cost to one call. Working as designed. | docs/usage-caps.md:261-264 | native:ac44fe6e |
| A15 | confirmed | low | Local CLI/studio metering is a process-local ledger keyed per machine; rotating a credential inside a long-lived studio process keeps reading the replaced account's window until reset or process restart. Documented boundary. | docs/usage-caps.md:256-259 | native:ac44fe6e |
| A16 | confirmed | med | Real doc contradiction. KEDA scale-to-0 SIGTERMs → drain → ErrRunInterrupted → failed_resumable + auto-resume; even a SIGKILL leaves the run to lease-expiry redelivery, not 'failed'. usage-caps.md:285 is wrong today (its cancel-first advice is merely conservative); one-line doc fix. | docs/usage-caps.md:285-287 ('a pod killed mid-run ends failed') vs pkg/runner/loop.go:343-347,703-704 (ErrRunInterrupted → failed_resumable, auto-resume) and docs/cloud-architecture.md:65-67 | native:f92bbd1e |
| A17 | confirmed | low | Doc still says present-tense 'the lame-duck drain is not deployed'; chart defaults drainMode: complete and renders ITERION_RUNNER_DRAIN_MODE. The sibling code comment was already corrected to 'whenever the drain is not active — or a run outlives its drain window'; the doc sentence was not. Pure doc staleness (audit's 2026-08-21 date is wrong — feature landed 2026-07-23). | docs/merge-gate.md:374-375 vs charts/iterion/values.yaml:239 (drainMode: complete, shipped 3ec777fa7 2026-07-23) and pkg/server/forge_gate_reconcile.go:22-24 | native:f92bbd1e |
| A18 | refuted | low | A run dying on its OWN in-flight claim is exactly what the reconciler handles: isGateInFlight + gateStatusSpeaksFor(own runURL) falls through and posts the synthetic failure, driven by both the outcome event and the 1-min sweep. The cited doc sentence is the setup for the very next section that closes it. Residual holes are narrow and Warn-logged (grant expired past TTL, forge statuses unreadable, run dead >60min with reconciler down). | pkg/server/forge_gate_reconcile.go:353-369,405-417 + forge_gate_sweeper.go:33-70 | — |
| A19 | refuted | low | Without PublicURL the launch indeed does not claim — but it Warn-logs that refusal, and the REPAIR is not off: the reconciler still posts the synthetic failure on an unanswered head (empty TargetURL). Only second-death escalation stands down, also via a Warn ('set PublicURL to enable second-death escalation'). Neither silent nor a disabled repair. | pkg/server/forge_gate_pending.go:79-85; forge_gate_reconcile.go:345-348,384-387 | — |
| A20 | confirmed | med | A grant-holding dead run with no gate_context in its inputs exits with zero log — deliberately, since 'has a grant' ≠ 'gates' (brancher/amender also get grants). Consequence: a gating repo that forgot to pin launch_vars.gate_context gets no repair AND no trace, contradicting the doc's own 'when a repair declines to act, it says so' (merge-gate.md:469-472). Documented at merge-gate.md:395 but undetectable at runtime. | pkg/server/forge_gate_reconcile.go:258-261 (bare return nil, not routed through abstain()) | native:f92bbd1e |
| A21 | confirmed | med | Coverage is uneven: usernotify (2-min sweep), gate reconciler (1-min sweep), board sources (poll/CAS-cursor) and retries (sweeper) have nets; the gate AUTOFIX lane and trigger-evaluator run-completion subscriptions (direct bot chaining) ride the lossy bus with no reconciliation — a dropped outcome event silently skips the fix pass or breaks a bot chain. Concurs with the prior refinement on C2: InProcBus drops ARE Warn-logged and Drops() exported — the gap is consumer nets, not bus telemetry. | pkg/server/forge_gate_sweeper.go:117-121 (sweep re-offers ONLY reconcileGateForRunID); forge_gate_autofix.go:78-82 (event-only, no sweep); trigger_coordinator.go:73 / trigger_cloud.go:220 | native:f92bbd1e |
| A22 | confirmed | med | Under default overlap:skip a crashed-host run stuck in running blocks its schedule indefinitely; schedgate applies staleness only to keepalive. Deliberate ('iterion does not guess a cutoff') and loud (tick audit names blocking_run_id), and the server's orphan reconciliation flips flock-releasable orphans — but a pure crontab host with no resident studio/server has nothing automatic. Real, mitigated, documented. | pkg/schedgate/live_runs.go:41-45,70 (staleAfter<=0 on non-keepalive → no run ever stale); docs/scheduling.md:152-157 | native:bad52449 |
| A23 | confirmed | med | RepoIntegrationID is resolved and stored at CRUD time but used only for list/delete-by-integration (mongo.go:53,109) — the tick launch never resolves the forge connection, so the clone credential falls back to forge_token secret resolution. An expired hand-set forge_token kills every tick at clone while studio/API launches (which mint managed tokens) keep working. Documented with the managed-secret workaround (scheduling.md:368-378), which caps severity, but the half-wired seam is real. | pkg/server/webhooks_gitlab.go:807-833 buildScheduledLaunchSpec (threads RepoURL/RepoRef, ignores RepoIntegrationID; no connection token minted); pkg/cloudsched/schedule.go:27; schedules_routes.go:188 | native:f92bbd1e |
| A24 | confirmed | low | Accurate and explicitly deliberate: usage-window retry excludes budget caps (retrying the same cap re-fails instantly; needs a human to raise + resume) and auth failures (time does not fix a credential). A gap only in the sense that no automated lane exists for those two failure classes — which is the phase-2 taxonomy question, not a hidden defect. | docs/scheduling.md:323-326 | native:393973b3 |
| A25 | confirmed | low | True: runs persist forever unless the operator wires iterion runs prune on a separate crontab line (the managed schedule block only runs bots). Documented, with a copy-paste remedy; failed_resumable excluded by default. A convenience gap, not a silent failure. | docs/scheduling.md:328-341 ('the store has no built-in retention') | native:bad52449 |
| A26 | confirmed | low | Accurate: only API requests and in-process LLM calls get transactions; engine/store/dispatcher/queue are not traced, by documented design (events.jsonl owns the run-level story). Legitimate residual: 'where did the time go' across queue-wait/dispatcher/store latency has no tracing answer cloud-side. Deliberate scope, worth at most a deferred card. | docs/observability.md:221-233 ('That is the whole list, and deliberately so: A run gets no transaction') | native:de18a16c |
| A28 | confirmed | med | Code matches doc: S3 is non-critical for readiness (deliberate per probes doctrine — evicting the fleet on a shared-backend blip is worse), so readyz answers 200 status:degraded while artifact IO fails. The real gap is that nothing pages on 'degraded' — visibility depends on scraping the response body or logs. Pairs with the phase-1 cloud operator-alerting item; do NOT 'fix' by making S3 critical. | docs/cloud-troubleshooting.md:90-92; pkg/server/health.go:106-107,258-283 (non-critical check → 200 'degraded') | native:f92bbd1e |
| A29 | refuted | low | Recovery is automatic: the lease TTL (~60s, 20s refresh) self-releases, JetStream redelivers, and the server's orphan queue-sweeper flips lease-less stale queued/running rows to failed_resumable every 60s (false positives self-heal via redelivery reconciliation). The troubleshooting doc's nats kv del is a force-early-release shortcut, not the mechanism — at worst that doc line under-sells the automation. | pkg/server/queue_sweeper.go:16-31,138-151 (auto CAS to failed_resumable when updated_at stale + no NATS KV lease); lease TTL 60s self-expires | — |
| A30 | already_fixed | low | K8s sandbox orphans (pod + plaintext-credential Secrets + NetworkPolicy) are GC'd automatically: self-terminating manifests + the runner's periodic label reaper, no Cleanup dependency; docker driver has ReapOrphanContainers at boot. The troubleshooting doc's 'restart the runner pod' is a residual manual fallback (and stale for the k8s case) — a doc touch-up at most. | docs/adr/070-kubernetes-sandbox-orphan-gc.md (accepted 2026-07-12); pkg/runner/reaper.go:32,68 (boot + interval label reaper, sibling runners reap dead runners' orphans); pkg/sandbox/iface.go:345 (activeDeadlineSeconds) | — |
| B1 | confirmed | med | Worse than claimed: cloud GET /api/runs/{id}/artifacts is empty for EVERY cloud run, not just parked ones — artifacts ARE persisted live (mongo/S3, pkg/store/mongo/artifacts.go WriteArtifact) but the list endpoint never reads that store. Checkpoint workaround (feed-watch.md:38-42) is the only cloud path. | pkg/runview/service_artifacts.go:70-84 — ListAllArtifacts walks <storeDir>/runs/<id>/artifacts on disk; comment says 'filesystem store only; cloud mode returns an empty list' | native:bcf9ae5d |
| B2 | partially_addressed | med | Incident fixed same-day operationally (dedicated CODEX_HOME session, 'one session, one record, one refresher'), but only in the bilan — the runbook doesn't carry the rule, and the engine has no credential-health probe/alert: a platform OAuth record dead from dual-client refresh rotation still surfaces only as a run-level 401. | docs/bot-runs/feed-watch.md:43-54; docs/cloud-llm-credentials.md (no dedicated-session rule documented; refresh worker mentioned line 253-255) | native:fc0c51d4 |
| B3 | confirmed | med | Team-tier-over-platform precedence is by design, but no operator-facing surface (run API/studio/CLI) names the tier/record that served a run; diagnosing the stale team codex record required log access and manual tier archaeology. Info logs are the only mitigation. | pkg/server/cloudpublisher/publisher.go:490 + 713 — serving tier is named only in server Info logs; feed-watch.md:47-49 + lesson line 82-83 | native:fc0c51d4 |
| B4 | confirmed | high | Claim accurate as stated: 'finished != delivered' is closed for feed-watch only (silence_alert on digests.jsonl). Engine-side there is still no generic signal distinguishing a zero-LLM/zero-output green exit from a healthy run; any other bot with an early-exit path inherits the 5-day-silent failure mode. | docs/bot-runs/feed-watch.md:108-129 (bot 1.3.0 silence_alert); pkg/alert/alert.go:1-32 — alert kinds are stall/budget/failure only, no zero-work-green-exit condition | native:e5b5084e |
| B5 | partially_addressed | low | The staleness hypothesis was never tested (the pod-side fetch was refused) and the 08-19 update leans against any cluster cause — collect worked again after re-seed with zero infra change. Outage resolved, no recurrence through 08-31, root cause of the 13-18/08 blindness never closed. Mostly a cold case, not an active defect. | docs/bot-runs/feed-watch.md:197-210 (hypothesis + never-run pod wget) vs :92-104 (08-19: collect resumed unaided, 'No hidden cluster cause') | native:de18a16c |
| B6 | confirmed | high | Real: a single mid-pass campaign node spanning the cap is killed at the per-node duration deadline (9001s/9000s) — exit grace never applies to an in-flight node, loop guard can't shorten a FIRST pass, and nothing scopes the ~10-min verify gate. Zero commits banked, exactly the shape commit-in-stride exists to avoid. No fix since 08-30. | docs/bot-runs/branch-improve-loop.md:17,26-34; pkg/runtime/budget.go:327-329 (per-node hard deadline at plain RemainingDuration) + budget_exit_grace.go (grace only for nodes STARTING on a spent budget) | native:e33385bc |
| B7 | confirmed | med | commits_this_pass exists only in the campaign bots' termination contract; no engine event, run-view field, or zero-commit-past-N-minutes warning surfaces it. Operator reads 'running' for 2h31 with nothing banked and no basis to cancel. | docs/bot-runs/branch-improve-loop.md:35-39; grep commits_this_pass in pkg/ + studio/src returns nothing outside bot contracts | native:fc0c51d4 |
| B8 | partially_addressed | high | Core defect confirmed at HEAD (probe answers 'available' for a dead token; a >1h run's commits die with the pod). But the audit's sub-claim is REFUTED: the card exists on the cloud board (it's just not on the local workspace board, which iterion issue show reads). Bilan also corrected the 1h-token hypothesis — run 5 pushed at 1h53; standing defect is the probe. | bots/branch-improve-loop/main.bot:1197-1216 — push_auth_probe checks file size + env presence + gh auth status only, no live validity call; card EXISTS: remote board native:54412d84-f5e0-42de-8896-406a4f729002 'Forge push token expires during long fixer runs — mint at use, and make push_auth_probe validate', state=inbox | native:54412d84 |
| B9 | already_fixed | low | The wedge is closed, not partial: dead-session degrade lands in FallbackUnclassified (exactly where the lived error_during_execution loop classified), retries ONCE fresh, loudly, with claw node-session eviction; narrowing to unclassified is deliberate design (R1486ff), pinned across all six session modes by tests (session_optional_degrade_test.go, session_continuity_modes_test.go), hardened by run 5's 11 commits on #544. | pkg/backend/model/executor_build_task.go:1305-1320 (inherit_if_available + persist set SessionOptional) + executor_retry.go:886-937 (one loud fresh-session retry on unclassified failure, session-store eviction, _session_degraded stamp + OnSessionDegraded event) | — |
| B10 | partially_addressed | med | Mitigation is now FLEET-wide (skip default), stronger than the claim's 'per-bot': a dead second-family credential costs one skipped peer attempt, no longer a blocked fixer. Root shape stands: family resolution counts a 401-dead record as a credentialed family, so auto still arms the peer off a corpse (wasted attempt + iterion-integration launch_vars pin plan_review=off as belt-and-braces). | pkg/reviewtopology/resolve.go:199-208 — auto → on iff two families 'backed by a credential' (existence, never probed for validity); f803f5451 'plan_review_policy defaults to skip fleet-wide (#548)'; bots/branch-improve-loop/main.bot:231 default skip | native:8885ab04 |
| B11 | partially_addressed | med | The sandbox-boundary half is fixed (host codex version / ITERION_CODEX_VERSION now crosses into the claw runner). Still open: a cloud runner pod with no codex binary and no env pin falls back to claw's baked-in version, and there is no launch-time pre-flight — the mismatch still surfaces as a mid-run 400; the iterion integration still works around it by pinning plan_review off. | docs/bot-runs/branch-improve-loop.md:70-74; pkg/backend/model/claw_backend.go:1082-1094,1128-1139 + commits 82f98df52/a2ac17a04 (host-probed codex version forwarded across the sandbox IPC) | native:8885ab04 |
| B12 | partially_addressed | med | The last-hop-visibility gap that cost 7 days is substantially closed: the escalation is posted on the PR itself (where the human already is), cross-replica deduped. The specifically-claimed org-scoped escalation NOTIFICATION (push/board-independent) remains an open follow-up card, so a repo nobody watches can still go dark. | docs/bot-runs/dep-update-guard.md:26-27,48-52 (7-day-unseen card; escalation now ALSO a PR comment, both dead runs named, UUIDv5 dedup — PR #508) + :69 ('org-scoped escalation notification' filed as follow-up card); pkg/server/forge_gate_relaunch.go:385 commentDeadGateOnPR | native:fc0c51d4 |
| B13 | partially_addressed | med | Incident real and fixed for that deploy (3.53 chart bump); chart now renders drainMode. But no recurrence guard: helm-lint and the kind e2e catch broken renders, not a values key a stale vendored chart silently ignores — the silently-inert-value class is unguarded. | docs/bot-runs/dep-update-guard.md:22-29; charts/iterion/templates/configmap.yaml:55; .github/workflows/tests.yml:526-540 (helm-lint) + cloud-e2e job | native:16e6c5dc |
| B16 | confirmed | med | Sandbox 2-5x slowdown + go per-binary 10-min default timeout mitigation lives solely in Vetty's verify-build skill; the 10 sibling verify-build.md copies and the engine have nothing. | docs/bot-runs/dep-update-guard.md:61-65; bots/dep-update-guard/skills/verify-build.md:125-135 (only copy with the guidance — grep '2-5' hits 1/11 verify-build.md files) | native:031ed395 |
| B19 | confirmed | high | Cost accrues only when a node reports; no streaming/in-flight budget check exists. Loop guard + exit grace bound iteration COUNT and post-hoc delivery, but a single v2 campaign node can still overshoot the cap unbounded ($111 over $23 headroom, later 3x cap on pi). | pkg/runtime/budget.go:653-671 (RecordUsage fed only from node OUTPUT); docs/bot-runs/docs-refresh.md:49-54 and 228-235; pkg/runtime/loop_budget.go (guard is between-pass only) | native:e33385bc |
| B20 | confirmed | med | max_cost_usd enforces imputed token value uniformly; a forfait run billing $0 real was killed at $231 imputed while the provider window already bounded it. Neither the estimate-semantics distinction nor the bilan's requested doc clarification landed. | docs/bot-runs/docs-refresh.md:43-48; pkg/runtime/budget.go + pkg/backend/cost/cost.go (no metered-vs-estimate flag); docs/dsl.md:610-640 covers unpriced only | native:e33385bc |
| B21 | confirmed | med | At HEAD the dispatcher still treats operator-cancelled as resumable and auto-resumes it — friction 7 unfixed in main; matches the phase-1 in-session fix plan (= C5). | pkg/dispatcher/retry.go:201-207 (switch includes store.RunStatusCancelled → resumable); docs/bot-runs/issue-triage.md:51 | native:f92bbd1e |
| B22 | already_fixed | low | setupWorktree now anchors on the checkout the run was launched from (studio --dir / dispatcher workspace), not the primary checkout's HEAD; finalize FFs the anchor's branch. Red-first test in the fixing commit. | commit ca728277c (2026-08-05), pkg/runtime/worktree.go:110-151 (anchorDir = FindRepoRoot(repoHint), worktree added at the anchor's COMMIT); pkg/dispatcher/engine_runner.go:380,445 (WorkDir = spec.WorkspacePath) | — |
| B23 | partially_addressed | med | Board half fixed (sandboxed board.create works via HTTP transport; residual docker-unreachable gap native:e6cd506e). The findings.md half stands: still written inside the worktree that finalize removes on a read-only run — durable home remains only the run artifact. | docs/bot-runs/sec-audit-source.md:98 and :80 (latest bilan re-confirms the gap); C082 board-emit fixed+validated 2026-06-14 (same file); bots/sec-audit-source/main.bot:1325 (findings.md still at workspace_dir/.sec-audit inside the worktree) | native:3f3264c1 |
| B28 | already_fixed | low | The .codex bridge into the sandbox exists (forfait sealed per-run, file-delivered, OAuth forced when no BYOK key), and the platform LLM tier makes preprod cred provisioning a one-call ops action. Remaining is an ops decision, not a code gap. | pkg/backend/model/claw_backend.go:1159-1172 (CODEX_HOME → CodexSandboxConfigDir, delivered by runtime.addCodexOAuthSecretFile — commit a4926ab04 #360); pkg/backend/model/registry.go:449-456 (openAIFromCtxForfait reads Credentials.OAuthDir(codex)); platform credential tier #466 | — |
| B29 | confirmed | med | Delivery to prod schedules for an overridden bot is the manual admin bots push; nothing detects that a committed bot change never reached the platform override — the only tell is a human reading a run's relative file_path. | docs/bot-runs/feed-watch.md:75-77; docs/platform-bots.md:21,55-56,104 (list shows digests; push warns only on webhook-projection drift; version-drift guard covers store-vs-launch races, not git-vs-override) | native:3ef3fd4a |
| C1 | confirmed | critical | AdvanceTriggerCursor (line 161) commits the batch before any Publish (173); a crash or publish error after the CAS loses those board events permanently (publish failure is warn-only). Evaluator warns-and-continues on Promote/Launch errors (74, 97) and ConsumeMatchLabels (87) fires before Launch (96), so a launch failure spends the one-shot label. No outbox exists at HEAD. | pkg/server/trigger_cloud.go:161-175; pkg/trigger/evaluator.go:73-98; pkg/trigger/board_source.go:139-141 | native:f92bbd1e |
| C2 | refuted | low | Original silent-drop claim is false: a full-buffer drop bumps a counter AND Warn-logs source/kind per event; Drops(name) is exported. Residual is only that Drops() is a test/observability helper with no metric surface — acceptable for a local-only bus whose consumers have poll backstops (documented at line 12-16). | pkg/eventbus/inproc.go:56-61 (Warn on drop), 127-136 (Drops exported) | — |
| C4 | confirmed | high | The autofix lane only rides bus.Subscribe on run outcomes; the merge-gate sweeper re-offers dead runs exclusively to reconcileGateForRunID, never to autofixForRun, and no autofix sweep exists anywhere. A dropped/lost outcome event on an opted-in repo means the red gate never launches its fixer until the next human push. | pkg/server/forge_gate_autofix.go:74-82; pkg/server/forge_gate_sweeper.go:121 | native:f92bbd1e |
| C5 | confirmed | med | resumableRunID and both loop.go resume-target sites treat RunStatusCancelled as auto-resumable, so an operator's cancel is silently overridden by the dispatcher retry. Root cause is real: the dispatcher's own stall cancel is a bare context.CancelFunc (state.go:125) yielding the same 'cancelled' status, so the two intents are indistinguishable — a typed cause (CancelCauseFunc) is needed to split them. | pkg/dispatcher/retry.go:201-206; pkg/dispatcher/loop.go:510-512,792-796; pkg/dispatcher/state.go:125; pkg/dispatcher/loop.go:186-190 | native:f92bbd1e |
| C6 | confirmed | med | Store-open/read failures in resume-eligibility, orphan-promotion-persist, re-park and stranded-pause decisions are Debug-logged and the decision silently flips to 'not resumable'/'leave alone' — invisible at the prod info level (the paid Debug-decline lesson). Sweep-list errors are Warn (parked.go:41,197) but the store-open and per-run read paths are not. | pkg/dispatcher/retry.go:188,194,252; pkg/dispatcher/parked.go:50,152,157,204,294 | native:f92bbd1e |
| C7 | confirmed | low | One failed Refresh → runCancel(ErrRunInterrupted), no retry within the ~40s TTL headroom (20s interval, 60s TTL) and no jitter. Mitigations already present soften it: RunnerHeartbeatErrors counter increments, Error-level log, and the interrupted cause yields failed_resumable + JetStream redelivery auto-resume — a KV blip costs an interruption, not a lost run. Design trade documented in CLAUDE.md/docs; a 1-retry budget inside the TTL would be cheap. | pkg/runner/loop_lock.go:71-80 | native:22d5ba09 |
| C8 | partially_addressed | med | The downgrade is real: LoadRun or ScheduleRunRetry persist failure returns usageRetryNotApplicable → the run falls back to JetStream redelivery, which cannot span a weekly usage reset (AckWait/MaxDeliver exhaust → DLQ). But 'silently' is wrong — both paths log at Warn/Error with the fallback named. Gap = no metric and no persist retry, not invisibility. | pkg/runner/usage_retry.go:233-237,249-253 | native:22d5ba09 |
| C9 | confirmed | med | IsRunLocked error → continue with no log and no counter (comment calls it fail-safe retry-next-pass). A persistent NATS-KV fault makes every pass skip every candidate, disabling orphan recovery indefinitely with zero signal — indistinguishable from 'no orphans'. | pkg/server/queue_sweeper.go:132-135 | native:f92bbd1e |
| C11 | confirmed | med | Scan and flip errors are Warn+skip with no failure counter; only success increments RunsOrphanRecovered. Contrast the retry sweeper, which announces itself precisely because its zero-counters are otherwise indistinguishable from 'nothing waiting' — the orphan sweeper has the same ambiguity un-fixed. | pkg/server/queue_sweeper.go:124-129,140-145 vs pkg/server/retry_sweeper.go:58-66 | native:f92bbd1e |
| C13 | confirmed | low | Unknown denial reason → permanent abandon, deliberately (comment: re-arm-forever would be worse) and tested including 'some_future_code'. Abandonment is Warn-logged + audited with the reason, so not silent — but any future TRANSIENT deny code (e.g. a new rate-limit shape) silently discards a paid usage-window retry unless someone remembers the switch. Worth a card (a shared transient-classifier or a conformance test over the deny constants), not a phase-1 fix. | pkg/server/retry_sweeper.go:189-200; test at retry_sweeper_test.go:184-190 | native:2635cef2 |
| C14 | confirmed | low | MaxAckPending is operator-configurable (cfg.NATS.MaxAckPending, 0 → 256) and the '≥ max runner pods' invariant lives only in comments/docs — no runtime assertion or startup warning compares it to anything (the process cannot see KEDA maxReplicas, but a low-value warn e.g. <8 would catch the historic =1 regression class). Default 256 makes real exposure low. | pkg/queue/nats/nats.go:86,116-118,442-448; docs/cloud-architecture.md:122-129; cmd/iterion/server.go:249, cmd/iterion/runner.go:101 | native:de18a16c |
| C18 | confirmed | low | Revoke is Warn-logged best-effort AND dead: no production call site, so board-MCP tokens live the full 24h TTL and forge-publish tokens DefaultMaxWait+24h past run end. Deliberate/documented; residual is token lifetime hygiene, not a silent failure. | pkg/server/valkey_stores.go:112-125,186-198; grep shows zero non-test callers of the token-store Revoke (mcp_board_handler.go:61 / forge_publish.go:85 interfaces implemented but never invoked) | native:2635cef2 |
| C19 | confirmed | low | Claim's framing is wrong — KEDA does NOT read this gauge (scaler hits /jsz), so autoscaling cannot go stale. But a persistently failing consumer-info poll leaves the gauge frozen with only Debug lines, and the schema-rollout runbook's Path A drain check reads that gauge: a stale value can green-light a cutover on a non-empty queue. Debug→Warn (on persistence) is the right fix. | pkg/runner/loop_nats.go:233-237 (Debug on error); docs/cloud-deployment.md:211-226 (KEDA scrapes NATS /jsz directly); docs/cloud-queue-schema-rollout uses iterion_nats_pending_messages=0 as the drain oracle | native:f92bbd1e |
| C21 | refuted | low | The best-effort SetState degrade at pipeline_admission.go:372-388 is unreachable defensive code: the fs store (the only backend the admission loop ever sees) implements the atomic claim, and the cloud/mongo board launches via the dispatcher's CAS Claim. No shipped wiring hits the documented V1 window. | pkg/dispatcher/native/store_issues.go:436 (ClaimForLaunch CAS) + boardstore.go:97 compile assert; pkg/cli/studio.go:311 is the only NativeTrackerStore assignment (admission loop nil in cloud, pipeline_admission.go:68-71); boardmongo has its own CAS Claim (store.go:361) | — |
| C24 | partially_addressed | low | The observed base==head destruction (run 019f8e08, 40 commits) has three shipped guards. Residual fragility: a LoadRunGitMeta read error (perr!=nil) falls through and writes the empty snapshot anyway, and a redelivered attempt with ≥1 commit of its own still full-replaces the prior attempt's commit list. The 'git config best-effort' leg overstates — author seeding and credential wiring FAIL the clone (loop_gitws.go:281-293); only hosts-pin and scratch-ignore are best-effort, with documented backstops. | pkg/runner/loop.go:309-342 (budget ack carve-out, sentinel AND code match); pkg/runner/loop_gitws.go:70-82 (empty range never replaces a recorded snapshot) and :52-58 (integrity skip) | native:2635cef2 |
| C25 | confirmed | low | All as claimed, but every path is deliberate, Warn-logged and backstopped (pool lease TTL + sweeper; mid-run guard behind the fail-open preflight — worst case one wasted call). The one durable gap with NO reconciliation: a failed OrgUsage.AddSpend permanently undercounts the org's monthly bucket, so a cost cap can trip late or never during a Mongo blip. | pkg/runner/loop_spend.go:38-42,72-83 (detached ctx, Warn-only); pkg/runner/usage_cap.go:94-98,110-147 (publish best-effort, preflight fails open) | native:2635cef2 |
| D1 | confirmed | med | Still deferred: cloud artifact files upload only at end of executeRun (loop_upload.go, best-effort). A running cloud run's artifact-files panel stays empty until terminal; ADR-073's acceptance bar was 'identical for a FINISHED run' only. Also still open per the ADR: migrate to-cloud skips the three new blob spaces. | docs/adr/073-cloud-twins-for-fs-only-run-detail-seams.md:119-122 ('Live visibility during a running cloud run is a follow-on'); pkg/runner/loop.go:1664 — uploadRunFiles called once, post-execution | native:bcf9ae5d |
| D2 | confirmed | med | Both legs hold at HEAD: the reattach record is best-effort (a lost write silently reverts to spawn-fresh, re-running the child), and nothing auto-resumes a failed_resumable parent when its answered child finishes — the operator must resume manually; no run-completion subscription or sweeper covers the parent case. | docs/adr/084 consequences ('best-effort … a resume of the parent is still required'); pkg/runview/subbot.go:244-245 (SetSubbotChild failure Warn-only → spawn-fresh degradation) | 9f550afa |
| D3 | confirmed | low | Accurate but pre-arbitrated, not an oversight: modes/kill-switch stay env-only so a runtime write can never re-arm a guard the operator disarmed, and soft→hard is a posture change 'witnessed by a deploy'. ADR explicitly says cheap to add later if the need materialises — a card should carry that arbitration, not reopen it. | docs/adr/090 decision 3 + 'Alternatives rejected: Making the modes runtime-mutable too' | native:5a014d82 |
| D4 | partially_addressed | low | Two of the audit's three legs are deliberate residuals that still stand (ownerReference nil without the chart's downward-API env; no activeDeadlineSeconds for unbounded runs — 'never invent a cap', reaper is the backstop). The third is stale: the lock-less cloud SERVER skips the reaper but the cloud RUNNER now runs it under its NATS-lease authority, closing the OOM-with-surviving-pod window; the in-cluster controller/CronJob is named only as an optional future server-side sweep, not a missing piece. | docs/adr/070:60-150 incl. the managed-cloud runner-reaper home + '2026-07-13' amendment; pkg/runner/reaper.go (ReapOrphanResources wired into the claim loop, lease-first predicate) | native:bfffa06e |
| D5 | confirmed | med | FailNode deferral real: the dispatcher still retries a deliberate FailNode termination fresh (bounded only by the attempt cap) — no typed-error classification, matching the phase-2 need. The 'silently disengages' leg is overstated: a failed FailedState move Warn-logs on every exhaustion, and FailedState defaults to 'blocked' (config.go:351,408), so disengagement needs an explicit failed_state: none or a board rejecting the transition. | docs/adr/015:84-99 (deferral 5); pkg/dispatcher/commands.go:596-604 (exhausted() gates on FailedState) and :650-656 (failed move → Warn 'keeping retry behaviour') | native:393973b3 |
| D6 | already_fixed | low | Both legs closed: a parked card sits in a dedicated 'Awaiting input' column with an awaiting-input badge, and the per-tick parked sweep files an out-of-band-resumed run (finished→completed, hard-failed→failed) and releases the claim while the dispatcher runs. ADR-014's own trade-off paragraph says the follow-ups closed; its Consequences 'open follow-ups' list is stale doc worth a one-line fix. | pkg/dispatcher/native/board.go:23-27,176-183 (StateAwaitingInput, auto-added to existing boards); pkg/dispatcher/commands.go:426-444 (park moves the card + badge); pkg/dispatcher/parked.go:26 + loop.go:33-36 (reconcileParked/reconcileStrandedPaused sweep out-of-band resumes per tick, no restart) | — |
| D7 | confirmed | med | By design and honestly documented, but the aggregate-signal gap is real: every fail-open (team load, active-run count, AllowRun metering) is an individual Warn line with no counter/alert, so a Mongo outage disarming concurrency+quota+metering simultaneously is invisible to dashboards while LaunchDeniedTotal stays flat. A launch_fail_open_total{check} counter is the phase-1-shaped fix; the fail-open policy itself is settled (ADR-032's rollback compensation likewise). | docs/adr/033 (fail-open by decision, 'operators must rely on logs'); pkg/server/launch_gate.go:133,195,251 (Warn-only fail-opens) vs :285-286 (only LaunchDeniedTotal is metered — no fail-open counter in pkg/cloud/metrics/metrics.go) | native:f92bbd1e |
| F8 | confirmed | med | ClaimTick CAS-advances next_fire_at (68) before Gate/Launch; a launch failure is Warn + audit-record-with-error only, slot never retried. Same CAS-before-effect class as C1, though weaker: intentional at-most-once-per-slot (documented, matches host crontab), the error IS captured in the tick audit, and the next cron slot fires normally — so loss is one tick, not permanent silence. For a daily/weekly digest bot one lost slot is a lost day. | pkg/cloudsched/ticker.go:68-78,98-104 | native:f92bbd1e |
Unassigned confirmed/partial items (tracked here until carded): none