DSL errors, persistence and resume

Read it when a run fails, resumes or rewinds, or a checkpoint looks torn (docs/resume.md carries the operator view).

Error Handling

• RuntimeError (pkg/runtime/errors.go) — structured error with Code (type ErrorCode), Message, NodeID, Hint, Cause
  • Codes: NODE_NOT_FOUND, NO_OUTGOING_EDGE, LOOP_EXHAUSTED, BUDGET_EXCEEDED, EXECUTION_FAILED, WORKSPACE_SAFETY, TIMEOUT, CANCELLED, JOIN_FAILED, RESUME_INVALID
• Diagnostics (pkg/dsl/ir/compile.go, pkg/dsl/ir/validate.go) — compile-time warnings/errors with sparse codes C001–C199 plus the C2xx async/bundle families (unknown refs, routing issues, unreachable nodes, undeclared cycles, attachments, presets, capability checks (C080–C082), cursor declarations (C083–C086), etc.; the codes live in pkg/dsl/ir (diagnostics_codes.go and the per-feature files beside it))
• Sentinel errors: ErrRunPaused (resumable), ErrRunCancelled (resumable with checkpoint), ErrBudgetExceeded
• Resumable failures: Most runtime failures produce failed_resumable status with a checkpoint. See docs/resume.md for the exhaustive matrix.

Store & Persistence

<store-dir>/runs/<run_id>/
  run.json              # Run metadata (status, inputs, checkpoint)
  attempts/<sha256>.json     # Immutable source/resources and effective budget per attempt
  events.jsonl          # Timestamped events (one per line, monotonic seq)
  artifacts/<node>/<v>.json   # Versioned node outputs
  interactions/<id>.json      # Human interaction records (questions/answers)
  report.md             # Generated by `iterion report` — chronological run report

The checkpoint embedded in run.json is the authoritative source for resume — events are observational only. See docs/persisted-formats.md for field semantics.

Run statuses: queued (cloud mode only — submitted to the NATS queue, not yet claimed by a runner pod) → running → paused_waiting_human or paused_operator → finished | failed | failed_resumable | cancelled

Key event types: run_started, node_started, llm_request, llm_retry, tool_called, artifact_written, human_input_requested, run_paused, run_resumed, join_ready, edge_selected, budget_warning, budget_exceeded, budget_exit_grace, run_finished, run_failed

Resume from Failed/Cancelled Runs

The engine saves a checkpoint after every successful node execution. When a run fails or is cancelled, the checkpoint is preserved, enabling iterion resume to restart from the failing node without re-executing upstream nodes.

Resumable statuses: paused_waiting_human (needs answers), failed_resumable (automatic retry), cancelled (user-interrupted, checkpoint preserved)

Execution failures routed through the checkpoint-aware path are resumable, including a failure on the first node (resume starts from the workflow entry when no older checkpoint exists). Reaching FailNode is intentional workflow termination and produces non-resumable failed; bootstrap/persistence failures that cannot save resumable state can also end as plain failed.

Common resumable failures: transient LLM errors (rate limit, timeout), budget exceeded (increase budget + resume), schema validation errors (fix workflow + --force), context timeout/cancellation, fan-out branch failures, router failures.

--force flag: allows resume even when the .bot source has changed (e.g., bug fix). Without --force, a hash mismatch produces an error. It never accepts the loss of the run's scratch (SCRATCH_NOT_PORTABLE): --accept-scratch-loss does (ADR-106).

See docs/resume.md for the current status, checkpoint, and override semantics. The console selects an immutable attempt snapshot; a resume with edited source never replaces the previous attempt's graph.

Concurrency

• Fan-out/convergence: Router fan_out_all spawns parallel branches; downstream nodes aggregate via await: wait_all or await: best_effort
• Semaphore: buffered channel enforces max_parallel_branches budget
• Workspace safety: only one mutating branch allowed (agents/humans with tools); multiple read-only branches OK
• Shared budget: mutex-protected token/cost/duration tracking across all branches

Admission, correction bounds and the rollout mode

The reliability invariants of #1006 have owning layers, mapped in ADR-152; the published ledger is docs/workflow-reliability-1006.md, where a criterion reads met only when a NAMED test refuses its violation — three guards in pkg/reliability pin the row count, the existence of every witness and every owner link.

• Admission (admitRun, pkg/runtime/admission.go) runs after the run document exists and before attachments, workspace setup or the first executor call, at launch AND at resume. It compares the normalisation error, the workflow revision against the run hash, the lineage, and the runner-expected revision/parent/policy; it does NOT compare the declared stores (RunStore, BusinessStores) — that half is owed a store-identity seam first. Every outcome is a durable AdmissionDecision: legacy records it unvalidated, report records and surfaces without refusing a run, enforce denies with FailureLaunchFailed.
• Correction is bounded at the producer, on two shipped paths. A tool node's unmet postcondition goes through ADR-044's ladder (pkg/backend/model/executor_verified_action.go) under the author's declared max_repair_attempts / max_agent_attempts, with the postcondition as the only success oracle and a durable node_verified_action event — one bound answering to no test, the self-repair rung's attempt COUNT, which is why criterion 4 reads partial; the self-repair rung runs only for a command or a script recipe (ir.ToolRecipeSelfRepairable: a registry recipe has no corrected command to re-run, and C179 warns when that leaves no rung reachable); a schema-invalid model output gets exactly ONE re-ask (validateAndRetry), witnessed, only for a parse fallback or a missing required field — an eligibility gate no test refuses either. The engine's correction loop (pkg/runtime/node_output.go) has no production implementor yet: it needs output validation enabled, an executor implementing runtime.OutputCorrector, a positive budget and a correction store, and the first two are test-only. ADR-152 decision 5 (the operator's verdict of 2026-11-03) wires it: ClawExecutor implements it, an agent's or a judge's re-ask becomes the round its durable episode counts (an LLM router keeps its in-place re-ask), one round by default, and the ladder stays outside it. Until that unit lands, do not read the loop as the production path.
• The rollout mode (ITERION_RELIABILITY_MODE, pkg/reliability/rollout.go) is the operator's FLOOR and its emergency cap: an explicit legacy is a cap no level overrules, a level may raise the gate and never lower it, and the database level above it is ADR-132 / #2699 territory. It is operator-grade refused to bots and to .env files (internal/envtrust), so no repo-provided environment sets it.