⏯️ Resume paused, failed, and cancelled runs
A run in iterion is durable: it can pause for a human, fail, or be cancelled, and later continue under the same run ID from an authoritative checkpoint — no replay, no lost work — whether that pause lasted seconds or days. The checkpoint, not events.jsonl, is the authoritative execution state.
Resumable states
| Status | Meaning | Public resume behavior |
|---|---|---|
paused_waiting_human | A human node, a mid-agent interaction, or a recovery pause on a failed node is waiting for answers. | CLI and POST /api/runs/{id}/resume; answers are required. |
failed_resumable | Execution failed after resumable state was captured. | CLI and HTTP resume; the restart node is re-executed. |
cancelled | The operator interrupted an active run and the checkpoint was preserved. | CLI and HTTP resume; no answers are required. |
paused_operator | A Studio soft pause, a daily-spend-cap pause, or a declared budget cap whose on_exceeded: policy is pause. | CLI and HTTP resume through the failure-style path; no answers are required. |
running | Execution should still have an owner. | --force-stale can promote a provably stale local run after 60 seconds without an event flush. Server startup also reconciles orphans. |
queued | Cloud queue state. | Internal publisher/runner resume transport; not a user-selected restart state. |
finished, failed | Successful or non-resumable terminal state. | Cannot resume. |
Who parks a run in those states and what brings it back, every edge named with the function or the predicate that owns it — the step table, the guards and the sources are on the flow page:
Reaching the reserved fail node intentionally produces failed. Compile or bootstrap errors can happen before a resumable run exists, and a store failure that prevents checkpoint persistence may also fall back to failed.
A named fail node (fail <name>:, see the DSL reference) may opt out of that with resumable: true, which parks the run failed_resumable — the ordinary resumable status above, resumable by the CLI and by HTTP. Declare it only when continuing is genuinely the cure. The reference case is a phase-budget guard whose remedy is "raise the cap and carry on": leaving that terminal makes the operator re-pay a phase the run already completed, which is the very cost the guard exists to avoid. A refusal that a resume could only repeat — "this lot is not actionable" — stays terminal, the default.
Either way the node's code: lands on the run's failure_code and its rendered message: on error, so what the resume is recovering FROM is legible without opening the run's artifacts.
The checkpoint anchors on the GUARD, not on the fail node. A resume starts execution at the checkpoint's node, so anchoring on the fail node would re-dispatch the fail node and reproduce the identical outcome — the guard that refused would never be re-evaluated and the raised cap would change nothing. The engine therefore anchors a resumable fail on the PREDECESSOR whose outgoing edge routed into it: the resume re-executes that guard against the new caps and takes the other edge. Concretely:
iterion resume --run-id RUN_ID --max-cost-usd 10Fallback, stated out loud. When no single predecessor can be named — the fail node IS the workflow entry:, or several branches converged on it, so no one guard owns the refusal — the promise cannot be kept. The run then ends terminal failed and the engine logs a WARN naming the node and the reason, rather than offering a resume that would silently do nothing. The code: and message: still land on the run; only the resumability degrades. (A fail node reached inside a fan_out_all / fan_out_each branch never takes this path at all: the branch reports the node's diagnosis as its error and the collector decides the run's fate.)
A typed failure is NOT auto-resumed. --auto-resume and the cloud runner's retry both gate on a closed allow-list of engine codes (EXECUTION_FAILED, TIMEOUT, RATE_LIMITED, USAGE_LIMIT_BLOCKED, NETWORK_TRANSIENT, TOOL_FAILED_TRANSIENT, and BUDGET_EXCEEDED with a raised cap — one table, see An ENGINE code can be un-retryable too below). A bot-defined code is outside it, and deliberately so: the run refused on purpose, and nothing an unattended retry can do changes the verdict — only an operator can (a raised cap, a different --var). The run stays parked at failed_resumable for a human, and the log says "not auto-recoverable (code <YOURS>)".
TIMEOUT is on that list, and it is the code a node that ran out its own timeout: ends with — but the in-run ladder does NOT re-run such a node: a retry would run it under a FRESH bound, which turns the one duration the author wrote into N of them, and a node that could not answer within d is no likelier to answer within the next d. The bound ends that attempt; the operator's lane (a raised timeout:, a resume, a redelivery) is what gives it another. The cut is read from the TYPE the site that froze the deadline names, never from the words the backend came back with — a sandboxed crossing reports a killed runner, and the bare context error the in-node retry loop hands up reads exactly like the network needle that would have retried it six times (#2456). A request-level timeout INSIDE the node is a different thing — the upstream client's own bound — and stays the transient fault the longer backoff absorbs.
That is enforced, not assumed: code: is refused at compile time when it collides with one of the engine's own store.FailureCode values (C248), so a fail node cannot mint USAGE_LIMIT_BLOCKED and be auto-retried as a transient provider block. The reserved set is derived from store.ReservedFailureCodes — one list, guarded against drift by a test that parses the constant block.
On a cloud runner the same rule holds, and by two independent barriers. The CLI gate above is one process's decision; a queued run is redelivered by JetStream, which knows nothing about it. So:
- The engine's refusal carries
runtime.ErrDeliberateFailureas the cause of its error. The runner ACKs it (ack-deliberate-failure) instead of NAKing — the shape theBUDGET_EXCEEDEDcarve-out beside it already uses. It is matched by SENTINEL, not by code: the code is bot-defined, so no allow-list could recognise it. - A redelivery that arrives anyway (an older engine, a re-publish) is refused a second time: the runner will not synthesise a resume for a run whose
failure_codeis notReserved(). Only an engine code is auto-resumable.
Without them, a resumable: true refusal looped: NAK → redeliver → the runner synthesises a resume → the guard refuses identically → repeat to MaxDeliver, where the DLQ park overwrites the bot's typed code with DLQ_PARKED — a pod and a sandbox per turn, and the diagnosis destroyed at the end of it. The run now stays failed_resumable with its own code, waiting for a human who changed something.
An ENGINE code can be un-retryable too
Being one of the engine's own codes does not make a verdict re-decidable. EXPRESSION_FAILED (a compute node: no LLM, no shell, inputs from a checkpoint that does not move), CONTEXT_LENGTH_EXCEEDED (the in-node recipe already compacted twice and gave up; a resume rehydrates the same conversation), IR_UNLOADABLE, BOT_REQUIRES_NEWER_ENGINE (a comparison between two constants — the bundle's requires.iterion and the build's own version), WORKSPACE_SAFETY, TOOL_FAILED_PERMANENT and their peers reach the same verdict on every attempt. Which codes those are is one table, pkg/retrypolicy's classification, read by every surface that decides "resume this failure automatically": the cloud runner's classifyExecResult and its redelivery disposition, the CLI's --auto-resume gate, and the dispatcher's retry ladder. Every code the engine declares has a row, guarded against drift by a conformance test.
AMBIGUOUS_EFFECT sits in that table for the opposite reason. Every other deterministic code is parked because a second attempt would reach the same verdict; this one is parked because a second attempt might reach a different one — it may duplicate a mutation that already landed. It is raised when a call that changes remote state loses its answer with no idempotency key to make a repeat safe: the request left, the vendor may have committed it, and nothing available to iterion can say which. Recovery refuses to retry it on any attempt (AmbiguousEffectRecipe), and the producer declares the ambiguity through the runtime.AmbiguousEffect interface rather than by type, so the engine never learns what a connector is. Reconcile the remote state before resuming — a resume re-executes the failing node, which here means performing the call again.
The request left is the load-bearing half of that sentence, and it is checked rather than assumed. A call can fail before a byte is written — the SSRF guard refusing a private host (the default for a self-hosted instance), a name that does not resolve, a connection refused, a header net/http will not send — and none of those leaves anything to reconcile. Those are ordinary transport failures: retryable, EXECUTION_FAILED, on the auto-resume list. The downgrade fires only on a cause that PROVES nothing was sent; an unrecognised failure stays ambiguous, because guessing in that direction is what duplicates a mutation.
The bar for deterministic is deliberately high: a resume re-executes the failing node on freshly resolved inputs, so anything an LLM decided — a SCHEMA_VALIDATION on an agent's output, a NO_OUTGOING_EDGE chosen from it — can differ on the next attempt and keeps its retries. So can AUTH_FAILED: every claim re-materialises the sealed OAuth-forfait blob into a fresh file and refreshes it when it is at or past its expiry lead, so the effective token can differ even though the sealed blob does not.
When a surface declines to bring a run back, it says so on the timeline: run_retry_skipped {reason: deterministic, code, error, hint, status?, also_needs_force?} — the counterpart of run_retry_scheduled. Without it a failed_resumable row whose redelivery was dropped on purpose reads exactly like one still waiting for a pod. Measured on run 01a07804 before the classification existed: seven resumes of one compute-expression failure in ten minutes, each a fresh pod, clone and sandbox. Its hint is the failure's own remedy when the failure names one — a scratch that did not travel names --accept-scratch-loss, never --force — and "fix the cause, then resume with --force" only otherwise; also_needs_force says the resume needs --force besides.
A cloud resume refused before anything claims the run — by the engine (a copy-based subbot child resumed on its own, an incompatible artifact contract, a scratch that did not travel) or by the runner itself (an IR it cannot load, an engine floor it is below) — would otherwise stay queued: the publisher flipped it there before publishing, and a refusal the runner acks is never redelivered. The runner puts the run back in the status the resume came from, which the publisher sends along (ResumeSpec.PriorStatus; failed_resumable from an older server): a paused run keeps its pending question, a cancelled one its cancel, and the refusal is its error (its failure code on failed_resumable). Only that attempt, and only while nobody claimed it — a newer resume or a run the engine claimed is left alone. The event then carries status, the status the run is back to. The revision the refused resume stamped stays recorded: resume with that source, or with --force. The run's error ends with the refusal's remedy when it names one (— hint: …), which the error's own words leave out.
CLI
The source path is optional when it was persisted at launch:
# Failed, cancelled, or operator-paused run: re-execute the checkpoint node.
iterion resume --run-id RUN_ID
# Human pause: answer fields individually or with typed JSON.
iterion resume --run-id RUN_ID \
--answer approved=true \
--answer note="ship it"
iterion resume --run-id RUN_ID --answers-file answers.json
# Human pause: upload a file for a `file`-typed output field (curl-style @).
iterion resume --run-id RUN_ID --answer music=@./theme.mp3
# Resume after deliberately changing the workflow source.
iterion resume --run-id RUN_ID --file workflow.bot --forceOne mismatch is accepted rather than refused: a run launched from a bundle's main.bot before the bundle's prompts/*.md and presets/*.md entered the workflow digest recorded the digest of the source bytes alone. A resume that finds that bare digest accepts it — and the artifacts the run published under that revision, as --force would — logs it, and goes on; nothing in the source changed, so no --force is asked for. The run keeps the digest it recorded (rewriting the document outside the engine's claim would race its other writers), so the acceptance is logged on every resume of such a run. Any other mismatch stays a refusal.
A forced source edit revalidates a convergence node's settled incoming edges against the current graph. If the node executes and publishes again after an artifact-supplying edge was removed, its new artifact contract records only the dependencies it now consumes. The old immutable revision keeps the dependencies it actually consumed before the edit. This works under enforce; --force still requires valid persisted producer identities and required historical dependencies.
--answer is repeatable and carries strings; the runtime coerces them to the paused node's output schema. --answers-file preserves JSON types. Explicit flags override keys loaded from the file. A file-typed field is answered with key=@./path — the path is staged as a run attachment before the node resumes (the same upload the studio gate performs); the @ convention is honoured only for fields the schema declares as file, so a string value that legitimately starts with @ passes through untouched, and a literal leading @ inside a file field is written @@.
Use iterion inspect --run-id RUN_ID before taking over a suspicious running run. --force-stale refuses when events.jsonl was flushed less than 60 seconds ago, and the per-run lock still prevents two resume processes from executing concurrently.
What the checkpoint preserves
Current checkpoints include:
- the restart node and accumulated per-node outputs;
- loop counters, round-robin positions, loop previous/current snapshots, and
foreach-relevant output state; - artifact version counters and resolved variables;
- node recovery-attempt counters;
- pending interaction questions and interaction ID;
- backend session/conversation information for mid-agent
ask_user; - tokens, cost, iterations, active duration, and cumulative run spend.
Budget accounting therefore does not reset on resume. Legacy checkpoints that predate the accounting fields deserialize them as zero, which retains the old fresh-budget behavior for those historical runs only.
A best-effort checkpoint is written after each successful node boundary. On a normal execution failure, the final checkpoint is anchored at the failing node; resume re-executes it. Cancellation and operator pause anchor the node that was about to execute. If a failure has no checkpoint, the runtime can restart from the workflow entry.
Treat restartable node effects as at-least-once: a node may run again. Tool commands should be idempotent or detect already-completed work, and coding agents should inspect the durable Git history before repeating an item.
The checkpoint restores outputs, not files
A cloud run bound to a repository (RepoURL on the queue message) gets its workspace from a clone the runner makes on every claim: it RemoveAlls the per-run directory before cloning, and deletes it again when the run returns. A resume therefore starts from a pristine checkout of the run's ref — anything an earlier node edited but did not commit is gone, and the resume is normally claimed by a different pod anyway.
The checkpoint still replays that node's outputs. So a downstream node reads "the previous node changed these files" against a tree where they no longer exist, and the two disagree with nothing failing. A re-executing run emits run_workspace_reset so the discarded tree is visible on the timeline — keyed on the checkpoint's existence rather than on the delivery being shaped as a resume, since a redelivery of a run still marked running re-clones the same way.
What the fresh clone does NOT lose is committed work. A re-execution restores what the run's earlier attempt left on the forge — the storage branch its death bank recorded (final_branch / final_commit), else the newest attempt ref a pause or an interrupted delivery parked — and emits run_workspace_bank_restored naming the branch, the head and the base it was put back on. The restore is two-step (the chain's own base, then a fast-forward to its head) so the clone's reflog still reads "started from the run's base": a bot that derives what the run changed from the newest reflog entry that is not its own commit (docs-refresh's scope gate) does not mistake the commits the target branch gained meanwhile for the run's work. A bank branch that moved past the recorded head, or a chain with no common ancestor, is refused loudly (restored: false + reason) and the run continues on the fresh clone.
The consequence for authoring: do not separate a node that mutates the workspace from the node that persists the mutation by a resumable boundary unless something checks the two still agree. Either commit inside the mutating node, or have the persisting step verify against the tree rather than against the upstream node's claim (dep-update-guard is the worked example — align_probe checks align's claim the moment it returns, against the tree it left and against the PR's own commits, and commit_check blocks when either shows work the branch head does not carry; the reading of commits survives the fresh clone a resume starts from, the reading of the tree does not). Git is the durable state; an uncommitted working tree is not.
The scratch travels in its own bank
${PROJECT_SCRATCH_DIR} is not the workspace and no commit carries it. In a sandbox whose scratch is container-local (kubernetes always; docker without the host bind) it dies with the sandbox, and a resume starts a new one. So the teardown of every run but a finished one — a park, a failure, a failed run a rewind can bring back — streams the scratch, as a gzip'd tar capped at 256 MiB, into the run's scratch bank, and every resume path extracts it into the new sandbox before its first node (ADR-106). The timeline shows both halves: sandbox_scratch_banked (banked, empty, bytes, or the reason it could not be banked) and sandbox_scratch_restored (restored, bytes; accepted and reason when a resume accepting the scratch's loss went on without it; host_backed when it was restored into a host directory, which keeps the scratch from then on — no bank is restored over it again).
A run whose last teardown left a scratch it could not bank (past the cap, a tar or upload failure, a store that keeps no bank) is refused SCRATCH_NOT_PORTABLE: its next nodes would otherwise find the scratch empty and fail far from the cause. So is a run that moved past its bank — an agent, tool or subbot node finished after the last teardown banked, in a sandbox lost without a teardown (an OOM kill, a lost node): restored, the bank would revert what that node wrote. So is a subbot child that executed in its parent's sandbox while that sandbox's scratch lived in the container, resumed on its own. These refusals come before the check of the workflow source — its digest and a shared dependency's identity alike — and name it when the source changed: the resume then needs both consents, --accept-scratch-loss and --force. --force alone never accepts the scratch's loss: resumes send it without condition (a conversational turn, an assistant's), and the loss needs a consent of its own, given once the refusal was shown. That consent is the resume's: an automatic retry of the same engine, a redelivery after its claim or an adoption does not carry it. A surface about to refuse a resume for a reason --force accepts (a changed source, an artifact contract) shows the scratch's loss first, even while the latest execution may still be banking. Over HTTP the refusal answers error_code: scratch_not_portable, with its hint and also_needs_force when --force is needed too; iterion remote runs resume prints both. A refusal only the engine meets — after the surface deferred to it, or at the restore — names the same consent on the run's error, on run_failed (hint, also_needs_force) and on run_retry_skipped. The resume surface refuses from a record the run's latest execution wrote, before anything moves the run (the studio and the API answer at once; a cloud resume is never flipped to queued). A run reads paused or failed while its teardown still banks: when the latest execution wrote no record yet, the surface leaves the decision to the engine, which checks again under the run's lock, before its claim. Relaunch the run fresh, or resume with --accept-scratch-loss to continue as it stands — without the scratch, or with the older bank, marked stale. The node a resume records as finished by its answer (a human node, or an agent that asked) ran in no sandbox: it does not age the bank. A teardown that cannot even list the scratch (the sandbox is already gone) records it as unknown: after a bank, that bank — still stored — is what the next resume restores (or refuses as moved past); otherwise the resume goes on without it, as it always did, and says so. The teardown writes the bank and its record under the run's identity, after the run's cancellation. It tries again what another try may cure — a blip on the exec, a failed tar, a failed upload (on the same archive) — and retries the record until its budget runs out. Before tar reads the scratch, the sandbox's other processes are stopped — in a sandbox whose process namespace is its own: a kubernetes pod, a docker or podman container not sharing the host's — so no write tears the archive unseen, and they go on once it is taken. A sandbox not read as isolated, a quiesce that fails, and one that could not stop or check every process are recorded (unquiesced, repeated on the restore). tar runs untranslated (LC_ALL=C); a member it catches changing while it reads it may be archived torn or missing, so tar runs again while it races, and the last complete archive is banked — even when a later try fails — with the raced members named (raced, repeated on the restore).
Only a node that ran in the sandbox and succeeded ages the bank — an agent, a judge, a tool, a subbot or an LLM router, as the workflow that executed it said (recorded on its finish, whatever an edited source says since); one that failed re-runs from the checkpoint, and a rewind takes back the nodes it dropped — those on no loop or foreach cycle, whose earlier passes it does not replay (a rewind across a loop or a fan-out is refused; --accept-scratch-loss resumes it on the bank). The scratch itself is not rewound. A resume that went on without the bank, its loss accepted, forsakes it; one that restored a stale bank makes it the run's scratch again.
At resume, the bank is read onto the host and checked to extract before anything reaches the sandbox. A bank that is gone or does not extract — or a resume that runs without a sandbox — parks the run under the same code; --accept-scratch-loss continues without it. A read that fails on the way (the timeline, the bank's store, the stream into the sandbox) parks it with no code, so the resume is retried, whatever consent it carries, and the bank is kept for that attempt.
Not banked: the host scratch of an unsandboxed run, or of a sandbox that binds it to a host directory. It survives a resume on the same host, and is lost on another one, which happens in the cloud runner's in-pod mode — and when the host state is switched off between two attempts.
When the final bank push fails
The workflow outcome and delivery of its commits are separate. Run detail (run.bank_state) and list (bank_state) responses expose banked when final_branch and final_commit are recorded, or bank_failed when final_branch_error is present. The field is omitted while no bank outcome is established, including runs with no work to bank. It is derived from those existing fields, so old runs need no migration. A finished workflow stays finished even if its bank fails. Run outcome trigger payloads carry the same bank state and final fields when known; merge routing escalates a bank failure.
The runner makes two push attempts by default, waiting two seconds before retrying. ITERION_RUNNER_BANK_ATTEMPTS=1 disables retries; ITERION_RUNNER_BANK_RETRY_DELAY accepts a Go duration (including 0). The retry re-reads the remote branch: an already-pushed head is success after a lost acknowledgement, a changed head passes the existing richer-chain checks, and an unreadable remote stops the retry rather than dropping its lease. A head that has NOT moved was already archived and compared on the first attempt, so the retry skips straight to its push instead of re-fetching and re-archiving it. Even observed absence is protected by a lease on the second push. Cancellation interrupts the wait. Each git operation keeps its existing ITERION_RUNNER_GIT_TIMEOUT bound (15 minutes by default); a shorter caller deadline is identified separately in the error.
The bank sequence as a whole is bounded too, so a wedged forge cannot pin a runner pod for attempts × operations × the per-op ceiling. A run past its own deadline gets a fixed 10-minute grace period; a run still inside its deadline — or launched without --timeout, which has none — gets twice ITERION_RUNNER_GIT_TIMEOUT, so the aggregate never pre-empts an operation the per-op setting allows and raising that setting extends the bank with it. Setting ITERION_RUNNER_GIT_TIMEOUT to 0 or less keeps git operations unbounded and leaves the sequence unbounded as well.
Inspect run_bank_retry for the failed attempt, planned delay, redacted git error, failure_kind (timeout, cancelled, process_exit, or git_error), and exit code when available. Once retries are exhausted and no earlier banked pair exists, run_bank_failed records the final cause and attempt count; recorded=false means the run-document write failed too. Integrity refusals also emit run_bank_failed when no previous bank is available. An earlier valid pair stays intact: a later refused attempt remains visible through run_bank_refused. Document-read failures and operator cancellations retain that existing refusal event, since the runner cannot establish a new bank state.
For recovery, inspect the run's final fields and timeline, then verify any remote ref against its recorded commit. A local final_commit accompanied by bank_failed is not proof that the commit reached the forge. Check the workspace-checkpoint hint below if the final bank has no branch. An exit code alone (including 12) does not identify the historical root cause: use the recorded stderr and timeout/cancellation classification before retrying. No bank failure automatically re-executes the workflow or authorizes a merge.
Recovering a workspace checkpoint after a pod dies
A copy-based sandbox can push a workspace checkpoint before teardown even if the run later dies without final_branch or final_commit. Inspect iterion remote runs get <id>: its workspace_checkpoint recovery hint names the latest successfully pushed checkpoint recorded in the run's timeline, with its ref, commit, event sequence and timestamp. The same hint is returned by iterion remote runs commits <id> when the commit list is unavailable. available=false and reason=no_baseline still mean that the API cannot produce the run's commit range; they do not imply that no saved work exists.
The JSON paths are run.workspace_checkpoint on GET /api/runs/{id} and workspace_checkpoint on an unavailable GET /api/runs/{id}/commits response. source=run_workspace_checkpoint identifies the persisted event provenance. A later failed push does not erase the last successful record. If there is no successful record, the field is omitted; no ref is invented from the run ID. An event-store read failure remains an error, rather than an absent hint.
Run the provided fetch_command from a clone of the run's repository with that repository configured as origin, then check git rev-parse FETCH_HEAD against the recorded commit. The record proves a push succeeded at that time, not that the ref still exists or still points to that SHA. A checkpoint may include unfinished or automatically committed work and is not a delivery bank or proof of a passed delivery gate: inspect and validate it before merging. Merely displaying this hint does not restore files, resume the run, or change merge eligibility.
Source integrity and bundles
The launch stores a SHA-256 workflow hash. Resume recompiles the source and refuses a mismatch unless --force is present. The CLI and HTTP service perform this check before dispatching the resumed execution, so a rejected Studio answer remains on the paused form; the Studio then offers an explicit Resume with updated workflow (force) retry. Force is useful after repairing the workflow, but it is an operator assertion that stored outputs, node IDs, schemas, and the new graph are still compatible.
The hash covers everything the run's program was made of: the main file's bytes, a bundle's prompts/*.md and presets/*.md, then — only when the unit has them — the fragments its import lines reach (see dsl.md) and the files its prompts' {{include}} markers read, nested ones too. A single-file bot without includes hashes exactly as it always has. A fragment or an included file edited while a run is parked is therefore a source change the gate refuses without --force; a run of a bot that uses {{include}}, launched before the include closure entered the hash, compares differently once and resumes with --force that one time.
--file defaults to the persisted FilePath. Bundle runs also persist their bundle path; resume reopens a .botz or bundle directory so prompts, skills, attachments, recipes, and the selected preset are restored. If the original archive moved, pass its new path explicitly.
Resume also rebuilds the execution environment: sandbox, run work directory, plugin contributions/hooks, library skills, subbot runner, and worktree context. A worktree run that eventually finishes goes through the normal persistent-branch and merge-finalization path. A worktree workspace whose .git pointer names a gitdir that no longer exists (the repository that registered it is gone) refuses to resume with an explicit error — every git command there would fail, and nodes would read the workspace as "no repo". Cloud runs never create that shape: with a rootless store (cloud Mongo) the engine skips worktree isolation and runs in place in the per-run clone, so the workspace survives the runner's clone-recycle between queue deliveries.
Overrides on resume
The resume command accepts the same recovery-relevant controls as launch:
| Flags | Use |
|---|---|
--max-cost-usd, --max-tokens, --max-duration, --max-iterations, --max-parallel-branches | Raise or replace effective workflow caps. Already-consumed accounting remains charged. Zero means inherit. |
--permission, --permission-allow, --permission-ask, --permission-deny | Rebuild the tool-permission policy; an allow rule can authorize the call that previously paused or failed. |
--model, --backend | Re-apply selector-based execution overrides. Launch-time rules are stored for display but are not automatically re-applied to the resumed executor. |
--sandbox, --sandbox-default-image, --sandbox-host-state | Replace the launch-time isolation choice persisted on the run. Empty inherits the launch's decision (a run launched with --sandbox none refuses docker on resume too); non-empty overrides it on purpose. |
--merge-into, --branch-name, --merge-strategy, --auto-merge | Replace the launch-time worktree-finalization choice persisted on the run. Empty inherits; non-empty overrides. |
--auto-resume N | Retry eligible transient/rate-limit failures, or budget/timeout failures when a larger cap was supplied, with bounded backoff and forfait-cap checks. |
--force | Permit a source-hash mismatch (or an artifact-contract change). Never the loss of the run's scratch. |
--accept-scratch-loss | Resume although the run's scratch does not travel (SCRATCH_NOT_PORTABLE): without it, or with an older bank. This resume's consent only. |
--force-stale | Take over an orphaned local running run after the staleness guard passes. |
The launch's decisions travel with the run
Every launch-time CLI override falls into one of three verdicts on resume — each row is a field of Run (pkg/store/run.go) or a runtime option that reads it back. Silently-lost overrides (marked ✗) meant a resume changed the isolation, storage-branch or merge behaviour behind the operator's back; ADR-based work brings each into the "persisted-and-replayed" column (✓). Fixed at #1435 and #1366.
| Launch flag / knob | Persisted at launch (field on Run) | Replayed on resume | Notes |
|---|---|---|---|
--sandbox | ✓ SandboxOverride | ✓ CLI resume + studio resume | Cloud runner reads its own cfg.SandboxOverride (architectural — the pod IS the isolation boundary). |
--sandbox-default-image | ✓ SandboxDefaultImage | ✓ | Cloud runner reads msg.SandboxImage. |
--sandbox-host-state | ✓ SandboxHostState | ✓ | Cloud runner defaults come from cfg.SandboxHostState. |
--merge-into | ✓ MergeInto | ✓ CLI + studio + cloud runner | Empty means "current" for local resumes. Cloud runner reads it back at pickup (pkg/runner/loop.go's engine options) so a --merge-into none launch stays none on redelivery — Philosophy #3 (cloud-native by construction). |
--branch-name | ✓ BranchName | ✓ CLI + studio + cloud runner | Fixes half of #1366: the default storage branch is keyed on the run ID — the stable key — so a consumer looking up the branch by id finds it whether the run was straight or post-resume. --branch-name still overrides; the cloud runner replays it too. |
--merge-strategy | ✓ MergeStrategy | ✓ CLI + studio + cloud runner | Was persisted before this PR; readback added here for all three surfaces. |
--auto-merge | ✓ AutoMerge | ✓ CLI + studio + cloud runner | Was persisted before this PR; readback added here for all three surfaces. |
--var | ✓ Inputs | ✓ | Replayed as run inputs. |
--preset | ✓ Preset | ✓ | Replayed. |
--skill | ✓ ExtraSkills | ✓ | Conversational-bot dock relies on this. |
--model / --backend / --effort-for | ✓ ModelOverrides | ✓ | Executor spec re-reads them; CLI can layer explicit overrides. |
--fallback | ✓ Fallback | ✓ | Usage-window auto-retries rely on this. |
--permission* | ✓ PermissionOverride, PermissionMode | ✓ | The strongest-precedence gate mode. |
--max-cost-usd etc. | ✓ BudgetOverrides | ✓ | Merged with any explicit resume --max-*. |
--recipe | ✓ (via compile) | (no resume equivalent) | Recipe is applied at launch; the compiled workflow persists. |
--auto-memory | ✗ (not persisted) | Recomputed on purpose | Resume --auto-memory re-states it; env ITERION_AUTO_MEMORY decides otherwise. |
--compress | ✗ | Recomputed on purpose | The rewriter chain is process-scoped; a resume re-resolves it from workflow + env. |
--loop-budget-guard | ✗ | Recomputed on purpose | Resume --loop-budget-guard re-states it. |
--supervisors | ✗ | Recomputed on purpose | Resume --supervisors re-states it. |
--repo-devbox | ✗ | Recomputed on purpose | Resume --repo-devbox re-states it. |
--review-mode | ✗ | Recomputed on purpose | The topology is applied at compile; a resume re-uses the resulting Inputs. |
--skip-mcp-health | ✗ | Recomputed on purpose | An MCP server that was down at launch may be up on resume — respect the operator's fresh choice. |
The chokepoint is pkg/cli/resume.go on the CLI side and pkg/runview/service_launch.go (Resume + engineOptions) on the studio/HTTP side, both reading the Run fields and layering explicit resume flags over them. pkg/runtime/engine_options.go's WithFilePath stores the launcher's path VERBATIM on Run.FilePath; the sandbox bind-mount source is absolutised at its own chokepoint (bundleResourceDir in pkg/runtime/sandbox_devbox.go), so a caller that hands a relative --file examples/foo.bot no longer leaks the relative form into the docker --mount source= argument while the run doc keeps the shape the launcher wrote (readers of Run.FilePath — the studio, the dispatcher, pkg/server/run_delegation.go — see the launcher's meaning).
Answers that survived a failed resume. A resume that fails AFTER recording the operator's answers (the classic case: sandbox start refused a mount; recordHumanAnswers already ran) does not re-ask on the next resume. The dispatch (Engine.Resume) consults a PREDICATE, answeredHumanGateReplay: the checkpoint must anchor on a human node whose blocking-pause interaction exists at the exact interactionIDForPause(runID, nodeID, loopCounters) for the current iteration, is answered, non-retired, and NOT stale (Run.LastRewindAt at or after AnsweredAt means a rewind replaced the execution the answer belonged to — the operator is re-asked). When the predicate holds, the run's status flips back to paused_waiting_human and the resume routes through resumeFromPause UNMODIFIED — so recordHumanAnswers (+ its human_answers_recorded event), the artifact publication, the markPreNodeBoundary parked-window close and the edge selection are THE pause path's code, never a copy that can drift. A non-empty caller --answer map is what gets recorded through that same write, correcting a stale stored answer. Fixes the third defect of #1435 and the PR #1490 gate findings Rac891d (rewind epoch + interaction retire), R62a836 (single transition, no duplicated boundary) and R60aa7e (corrected answers recorded).
The predicate is also consulted on a paused_waiting_human run whose pause pointer is already consumed — the shape a gate replay lands in when it fails between the status flip and the claim — so that wedge recovers on a plain retry instead of dying on an empty interaction id. One carve-out: a durable mission resume (expectedResumeStatus, ADR-095) skips the replay to keep its exact-status CAS contract — the mission re-asks the human gate rather than flipping the status out from under its own claim.
When raising a budget, choose a cap above the amount already consumed. Merely repeating the old cap causes the re-executed node to hit the same guard.
Cloud runs. POST /api/runs/{id}/resume (and iterion remote runs resume) accepts the same budget-override flags as the local CLI — --max-cost-usd, --max-tokens, --max-duration, --max-iterations, --max-parallel-branches. The wire body carries them as {"budget": {"max_duration": "4h", ...}}; the override MERGES per field over the launch ask persisted on the run doc — a non-zero field in the spec beats the doc, a zero field inherits it. Passing only --max-duration 4h therefore raises the duration without erasing the launch's cost/tokens caps. Without any override the resume replays from the doc as before.
Two mechanics matter when raising a cap here:
- The consumed accounting travels across resume. The checkpoint restores
budget_elapsed_ns,budget_cost_usd,budget_tokens_usedandbudget_iterations_usedon every axis, so a resume with no override restarts against the same clock that killed it. Measure: a run parked on a 2h30m duration cap (elapsed = 9902 s) resumed bare walks another 5 min and dies at 9901.6 s + 10% = the exit-grace ceiling. The escape hatch is exactly what--max-*on resume raises. - The merged ask is persisted onto the run doc. So a subsequent unattended auto-retry (usage-window sweeper) keeps the raised cap instead of reverting to the launch ask that already killed the run. A run resumed with
--max-cost-usd 120retries at $120, not at the launch's $10.
The --max-* flags are the recommended path. The historical "source-swap" recovery (POST {"source": "<the .bot with a larger cap>", "force": true}) is NOT equivalent: it edits wf.Budget before compile, and resolveResumeBudgetAsk then still merges the persisted launch ask over top — on a run whose launch ask carried an explicit cap the swap is overridden and the run dies at the persisted cap. Prefer the flags; the swap is retained as a last-resort escape for the case where no launch ask was ever recorded.
Rewind: resume from an earlier node
Resume always restarts at the checkpoint node. iterion rewind moves that checkpoint backwards onto a node the run already executed, so the next resume replays from there instead of from where the run stopped. It is the loop for iterating on a bot's configuration: run it, watch a node misbehave, fix the prompt/schema/edges, replay just the affected part.
# edit main.bot, then let iterion locate the edit:
iterion rewind --run-id RUN_ID --auto
iterion resume --run-id RUN_ID --force
# or name the pivot yourself:
iterion rewind --run-id RUN_ID --node implement--auto: rewind to the node you edited
--auto diffs the workflow source the run executed against the source as it is now, and rewinds to the earliest node the edit affects — so the loop is edit, rewind, resume, with nothing to translate by hand. "As it is now" is the file on disk for a local run, and for a run served by a stored bot (a team bot or a platform override) the server re-resolves that bot's row at its current version: such a bot has no file on the pod, and the path resolves to the baked catalog twin, which is a different program. It prints what it detected, so you can confirm it understood the change before resuming.
A bot in several files (import "lib/x.bot") records every file of its unit on the run (workflow_sources, beside workflow_source for the main), so an edit in a fragment is seen like one in the main. A run of such a bot that recorded its main alone — launched before the unit's files were recorded, or over the 1 MiB cap — is refused rather than diffed on the main, and asks for --node. A fork of a run over that cap can no longer pre-check--new-inputs against the recorded source: the fork is admitted with a note, the supplied values are recorded on the child, and the child's first resume judges them against the var constraints — see the fork command. Forking it without changing an input — the recovery path — is unaffected, re-sending the parent's own values included.
Detection is declaration-granular and resolves indirection:
| You edited | --auto rewinds to |
|---|---|
| A node's prompt, model, schema ref, tools, command… | that node |
A shared prompt / schema / cursor body | every node referencing it, earliest first |
An edge (condition, with mapping, loop bound) | the node the edge leaves, which re-selects it |
vars:, budget:, sandbox:, permission:, entry: | the entry node — workflow scope reaches everything |
| Several nodes on one chain | the upstream-most one, so a single pass tests them all |
It errs toward reporting more nodes than strictly necessary: a false positive costs re-executing a node that did not need it, while a false negative would test the new configuration against stale downstream state — the failure the feature exists to prevent. Nodes the run never executed are ignored, and edits on independent fan-out branches are refused with the candidates named, since no single pivot covers them.
Iterating repeatedly is safe: the source is re-stamped as each resume takes the run, so the second rewind of a session diffs against what the first one actually ran rather than re-reporting its edits.
That re-stamp has two authors, because the text lives in two different places. A local resume is stamped by the engine, which compiled the file itself. A cloud resume is stamped by the publisher, immediately before it enqueues: the queue message carries the compiled IR and the identity hash and never the files, so the runner's engine has no text of its own to record. It writes the source together with that compile's hash — a source stored beside another revision's hash is a false baseline, which a forced resume clears on purpose. A cloud resume that is refused (credentials, a queue outage) puts the previous baseline back with the status, so only an attempt that actually reached a runner moves it.
--auto needs Run.WorkflowSource, the .bot text of the program the run is executing — captured at launch and re-stamped by each resume (WorkflowHash only answers whether the source changed, never which node). Runs started before that capture existed refuse --auto and still accept --node. Comments and line shifts are not changes: the comparison runs on the AST encoder, which omits source spans.
The run keeps its id, name, inputs, lineage, and budget accounting. Choose between the two operations by intent:
iterion fork | iterion rewind | |
|---|---|---|
| Run id | New child run | Same run, mutated |
| Original | Untouched | Outputs downstream of the pivot invalidated |
| Intent | An alternative future worth keeping side by side | "I misconfigured this — back up and replay" |
| Anchor | Requires a turn checkpoint, so LLM nodes only | Any node with a recorded output, including tool and compute |
| Audit | Two runs | One run plus a run_rewound event |
What gets invalidated
The pivot's output is dropped, plus every node forward-reachable from the pivot that cannot reach it back. Subtracting the ancestors is what makes loops behave: in implement -> verify -> implement as fix(3), verify is both downstream and upstream of implement, and its output is exactly what implement re-reads through {{loop.fix.previous_output}} on re-entry — so it survives. Over-dropping is as harmful as under-dropping, because a node whose output is deleted without re-executing before something reads it resolves to nil rather than to a re-run.
Deliberately not reset:
- Budget accounting and loop counters. Refunding them would make rewind-then-resume an unbounded way around
max_cost_usdandmax_iterations. Raise the caps on the resume instead (--max-cost-usd,--max-iterations). - Artifact versions. Re-execution appends a new version, so the superseded artifacts stay on disk and remain readable for comparison — invalidation here is logical, not a delete.
events.jsonl. Append-only, never truncated. The dropped nodes' original records stay in the timeline, and therun_rewoundevent explains why they are about to repeat. Its payload is{from_node, to_node, dropped_nodes, tombstoned_artifacts, orphaned_child_runs, retired_output_corrections, promoted_from, files_reverted, files_ref, files_revert_commit, files_backup_ref, files_skip_reason, files_restore_scope, files_scope_count, files_overwritten, files_left_in_place, files_overwritten_paths, files_left_in_place_paths, receipt_id}.
Reset, because carrying them into the replay would be wrong: any pending interaction (the rewound run never asked that question) and the backend session/conversation rehydration (the pivot must replay against the edited prompt, not the conversation the operator is trying to change).
Preconditions and limits
The run must be failed, failed_resumable, cancelled, paused_operator, paused_waiting_human, or queued — never running, whose engine owns the checkpoint and would overwrite the rewind at its next node boundary. Cancel or pause it first. The claim is a CAS, so a concurrent resume loses the race rather than being rewound out from under itself.
failed stays semantically distinct from failed_resumable: the graph deliberately abandoned the run (no auto-resume), it did not crash. But the distinction no longer destroys the recovery point — a run that reaches the DSL fail node keeps its checkpoint, so an explicit rewind can still recover it. Runs that failed before that preservation existed carry no checkpoint and stay unrecoverable (run ... has no checkpoint — nothing to rewind).
An untouched worktree released after an early terminal refusal is rebuilt at its original launch commit when rewound, even with --restore-scope none. This applies only to a recorded deliberate reclamation; an unexpectedly missing checkout or a broken git link still raises an error. Resumable failures keep their worktree in place.
After a successful rewind, the run is parked in paused_operator, not cancelled: it is ready at the chosen checkpoint and has not been abandoned. It also carries an internal explicit-resume-required marker. A cloud runner therefore drops any old launch delivery instead of auto-resuming it; only a real Resume request may continue. That request carries the normal source hash check (--force after an edit). A genuine automatic Resume request from a separate recovery path is still allowed, but the same hash check prevents it from executing an edited workflow without force.
The rewind reaches that park through cancelled: it claims the run by writing that status on the exact document it read, so a workspace revert that fails after the claim leaves the run cancelled with its pre-rewind checkpoint.
Rewind resolves "downstream" against the workflow source as it is now, which is why it performs no hash check — you rewind precisely because you edited the .bot. The resume that follows still needs --force. Pass --file when the source is not where the run recorded it.
Subbots and parallel branches
A subbot node is an ordinary graph node, so the topology rules above apply to it unchanged. But the child run is tracked outside the checkpoint, in Run.SubbotChildren (ADR-084), and ReattachSubbotChild consults only that map and the child's status — never the parent's checkpoint, and before the child .bot is even compiled. Dropping the node's output is therefore not enough on its own: rewind also releases the child pointers of every dropped node, so the replay launches a fresh child against the edited child workflow instead of adopting the previous one. The released child run ids are reported (orphaned_child_runs); the runs themselves are left alone, so cancel one that is still burning budget.
--auto does not see edits to a child .bot: it diffs the parent's source, and SubbotNode.Source points at a different file. After editing a child workflow, target the subbot node explicitly with --node.
The checkpoint is granular to the node, never to the execution: at convergence the engine writes one entry per node id (last write wins), so N parallel executions of the same node collapse to one output. A rewind naming a node inside a fan-out body is therefore promoted to the router that orchestrates it, and the whole fan-out replays. Anchoring on the body node would replay it once, linearly, with no each context — silently testing one iteration instead of all of them. The promotion is reported as promoted_from; nested fan-outs promote to the outermost router, since an inner one re-run alone would itself lack the outer iteration's context.
Sequential loop and foreach bodies need no promotion — they execute one at a time. Rewinding into one resumes at the current iteration, because loop counters are preserved; restarting the loop from zero would also refund the max_iterations budget.
A concurrent run edit can make a rewind or rename return HTTP 409. Reload the run before retrying: full-document saves compare the version that was read, so they cannot undo a cancellation, resume, or checkpoint written meanwhile. A failed final rewind save can leave workspace restoration already applied; inspect the workspace before retrying when file restoration was requested.
Two further limits worth planning around:
- Only engine state is rolled back. Board cards, forge comments, pushed commits, and already-launched subbot child runs are not. A rewound subbot node re-runs and launches a new child; the old one stays, orphaned but traceable.
- A rewind cannot target one branch of a fan-out. The pivot is promoted to the router and every branch replays (see above) — there is no per-branch state to rewind to.
Files the dropped nodes produced
A node's real product is often not its output map: a docs or code bot writes dozens of files and returns a summary. Rewinding the checkpoint alone would leave that half-written tree in place, and the replayed node would build on top of its own previous production instead of meeting its prior conditions.
So a rewind also restores the workspace to the state the pivot started from. The anchor is refs/iterion/runs/<run>/pre/<node>/<iter>, written by markPreNodeBoundary when the node starts — distinct from refs/…/nodes/<node>/<iter>, written when it finishes and therefore holding that node's own output. The two bracket each node's effect. The pre-boundary marker is an alias of the previous boundary's snapshot commit (nothing touches the tree in between), so it costs one update-ref and no extra index walk.
It is a revert, not a reset: the prior tree is committed on top of HEAD, so the run's own commits stay in git log, and the state at the instant of the rewind — uncommitted and untracked work included — is banked first under refs/iterion/runs/<run>/rewind/<node>/<seq>. Nothing the run ever had becomes unreachable. --restore-scope none (formerly --keep-files) opts out.
A run without a worktree cannot use that path at all: its workspace is the operator's live checkout, and git add -A there would stage their own uncommitted work as a side effect of running a bot. That is the default shape (17 of 30 catalog bots), so those runs are versioned by iterion itself — see workspace versioning. The rewind picks the mechanism per run and reports which one ran:
| Run shape | Files restored by |
|---|---|
worktree: auto (docs-refresh, feature-dev, wiki-gen…) | per-node git snapshots |
| in-place (whole-improve-loop, modernize, review-pr…) | pkg/workspacetrack |
| paths the ignore rules exclude, oversized files | neither — reported in files.skip_reason / files.restored.skipped |
| paths outside the workspace | neither |
| paths no execution of the run recorded changing (in place, default scope) | neither — reported in files.left_in_place |
Either mechanism restores files, and neither rewinds the checkout's git history. In a worktree the revert commit keeps the two in step: the replayed node meets a tree that no longer holds its output, and git log says why. In place, a commit the run made after the pivot stays while its files go back, so a node whose product IS a commit — its own, or that of a subbot child working in the same tree (one the parent did not declare isolated:, or an isolated one wherever worktree: auto could give it none: a store with no filesystem root, an adopted sandbox, a child declaring worktree: none, a workspace git cannot serve) — replays against a tree that already carries it, git commit reports nothing to commit and the node fails. What makes such a run rewindable is OWNING a worktree: the run an operator launches does wherever worktree: auto — the declaration default — can give it one, a subbot child working in its parent's tree does not, whatever its own .bot declares — so rewind the run that owns the tree, or make the committing node tolerate its own replay.
Out-of-tree scratch is untouched by any scope: ${PROJECT_SCRATCH_DIR} resolves outside the workspace, so a campaign bot's verify.sh / verify.log survives a rewind. That is correct — but it does mean a replayed verify gate can meet a stale script.
How much comes back: --restore-scope
On an in-place run the workspace is yours, and a rewind is launched right after you edit it — --auto derives the pivot by diffing that very edit. So at the moment of every rewind the tree holds human work by construction, and forcing the whole tree back would revert it. --restore-scope is the dial:
produced(default in place) — only the paths this run is recorded to have changed after the pivot started. That set is the union of the diffs between consecutive boundaries in[pre:<pivot> … the run's last boundary], not a diff of the two endpoints: a file one node rewrote and a later node put back is identical at both ends while demonstrably being a file this run writes to, and excluding it would let a third, uncaptured write survive.full(default in a worktree) — every versioned path in the snapshot. "Versioned", not "the whole disk": ignored, protected and never-captured paths are still untouched.none— nothing; the node replays against the tree as it stands.
produced is refused, not silently widened, on a worktree: auto run: git is the mechanism there and it reverts the whole tree or none of it, so the rewind says so and leaves the workspace alone rather than substituting the maximal blast radius for a request to narrow it.
What iterion cannot attribute, it reports. A node that dies before its boundary is written and an operator editing in another terminal leave identical evidence, so the rewind names both sets instead of guessing: files.overwritten (in-scope paths whose disk content matched neither side of the recorded range — work that arrived after the run stopped recording it) and files.left_in_place (changed since the run's last boundary, not restored). Both also land in the run_rewound event, so an API- or agent-driven rewind leaves the same audit trail as a CLI one.
To keep that honest the engine writes a fail:<node>:<iter> boundary when a node's execution does not complete — a failure, an interruption, an operator cancel — a pause:<node>:<iter> one when it parks on a human gate, and a resume:<node>:<iter> one when it picks back up into a node that already has a pre: boundary (rather than overwriting it, which used to redefine "what this node started from" as "whatever is on disk now"). Without them a run that stopped inside a node would have nothing recorded after the state that node started from (pre: is an alias and does not advance the chain head), the scope would be empty, and the node's own debris would survive its rewind. When the scope is empty the rewind says so and restores nothing, rather than reporting a success it did not perform.
pause:/fail: open, and resume: closes, the one interval in which nothing of the run is executing — and the scope excludes it. A file that appears while a run waits for you, or between a failure and the resume you trigger after triaging it, came from your editor, not from a node; without the exclusion the capture the resume takes folds it into that node's apparent output and the rewind deletes it. That is the only place authorship is decidable. An edit made while a node is running is not, and lands in files.overwritten if the restore takes it.
Human and agent interaction resume
For an ordinary human node, answers become that node's typed output and edge selection continues from it. Review gates use a dedicated multi-turn path and can re-pause on the same interaction while preserving the dialogue.
When an agent or judge calls ask_user, the node has not completed and is re-invoked after the answer:
- the in-process
clawbackend persists the opaque conversation and pending tool-use ID, then appends the answer as the matching tool result; - CLI-agent backends reuse their session ID when supported; the runtime also injects the prior question and answer into the prompt as a compatibility fallback.
This keeps multi-turn interaction attached to the same node rather than mistaking the pause for a completed human node.
Recovery pauses re-execute the node
A recovery pause is written by the recovery dispatcher for a node whose execution failed (RecoveryPauseForHuman: the provider rejected the credential, the budget is exhausted, a policy asked for a human). Its synthetic question (acknowledge_recovery, plus recovery_code and last_error) promises a retry, and the resume delivers one: the answer is recorded on the interaction (kind: "recovery") as the audit trail of what was fixed, and the node re-executes from its own dispatch — exactly as a failed_resumable resume restarts it — before any successor runs. The acknowledgement never becomes the node's output. The checkpoint carries the marker (recovery_pause / recovery_code), and the run_resumed event names the retry: {resumed_from: "recovery_pause", restart_node, recovery_code}. The per-(node, code) attempt counters survive the pause, so a second identical failure is judged as the second attempt. A budget pause resumed without a raised cap is refused up front with the real cause, the same pre-flight the failure path applies — raise it with the --max-* flags.
Distinguish it from a delegate pause: an agent that asks (ask_user, _needs_interaction) has not failed; it re-invokes mid-conversation with the answer merged into its input (above), and its checkpoint carries the backend session, not the recovery marker.
Dispatcher and last_run
A run is durable across a studio or dispatcher restart. If the board card still points at that run (issue.last_run) and the status is live or waiting, the dispatcher does not mint a new run id.
last_run | Dispatcher action |
|---|---|
paused_waiting_human / paused_operator | Re-park the card in awaiting_input (dispatcher-owned runs). No auto-resume (no answers). |
running | Hold while the owner lives. An orphaned run (no live lock, past the grace window) is promoted to failed_resumable / failed first — never a sibling from the workflow entry. |
queued | Hold without a lock probe: pipeline-queued runs have no lock owner until their concurrency slot opens. |
failed_resumable / cancelled | Engine.Resume on the same id. |
finished | Fresh run allowed — dragging the card back to ready is the re-queue gesture. |
none, or hard failed + ticket explicitly back in ready | Fresh run. |
iterion resume --force (or the studio force-resume) continues this run after a bot edit. If that resume parks again on a later human node, the next dispatcher tick re-parks the same card — it does not start the workflow over. To throw the work away, finish the run or let it fail hard: a cancelled run is resumed from its checkpoint, not replaced.
See dispatcher.
Failure behavior
Most runtime errors use the checkpoint-aware failure path: LLM/delegate errors, schema validation, edge/routing failures, budget and timeout errors, fan-out failures, and resumable sandbox startup failures. A recovery policy may retry, repair, or ask a human before the final status is written.
What a resume rebuilds
A resume does not inherit the previous attempt's environment — the sandbox went away with the process that owned it. Before the checkpoint node re-executes, the engine re-runs the whole setup: it re-mirrors the bundle's skills and plugin contributions into .claude/ (so a bundle upgraded between the two attempts takes effect), re-resolves the run's vars from its persisted inputs, and starts a fresh sandbox — a new container or pod, going through the same phases as a launch (image pull / pod Ready, workspace copy, git fixup, post_create). On the kubernetes driver a resume force-deletes and re-creates the pod, under the scheduling policy of the runner that claims it.
Which means the setup timeouts apply identically to a resumed run — the point where iterion used to differ, and where the operator noticed it: the same stall that parked a launched run with a typed cause left a resumed one parked with none. The caps and the environment override are in sandbox.
What the resume does NOT redo: the nodes already recorded in the checkpoint. See what the checkpoint preserves.
Sandbox startup — which failures are resumable
A failure at sandbox start happens before the first node runs, so it is classified from the DRIVER's typed error — or the engine's own, for the attachments — rather than assumed either way. The resumable codes are persisted on failed_resumable, so iterion resume and the cloud runner's redelivery pick them up — the CLI's --auto-resume loop does not resume them; anything else stays terminal failed, because a redelivery would re-hit it identically and only spend a pod per attempt.
| Failure | Code | Status |
|---|---|---|
A bounded setup phase that RAN and stalled (workspace copy, git fixup, post_create) | SANDBOX_SETUP_TIMEOUT | failed_resumable — a fresh pod routinely clears the stall |
The pod is still Pending past the deadline, unscheduled (Unschedulable, Insufficient cpu: the fleet is at its request ceiling) | SANDBOX_CAPACITY | failed_resumable — the run executed nothing; a later attempt re-places it |
The pod is still Pending, scheduled, its node not having started the container (ContainerCreating, PodInitializing, or no container status reported yet) | SANDBOX_CAPACITY | failed_resumable — same: Pending IS the API's guarantee that no container was created |
A broken image reference (ErrImagePull, ImagePullBackOff, InvalidImageName) | — | failed — every pod re-hits it; the operator fixes the reference. Overrides the phase: such a pod is Pending too |
An invalid spec (CreateContainerConfigError, CreateContainerError) or a crash-looping container | — | failed |
The pod reached Running (or Unknown) but never Ready | — | failed — a container came up; nothing says the run did nothing |
| The pod could not be inspected at all (RBAC, apiserver blip) | — | failed — no evidence, nothing claimed |
No driver can honour an EXPLICIT sandbox: { mode: inline, … } (no runtime on the host / no usable cluster) | SANDBOX_DRIVER_UNAVAILABLE | failed — every redelivery reaches the same absent runtime, so pkg/retrypolicy classes it Deterministic and the runner acks. sandbox: auto does not land here: it degrades to an unsandboxed run with a sandbox_skipped event (#1425) |
| The run holds attachments and the driver can neither bind-mount their directory into the container nor copy it in | ATTACHMENTS_UNDELIVERABLE | failed — a redelivery reaches the same driver; run the bot on one that delivers files, or launch it without attachments (#2086) |
| On a store with no filesystem root, the store did not hand the run's attachments over before the sandbox (a bucket blip) | ATTACHMENTS_UNFETCHABLE | failed_resumable — nothing of this attempt executed; a redelivery fetches them again after two minutes (#2086) |
The resumable codes are re-offered by the cloud runner on a DELAY rather than at once: 2 minutes for SANDBOX_SETUP_TIMEOUT (the stall is usually infrastructure catching its breath, and a bare re-offer would burn the whole delivery budget as back-to-back pods — 8 × a 15-minute phase ≈ 2 hours — with nothing on the run's timeline in between), longer for SANDBOX_CAPACITY (the cure is a cluster autoscaler adding a node, not the pod retrying), and 2 minutes for ATTACHMENTS_UNFETCHABLE (a storage blip clears on the store's side; a bare re-offer would spend every delivery on the same blip). Each redelivery is recorded on the run's timeline. A condition that persists through every permitted delivery ends parked on the DLQ like any other repeated failure — announced, rather than re-offered forever. A one-off stall needs no operator action at all; iterion resume on the parked run is the same recovery, taken by hand, when you would rather not wait for the redelivery.
The deadlines are per phase: the pod-Ready wait is ITERION_SANDBOX_K8S_POD_READY_TIMEOUT (see sandbox), the copy and the post_create snippet have their own — see setup phases and their timeouts. Raise the one the error names: the message says which phase timed out, how long it ran and what its budget was.
Cancellation saves state with a detached, bounded store context so Ctrl-C can still persist after the execution context is cancelled. If checkpoint writing itself fails, the runtime logs the loss and can fall back to non-resumable failure; resume never reconstructs execution state by replaying events.
Implementation map
| File | Responsibility |
|---|---|
pkg/store/run.go | Run statuses and the complete Checkpoint schema. |
pkg/store/store_run.go | Atomic pause/failure/status persistence. |
pkg/runtime/checkpoint.go | Checkpoint construction and budget restoration. |
pkg/runtime/run_failure.go | Resumable failure and cancellation paths. |
pkg/runtime/resume.go | Status dispatch, state/environment restoration, answer handling, and restart execution. |
pkg/runtime/pause.go | Operator, daily-cap and budget-cap checkpointed pauses. |
pkg/cli/resume.go | CLI admission, source reopening, overrides, locking, and stale-run takeover. |
pkg/runview/rewind.go | In-place rewind: pivot validation, downstream computation, checkpoint mutation, artifact tombstones. |
pkg/runview/rewind_auto.go | --auto targeting: declaration-granular source diff → impacted nodes → earliest pivot. |
See also persisted formats, human interaction, permissions, and merge policy.