Queue schema rollout runbook

How to ship a queue.RunMessage schema bump (pkg/queue/types.go, SchemaVersion) without executing a payload with silently dropped semantics, and without losing runs while the server and runner fleets run mixed versions. Read it before every schema bump.

This runbook is mandatory for every schema bump, and ordering is the first decision it makes for you: server-first by default, because that puts any parked run on the side a resume recovers as soon as the fleet is upgraded. Runner-first is an optimization with a precondition — see Deploy ordering — and the drain/resume paths below are what you fall back to when neither ordering can spare the queue.

A parked run is resumed, never replayed. Every park marks its run failed_resumable with DLQ_PARKED, a code a runner drops on admission: POST /api/admin/dlq/{seq}/replay refuses it (409) and keeps the copy. A resume (POST /api/runs/{id}/resume, iterion remote runs resume) re-publishes the run at the server's current schema version; discard the parked copy afterwards.

The stop-new-launches switch is maintenance.fence_launches; see admission maintenance for the dated super-admin API, existing-run resume exemption and local iterion contracts probe. This switch does not drain already admitted work.

Wire compatibility policy

• Explicit compatibility window. A consumer accepts only [MinSchemaVersion, SchemaVersion] and rejects anything outside it in both directions. The current v28 consumer accepts v27 and v28: a v27 message predates the wire-carried bot_source_tier, and the run document its publisher persisted already carries the stamp — the runner re-reads the tier from there, so the #1959 floor never silently drops.
• Server first by default. Both orders can park a message, and a resume recovers the run either way — but not at the same moment. Old runners rejecting the new version park runs a resume recovers as soon as the fleet is upgraded; new runners rejecting a version below their MinSchemaVersion park runs a resume recovers only once the server publishes a version they accept. Server-first keeps the risk on the side recovered first. Roll the runners first only when nothing below Min(new) can still be queued — automatic when the bump leaves MinSchemaVersion alone, and otherwise not worth asserting from a gauge. The precondition and the measured case are in Deploy ordering below.
• Additive field whose omission changes operator intent = breaking change. If a new field carries a decision the caller explicitly made (budget caps, skills, auto-memory, loop guard, model pins…), a stale runner that silently falls back from it is a correctness bug, not a cosmetic one. Such a field REQUIRES a schema bump. History: v4 Budget, v5 Contributions, v6 AutoMemory, v7 LoopBudgetGuard — each exists because dropping it would quietly re-make the operator's choice on the pod.
• Known historical debt. ModelOverrides was added during v7 without a bump (commit 427a9f44e), so an earlier v7 runner could accept and ignore those pins. The later v8 bump cannot retroactively repair that window; this rule records the lesson so the next intent-bearing field ships atomically.
• The exemption, and how to tell. A field is exempt only when a runner that ignores it cannot fail OPEN. BudgetOverrides.cap_imposed was added inside v8 with no bump on exactly that ground: it carries no operator choice (the publisher derives it from a clamp — see credential-pool.md), and the only consumer that would act on it, the runtime's budget exit grace, shipped in the same commit. A runner old enough to drop the field is old enough to have no grace to refuse, so ignoring it costs nothing. Apply the test in that direction: not "is the field new?" but "what does a runner that never sees it do instead, and is that safe?"
• Until the bump ships, reject — never drop. A launch that carries a field the current wire version cannot transport must fail loudly at publish time. Once the carrier and version bump ship together, the rejection can be removed. Schema v8's Supervisors kill switch is the reference transition.

What a mixed fleet does to a mismatched message

Since #481, a version mismatch is transient and recoverable:

1. The consumer Naks with a 30s delay (nats.SchemaMismatchNakDelay, configurable with ITERION_RUNNER_SCHEMA_MISMATCH_DELAY or runner.schema_mismatch_delay), so the MaxDeliver budget (default 8) stretches over ~4 minutes of wall clock instead of being burned in seconds — enough for a rolling restart of the runner Deployment to schedule a pod that speaks the new version.
2. If the budget is exhausted anyway, the consumer parks the payload verbatim on the DLQ, Terms the queue entry, and flips the run document from queued to failed_resumable (DLQ_PARKED) with an actionable error pointing at this runbook. The run is never silently dropped and never left queued with no recovery path.
3. Once the fleet accepts the version the server publishes, a resume of the run re-publishes it at that version and a runner picks it up. A DLQ replay of the parked bytes is refused: a runner drops a DLQ_PARKED run's redelivery on admission.

v27 absolute budget ceilings

BudgetOverrides.donor_max_cost_usd carries the donor's cumulative absolute cost ceiling independently of the declared cap. The publisher writes it; the runner records it alongside the platform ceilings already read from ITERION_CLOUD_MAX_*. Grace and live raises cannot exceed either authority.

Both SchemaVersion and MinSchemaVersion are 27. Earlier messages may carry a $9.50 ask under a $10 donation without even cap_imposed; neither that missing marker nor the missing amount admits a safe migration. Old payloads therefore receive the ordinary typed schema refusal. A resume re-publishes the run using a newly resolved grant and the explicit ceiling.

Use server-first, with Path A (drain before cutover) or Path B (park and resume). Runner-first without a drain is unsafe because the minimum moves. The historical versions below remain in their original chronological order; their compatibility windows describe those releases, not the v27 consumer.

v28 bot-source tier (isolation floor)

bot_source_tier mirrors the run document's stamp of which bot-resolution tier served the launch (team / platform / baked) so the runner applies the #1959 operator isolation floor — neutralizing a non-operator source's sandbox: none — before any document read, and a subbot's child message (a struct copy) inherits it. Breaking by the additive-field rule: a stale v27 runner must reject the message rather than honour a team-authored opt-out the floor exists to neutralize.

MinSchemaVersion stays 27: every v27 publisher already stamps BotSourceTier on the run document, so a v28 runner consuming a v27 message re-reads the tier from the document (the floor's second carrier) instead of dropping the decision — the one direction in which ignoring the field cannot fail open. A run whose document predates the stamp carries an empty tier and reads as the operator's own launch: unfloored, as it was.

Use server-first; old runners park the v28 message replayably and a resume recovers once the fleet is upgraded.

v12 runner-epoch bootstrap

Schema v12 adds runner_epoch. A stale v11 runner must reject v12; if it ignored the field it could execute work after its generation was superseded. A v12 runner treats v10/v11 messages with no field as epoch 0.

Do not combine the schema bump and first non-zero epoch:

1. Release A: deploy v12 with config.rollout.runnerEpoch: 0. v11 → v12 leaves MinSchemaVersion at 10, so the runner-first precondition holds: roll the runners, then the server, and neither Path A nor Path B is needed — see Deploy ordering below. (Rolling server-first here is also correct; the paths are only dispensable if you actually take the runner-first order.) Verify all server and runner probes show epoch: 0. (Done in prod on 2026-09-02 exactly that way: runner fleet, then server, DLQ untouched throughout — the measurement at the end of Deploy ordering.)
2. Release B: set the epoch to 1. New publishers stamp epoch 1. Old v12 runners delayed-Nak those messages; new runners accept both epoch 0 and 1.

Epoch refusals use their own ITERION_RUNNER_EPOCH_MISMATCH_DELAY (default 2m), but share the final DLQ + failed_resumable disposition. Both mismatch delays feed RedeliveryWindow() so the orphan sweeper cannot race a message legitimately waiting for a compatible runner.

The persistent rollout high-water mark makes an epoch decrease non-ready and blocks run publication/consumption. A rollback is therefore a roll-forward: re-release the previous fence-aware image with an epoch greater than the current high-water mark. Never restore a lower-epoch Helm revision directly.

This section describes runner builds from v8 onward. A pre-#481 (v7 or older) runner answers a mismatch with an immediate bare Nak(): it burns the MaxDeliver budget in seconds, JetStream drops the message, and nothing is parked to replay. That asymmetry drives the path choice below.

Deploy ordering: which side rolls first

Both orders can park a message. They differ in whether the park is recoverable, and that — not "which side rejects less" — is what decides.

roll first	what gets rejected	recovery
server	old runners reject the new vN+1	recovered at once: as soon as every runner is new, resuming the run re-publishes it at vN+1 and it is accepted
runners	new runners reject anything still queued below Min(new)	recovered only after the server rolls: a resume re-publishes at the server's version, which the new fleet rejects while the server is old (Path B step 5) — recovery is a per-run resume once the server is new, plus a DLQ delete

Server-first is therefore the safe default, for that reason and not a historical one: it puts the parking risk on the side that can be undone. The compatibility window changed how often a mismatch happens; it did not change which mismatch is recoverable.

Runner-first is an optimization, valid under one condition: that nothing below Min(new runner) can still be in the stream. Then new runners admit everything — including what the old server is still publishing — and nothing is rejected in either direction, so neither Path A nor Path B is needed.

Deciding which you are in

MinSchemaVersion unchanged by the bump ⇒ runner-first is safe with no further check: the new window is a superset of the old one, so no resident message can fall below it.

MinSchemaVersion raised ⇒ do not attempt runner-first on an observation. Take server-first, or drain first (Path A) — draining makes the queue empty instead of trying to prove that it is, which is the only form of the check worth trusting when the failure is unrecoverable.

iterion_nats_pending_messages looks like the oracle and is not: pollPending leaves the gauge at its last successful value when the poll errors, warning only after five consecutive failures (pkg/runner/loop_nats.go), so a queue that filled while polling was broken still reads 0 — stale in exactly the dangerous direction.

Min does move. Across the versions that have one:

v9: 8 → v10: 9 → v11: 10 → v12: 10

two of the three transitions raised it, each by exactly one — to equal the previous SchemaVersion. That is the shape that fools the naive test Min(new) <= SchemaVersion(old server): it passes, while a message one version older still queued is rejected unrecoverably.

For what it is worth, "still in the stream" is narrower than it sounds: the runs stream is Retention: WorkQueuePolicy (pkg/queue/nats/nats.go), so an ACKed message is removed immediately and MaxAge (24h) bounds only how long an unacked one lingers. Residue is what is pending or still being redelivered, not everything published in the last day. That makes the residue small — it does not make it observable.

The cost of server-first, stated honestly

A vN+1 server publishing into a vN-only fleet gets every delivery delayed-Naked until a new-version runner is Ready. That is a race, not a certainty of loss: the 30s delay exists precisely to stretch the 8-delivery budget over ~4 minutes and cover a rolling restart (see What a mixed fleet does above and the constant's own justification), and under the chart's maxSurge: 100% / maxUnavailable: 0 the first new runner is usually Ready early in the roll. What races the budget is time-to-first-Ready-new-runner, not the full roll — which under drainMode: complete can take hours.

A fleet slow to become Ready spends the budget and parks the run failed_resumable — recoverably, by a resume once a new runner is Ready. Runner-first removes that race; it is worth taking when the condition above holds, and it is what the v11 → v12 cutover used (see the measurement below).

Sequencing the two Deployments

A plain helm upgrade rolls both together: the server and runner Deployments resolve to the same image tag (include "iterion.image" .). Use the chart's per-runner image override to sequence the phases — here in runner-first order:

# Phase 1 — pin the runners to the NEW image, leaving the server on the old tag.
helm upgrade iterion ./charts/iterion \
  --set image.tag=<old-tag> \
  --set runner.image=ghcr.io/socialgouv/iterion:<new-tag>

# Phase 2 — once every runner is Ready on <new-tag>, roll the server.
helm upgrade iterion ./charts/iterion --set image.tag=<new-tag>

The snippet above is the runner-first order — the optimization, not the default. Path A and Path B below are server-first, and that is a different command, not a swap of the two values (image.tag takes a tag, runner.image a full reference, so exchanging them renders an invalid image):

# SERVER-FIRST — Path A step 3 and Path B step 1 only.
# Phase 1 — roll the server, holding the runners on the OLD image.
helm upgrade iterion ./charts/iterion \
  --set image.tag=<new-tag> \
  --set runner.image=ghcr.io/socialgouv/iterion:<old-tag>

# Phase 2 — once the server is Ready on <new-tag>, release the runners.
helm upgrade iterion ./charts/iterion --set image.tag=<new-tag>

Getting that backwards on Path B is the destructive direction: it rolls the runners into a queue full of vN messages they reject, and per Path B step 5 those parked messages are unreplayable — recovery is a per-run resume plus a DLQ delete.

Both snippets assume the install moves its image via image.tag. If your values pin the image some other way, --set image.tag= may not move it at all — check that the rendered PodTemplate actually changed before treating a phase as done. Note runner.image, when set in a values file, already overrides iterion.image for the runner container, so phase 1's --set runner.image= must name a full reference and phase 2 must clear or advance it rather than rely on the shared tag.

If your deploy pipeline sequences Deployments itself, use it instead — the requirement is only that the two Deployments do not roll together.

Both snippets sequence the IMAGE only. A release that also moves config.rollout.runnerEpoch does not get sequenced by them: that is a single value rendered as a literal ITERION_RUNNER_EPOCH into BOTH PodTemplates (server-deployment.yaml and runner-deployment.yaml), so phase 1 rolls both Deployments. Move the epoch in its own release — which is what the two-release Release A / Release B protocol above already prescribes — not in the same one as a schema bump.

Measured on the v11 → v12 cutover (2026-09-02, prod: 12 runner pods, 3 server pods, live traffic including a run executing across the runner roll). Runner-first left the DLQ byte-identical — depth 30, last_seq 220, newest message hours older than the rollout — with no iterion_runner_admission_rejected_total sample anywhere in the fleet.

Rollout procedure for a schema bump vN → vN+1

These two paths accompany the server-first default. Choose one of them: ordering alone is not a third option, because a server-first roll still parks the new version on the old fleet if no upgraded runner becomes Ready inside the delivery budget — Path A avoids that by draining first, Path B accepts it and resumes the parked runs afterwards.

Both become unnecessary only when you actually roll runner-first AND its precondition holds — nothing below Min(new runner) can still be queued, which is automatic when the bump leaves MinSchemaVersion alone. Then nothing is rejected in either direction, so there is nothing to drain or resume. The precondition alone does not exempt you: a server-first roll still needs one of these paths whatever the precondition says. See Deploy ordering above, and do not infer the precondition from the old server's version alone. Path A's "recommended default" below is scoped to the bumps that still need a path.

Path A — drained queue before cutover

Mandatory for v7 → v8 (see below); the recommended default for every later bump as well.

1. Stop new launches (maintenance window) or accept that launches during the window follow Path B — which, for v7 → v8, they cannot: keep the window.
2. Wait for the queue to drain: iterion_nats_pending_messages = 0 and no queued runs older than the AckWait window.
3. Deploy the server (vN+1), then roll the runners (vN+1) — the SERVER-FIRST snippet above, not the runner-first one. (If step 1 held a real maintenance window and step 2 reached zero, ordering is moot here: nothing is left for either side to reject — and a drained queue is exactly the precondition that also makes runner-first safe, so either order works. It is not moot when step 1 was skipped and launches continued, which the step explicitly allows — so follow the order regardless.)
4. Sanity-check: one launch end-to-end, DLQ depth 0.

Path B — DLQ identification + resume after cutover

Valid only from v8 → v9 onward. Path B leans on the delayed Nak, the DLQ park and the status flip — and those ship with schema v8. For the v7 → v8 cutover itself the outgoing runners are pre-#481 builds: they bare- Nak() a v8 message, burn its MaxDeliver budget in seconds, and leave an empty DLQ and a run stuck queued — the exact loss #481 closes. Do not run Path B for v7 → v8; use Path A.

1. Deploy the server (vN+1) first, then roll the runners — the SERVER-FIRST snippet above, not the runner-first one. This is the path where the wrong order parks messages that step 5 cannot replay. During the window, stale runners hold vN+1 messages via delayed Naks; after MaxDeliver they park them on the DLQ and flip the runs to failed_resumable.

2. Once all runners run vN+1, list the DLQ and identify the parked messages from the transition — the Iterion-DLQ-Reason header reads queue: schema version: N+1 unsupported (want N):

   curl "https://iterion.example.com/api/admin/dlq?limit=200"
   curl "https://iterion.example.com/api/admin/dlq/$SEQ"   # peek: check the reason + run_id

3. Resume each parked run: the publisher re-queues it at vN+1 and a vN+1 runner picks it up. Do not replay the parked message — the park marked the run DLQ_PARKED, which a runner drops on admission, so the replay endpoint answers 409 and keeps the copy:

   curl -X POST "https://iterion.example.com/api/runs/$RUN_ID/resume"

4. Verify each resumed run leaves failed_resumable for running, then discard its parked copy (curl -X DELETE "https://iterion.example.com/api/admin/dlq/$SEQ") and confirm the DLQ returns to its pre-rollout depth.

5. The reverse direction does not replay. If the queue still held vN messages when the vN+1 runners came up, they parked with reason N unsupported (want N+1) — and a replay re-publishes those exact bytes, which the vN+1 fleet rejects identically and re-parks: an operator replay loop that never succeeds. Recover those runs by resuming them instead (POST /api/runs/{id}/resume: the publisher stamps the CURRENT SchemaVersion and a resume-specific Nats-Msg-Id, so JetStream cannot mistake it for the original launch inside its five-minute deduplication window), then discard the stale parked copies with DELETE /api/admin/dlq/$SEQ.

Checklist: v7 → v8 (Supervisors and #481 safety)

The v8 bump belongs to the launch-time Supervisors kill switch. The v8 wire also carries model_overrides, but those had already entered v7 without a bump (the known debt above); do not read this transition as retroactively making every v7 build safe for model pins. The rollout preconditions are:

• [x] Delayed mismatch Nak, final DLQ park and actionable status flip.
• [x] ModelOverrides on queue.RunMessage, set by the publisher and applied by the runner executor; resumes preserve the same pins.
• [x] SchemaVersion = 8 for Supervisors, so a stale v7 runner rejects a v8 payload instead of re-deciding the operator's supervisor kill switch.
• [x] The live-JetStream mixed-fleet integration test covers both version directions and the recovery paths.
• [ ] Operators roll out per Path A (drained queue) — Path B is not an option for v7 → v8: the outgoing v7 runners predate the delayed-Nak / DLQ-park mechanics this release introduces, so only a drained queue protects in-flight messages during this specific cutover.

Checklist: v13 → v14 (execution context admission)

v14 carries the resolved execution_context contract. The publisher also persists it on the queued run document; the runner compares the wire copy with that document before attachments, workspace setup or a model call. A stale v13 runner must reject v14 rather than silently run without admission context.

• [x] ExecutionContext is carried by RunMessage and the queued run.
• [x] Schema version is bumped to 14; MinSchemaVersion remains 10 for the existing rolling-accept window.
• [x] Local, CLI, cloud and nested subbot engines run the same pre-model gate.
• [ ] Roll out the server publisher before runners, then observe admission decisions (allowed, report_only, denied) before enabling enforce.

v15: model effort and run-level permission overrides

Schema v15 adds RunMessage.permission. This is intentionally a versioned change even though the JSON field is additive: omission changes explicit run intent. A stale runner would otherwise accept an operator's deny launch and execute with the workflow/node default, silently weakening the tool gate. The publisher persists the launch override on the run document and republishes it on resume; the runner applies it at run precedence, above node and workflow declarations. MinSchemaVersion remains 10 so new runners can drain older queued messages whose permission field was necessarily absent.

This version also carries ModelOverride.Effort. The assistant branch's former v10/v11 additions are renumbered to v15 when integrated with main's v14 contract, so existing main runners reject choices they cannot enforce.

Checklist: v17 → v18 (loop cap expressions)

v18 carries AST loop caps that may contain expressions over vars and outputs. An older runner cannot compile that meaning of max_iterations_expr; it must reject the envelope before admission. Every new publication uses v18, including runs whose loop caps are still literals or templates.

• [x] Schema version is 18 and MinSchemaVersion remains 10.
• [x] The AST JSON retains the authored cap for runner-side compilation.
• [ ] Roll out v18-capable runners first, then the server publisher. This order is safe because the minimum accepted version remains 10; the new runners can still consume all messages accepted by the outgoing fleet.
• [ ] Confirm the fleet reports support before publishing v18. A rollback to the old publisher emits v17, which the new runners continue to accept.

Checklist: v18 → v19 (literal template delimiters)

v19 carries the canonical {{"{{"}} source form. It is rendered once as literal opening braces in prompts, tools and runtime mappings, after compile-time includes and group specialization. An older renderer cannot implement these semantics and must reject the envelope before AST decoding. Every new publication uses v19, including workflows without a literal delimiter.

• [x] Schema version is 19 and MinSchemaVersion remains 10.
• [x] The AST preserves the canonical source form through transport; all final rendering sites understand the literal delimiter.
• [x] A real v19 message from a compiled literal prompt was refused by the actual v18 reader at bd3906a725ee4b24fd7669e58e57a0556dbb2ce1 before AST decoding: queue: schema version: 19 unsupported (want 10–18).
• [ ] Roll out v19-capable runners first, then the server publisher. The minimum remains 10, so the new window includes everything the outgoing fleet accepts and the runner-first precondition above holds.
• [ ] Confirm fleet support before publishing v19. Rolling the publisher back emits v18, which the new runners still accept. Keep the new runners until no v19 messages remain queued or resumable; an old runner cannot execute those literals safely.

Checklist: connector packages — NOT YET BUMPED

The bump will carry the connector packages a run resolved at launch (ConnectorRefs), so a runner materialises the same immutable package the publisher compiled against. See ADR-098.

ConnectorRefs is not on the wire yet, and this entry exists so the ordering is decided before it is rather than discovered during a deploy. The trap to avoid is specific: SchemaVersion is stamped on every message, so the moment the server emits the new version, a digest-pinned fleet still on the current one rejects all new messages — connector runs and ordinary ones alike. That parks the whole queue, not one feature.

• [ ] ConnectorRefs added to RunMessage, carrying an immutable content-addressed package reference (never the package bytes: Forgejo is 651 KiB and GitHub 4.8 MiB, so the payload goes out of band beside the IR's own offload — ADR-075).
• [ ] MinSchemaVersion stays 10. The addition is additive from a new runner's point of view, so new runners keep consuming the current version for the whole window.
• [ ] Runners first, then the server — the inverse of the default order, and legitimately so per the rule above: nothing below MinSchemaVersion(new) exists, so rolling runners first parks nothing. The runner is digest-pinned, so this is an explicit values bump, not a restart.
• [ ] Only after the runner fleet reports the new version capable: let the server publish it. A rollback then re-publishes the previous version, which the new runners still accept.
• [ ] A package referenced by a queued or resumable run is retained. A reference plus a drift guard cannot retrieve a package replaced after the message was published — that turns a resumable run into a permanently failed one, which is why the packages are content-addressed rather than fetched-by-slug-and-version.

If something went wrong

• Runs stuck queued after a rollout: check the DLQ (they parked there if this runbook's mechanics were deployed) and the orphan sweeper; resume per Path B.
• A run executed with dropped semantics (e.g. wrong model): that means a field crossed the wire without a schema bump. Treat as an incident, add the field to the additive-field rule above, and bump.