Platform bot overrides — DB-backed bots, no rollout

The bot catalog (bots/) is baked into the server and runner images, so changing a native bot historically cost a build + image publish + rollout. Platform bot overrides move that iteration loop into the database: a super-admin pushes a bundle, and from the next launch every tenant and every launch surface (studio, webhooks, schedules, board dispatch, triggers, resume) runs the pushed version. Deleting the override reverts to the baked catalog. Nothing restarts.

The same change also runs runtime-mutable platform settings for the webhook role bots and the sandbox default image (see below).

The iteration loop

# edit bots/review-pr/ in a local checkout, then:
iterion remote admin bots push bots/review-pr
# → next launch of review-pr runs the pushed bundle (prompt, skills, devbox, manifest)

iterion remote admin bots            # list overrides (slug, version, digest)
iterion remote admin bots show review-pr
iterion remote admin bots pull review-pr --out /tmp/review-pr
iterion remote admin bots fork review-pr   # seed the override from the baked bundle
iterion remote admin bots rm review-pr     # revert to the baked catalog

push compiles the bundle server-side before persisting — a bot that does not compile is rejected with the diagnostics, never left to fail at launch. The response carries non-fatal warnings (see Known gaps). The studio surface is Admin → Bot overrides (list, digest, revert) plus a platform override badge on the /bots gallery card; the push loop itself is CLI-first.

How it works

Diagram: Bot resolution — team row, platform override, baked catalog — the cascade below; the step table, the guards and the collections it touches are on that page.

• Storage: an override is an ordinary pkg/botsource row under the reserved sentinel tenant platform: — same collection, validation and compile check as team-authored bots (the pattern platform LLM credentials established). Hard limits: 6 MiB / 512 files per bundle.

• Resolution precedence (most specific wins): team botsource → platform botsource → baked catalog FS, on every launch surface — the studio button, the board dispatcher, the trigger spine, a cloud schedule and an inbound webhook — because each launcher knows the team it launches for (the card's, the subscription's, the schedule's, the webhook token's). A team that forks a bot runs its fork on its own automation, and every launch records which tier served it (bot_source_tier) and shows it — the run API returns it on the run header, the studio badges a team bot / platform override run beside its bot chip, and Launched with names the tier in full. A run that recorded no tier shows nothing rather than defaulting to baked: an unresolved tier reading as a working one is the failure mode this whole section exists for. Enforced by the central resolver in pkg/server/bot_resolver.go and its static sweep tests — one of which fails a launch site that hardcodes an empty team.

• Spelling. All three tiers tolerate the same variants (feature_dev / Feature-Dev / "feature dev" → feature-dev), because a board card, an agent's set_bot and a hand-written subscription all carry operator-typed names. A slug may legitimately be stored with _ too, so the team tier compares both sides normalized.

• A metadata read matches the tier that will SERVE the launch, or the two disagree in silence — a fork that renames a consumes: var gets an empty seed, a fork that renames its /command stamps the operator's text under a var the running bundle never declared, and neither is an error. Which tenant a lane passes depends on what it describes:

  • a delivery about to launch reads the launching team's tier — the webhook hand-off seeds, the gate-var defaults, the retry policy, the command-routing discovery and the labeled-issue route, the converse-bot existence probe, the config-share surface, the bot home's "enable this trigger", and the forge auto-provisioning lookups (which build the webhook's CommandMap and its event subscription). The forms are effectiveEntriesForTeam / effectiveFindByNameForTeam / botManifestFor / botExistsForTeam / entryOriginFor, all over the same row resolution the launch uses, so an operator-typed spelling reaches both;
  • a run that already launched reads that run's ownbot_source_tenant — the pause-notice role and the hand-off PRODUCER set (teamBotManifest). The ambient tenant is the wrong question there: a team's own run may still have been served by the platform or baked tier, and the run records which.

  The two FS-catalog write paths (PUT /api/v1/bots/{name} and its /overlay) refuse a stored bot with 409 instead of editing the bundle every tenant shares. Deliberately tenant-free: the native board's comment dispatcher (one local store, no tenancy) and the platform + baked floor each team-aware form falls through to. A static sweep (bot_resolver_sweep_test.go) fails a new tenant-free metadata read that is not declared with its reason.

• Launch: the server resolves the override ONCE, materializes it to a temp dir, and compiles against it (prompts/ participate in IR and the workflow hash). The queue message's bot_bundle ref carries the frozen bundle collection and its sha256 digest — inline, or as an IR blob reference when the envelope would not fit the payload limit; the runner materializes those bytes after the digest check and attaches them as the run's bundle instead of the stale baked one — full fidelity: manifest, skills/, prompts/, devbox.json. A ref carrying no snapshot is the legacy path: there the runner fetches the row from Mongo and holds its version to the ref's.

• Version drift: a push racing an in-flight launch cannot repoint it — the snapshot the ref carries is what runs. Only a legacy ref with no snapshot re-reads the row, and that attempt fails loudly (version drift: store has vN+1, launch resolved vN) rather than pairing the launch's IR with newer resources. A resume re-resolves the current row, so a skills-or-manifest-only push self-heals — but a push that changed main.bot or prompts/ moves the workflow hash, so the resume is REFUSED until you force it (and the auto-retry sweeper, which never forces, re-arms then abandons). Same for a deleted row.

• Resume re-resolves by ORIGIN, not by path: the launch persists which tier served the run — bot_source_tier (team | platform | baked) and, for the two stored tiers, the row's owner in bot_source_tenant (the team id or the platform: sentinel; empty for baked, which is why the tier is its own field). A resume/auto-retry reloads the SAME row at its current version; a row deleted mid-run fails the resume explicitly (relaunch, or resume with inline source). A run launched from the BAKED catalog picks up an override pushed since — the "effective at the next launch" rule. The pinned sandbox_image is likewise re-resolved per resume (attempt N and N+1 can differ if the setting changed between them); only redelivery of one queued message is guaranteed identical.

• Metadata: listings (/api/v1/bots), webhook command discovery, hand-off producers/consumers, gate-var defaults and retry-policy manifest reads all consult the platform overlay (TTL-cached ≤30 s per replica; the mutating replica reads its own write immediately).

• Audit: every mutation lands on the platform audit log with a sha256 content digest over the sorted file map — the provenance record for "what exactly is deployed". admin bots list shows the same digest.

Shipping a baked-catalog change — two halves, or it did not ship

A change to bots/<slug>/ that is MERGED reaches a deployment in two halves, and a run only sees it when both have moved:

• the runner image — the pod that EXECUTES the bot (its engine evaluates the bot's expressions, its baked bots/ is what the by-ref rebuild reads); it is pinned by digest in the deployment values, so a merge changes nothing until the digest is bumped;
• the server — the process that RESOLVES the bot at launch (team → platform → baked, pkg/server/bot_resolver.go) from ITS OWN baked catalog and stamps the ref on the queue message. On a digest-pinned deployment, move image.digest in its GitOps values as well as runner.image; restarting the server or changing image.tag keeps the pinned image.

Bump the runner and forget the server, and every launch still resolves the OLD bot (2026-09-06: Billy 1.6.0 on the runner, 1.5.x served — no delivery_reserve node in the run). Push a platform override to skip the image rollout, and the override runs on the runner's CURRENT engine: a bot that needs a builtin the engine does not have (1.6.0's variadic min/max, #830, on a v3.112.7 engine) compiles, then fails at its first evaluation — and the failure auto-resumes in a loop (#857, #858). The order that works:

1. bump the runner digest to an image built from the main that carries the bot AND the engine it needs (docs/cloud-deployment.md § pinning);
2. bump the server digest to that release too, then verify the applied Deployment generations and image IDs;
3. reconcile stored platform and team overrides: they still win over baked sources regardless of their version. Preserve customizations and use a version-guarded write; an image update does not replace stored sources;
4. only then dogfood; a platform override is for iterating on a bot the deployed engine already supports. See model rollout for the live-variable and active-run precautions.

A bundle may declare the engine it needs — requires.iterion

The two-halves rule above was documented in prose, and a production push broke it anyway. A bundle can now state its own floor:

# bots/<slug>/manifest.yaml
requires:
  iterion: ">= 3.112.14"

push then refuses (409) when the deployment cannot honour it, naming what the bot asked for, what the deployment runs, and where that number came from:

bot "branch-improve-loop": bot requires iterion >= 3.112.14 but this build is
v3.112.7+abc123 — upgrade the engine (or the image this bot runs on), or relax
the manifest's requires.iterion (floor from runner v3.112.7+abc123). Bump the
runner image and restart the server first (docs/platform-bots.md § Shipping a
baked-catalog change), or push anyway with --force

The floor is the minimum of this server's own build and every runner build observed on runs in the last 7 days (Run.runner_version, the build each runner stamps on what it executes). Both halves count: the server compiles the bot at launch, the runners evaluate it, and a queued run lands on whichever pod takes it.

--force is the escape hatch — pushing ahead of a rollout is legitimate — and is never silent: the response carries the overridden requirement as a warning.

If the push lands anyway (forced, or the guard could not decide), the runner refuses at launch: the run ends failed — not failed_resumable — with BOT_REQUIRES_NEWER_ENGINE, and the delivery is acked so no redelivery repeats the same arithmetic. iterion validate reports the same thing locally as C250 (unmet) / C251 (this build has no orderable version, so the check could not run). Full contract: docs/bundles.md.

Trust model

A platform override executes across all tenants, with each tenant's own credentials and bindings for that bot slug — exactly the trust level of the baked image it replaces. That is why the surface is super-admin only, safe-origin-gated, and digest-audited. Treat a push like a deploy: review the diff first (admin bots pull + git diff against the repo).

An override outlives the release that made it necessary

The tier's whole point is that a stored bundle outranks the baked catalog at every launch surface. The consequence is easy to miss: an override pushed once keeps serving after a later release bakes a newer bundle for the same slug. The image moves; the bot does not.

Measured on 2026-09-06: review-pr's override, pushed 2026-09-04 for the 0.7.0 cost pass, was still serving every production review 29 hours after #742 baked the 0.8.0 review tiers into the image. Nothing said so — the release notes, the runner digest and the bilan all reported the tiers as deployed, while the graph that actually ran had no tier_expand node.

iterion now reports it, and still does not refuse it (pinning an older bundle is a legitimate choice — a rollback is exactly this):

• GET /api/admin/bots returns bundle_version, shadowed_version and shadows_newer_version on every row. The last is omitted unless true, so a healthy inventory stays quiet. This is the check to run after any release that touched a bot you have overridden. shadowed_version is what would serve without that row, which is not always the bake: resolution is team → platform → baked, so a team row is measured against the platform override when one exists. The same fields appear on the team listing (GET /api/teams/{id}/bot-sources). If the catalog itself cannot be read, the response carries shadow_check_unavailable: true and the per-row shadow fields are absent — an inventory that looks clean because the check could not run would be worse than one that admits it did not run. That holds for a catalog that goes unreadable after serving once, not only on a cold start: the walk is TTL-cached and a failed refresh keeps serving the last map, which is a reading no caller may publish as current.
• The resolver logs one Warn naming the tenant, both versions and the two ways out — once per (tenant, origin, slug, stored version), not per launch. The tenant is in the key on purpose: many teams can hold a row for the same slug, and a slug-only key would let the first one to launch silence all the others.
• A poll publishes the same verdict as a metric, so neither of the two above needs someone to show up — the inventory waits for a caller, the Warn for a launch, and a bot nobody launches is exactly the one whose override rots. Every five minutes — twelve readings inside the hour its alert waits, and a pass costs a full read of every platform bundle plus a walk of the catalog — iterion_bot_override_shadows_newer{slug} is 1 while a platform override is older than what the deployment would otherwise serve, and the IterionBotOverrideShadowsNewer alert names the slug and both cures below. Every row is rewritten on each pass and the slugs that disappeared are retired after those writes, so an override you drop loses its series instead of paging on a row that no longer exists, while no scrape ever reads an empty inventory for one that is still shadowing; and when a leg of the check cannot be read, the per-slug values HOLD behind iterion_bot_shadow_check_unavailable == 1 rather than publishing the clean zero of a check that did not run — including when no configured bot root is on disk at all, which would otherwise read as "nothing serves below any override". A pair that does not order lands on iterion_bot_override_unorderable_version{slug}, which no alert reads — and so does a pair where one side declares no version at all (a bundle pushed without a manifest, a baked loose .bot): the override outranks the tier below it just the same, so the question is unanswered, never answered clean.

Versions are compared as dotted numeric components, so 0.10.0 correctly beats 0.9.0. Manifest.version is free-form, so a pair that does not parse numerically is treated as unordered and never flagged — a false staleness alarm on an operator's own naming scheme would be worse than the silence it replaces.

To clear a shadow, either re-push the current bundle (iterion remote admin bots push bots/<slug>) or drop the override and let the image serve (DELETE /api/admin/bots/<slug>). Prefer dropping it once the reason for the override has shipped: it restores the normal flow where releases carry bots, and removes the trap for the next release.

Known gaps (v1)

• The shadow poll covers the platform tier only, and only versions it can order. botsource.Store lists by tenant and nothing wider, so a sweep of the team tier would be one query per team per pass: a team row shadowing a newer platform override is still reported on its team listing and in the resolver's Warn, but never on a gauge. And a pair it cannot order — a deliberate suffixed pin, or either side declaring no version — only ever reaches the informational iterion_bot_override_unorderable_version, which no alert rule reads: a page on a version iterion cannot order is one nobody can clear. Semver-precedence ordering, and a cross-tenant listing, are the follow-ups.
• A deployment that bakes no catalog gets no verdict at all. The question the check answers is "what would serve without this row", read off the bot roots THIS replica has: with none of them on disk the answer is unknown, so iterion_bot_shadow_check_unavailable goes to 1 and no override can be certified current (a dropped ITERION_BOTS_PATH, a custom image or an overridden command: all land here, and the official image bakes /opt/iterion/bots). A root that exists and holds no bundle is the opposite fact — nothing serves below, so every row reads a published 0 — which is how a deployment serving stored bundles only declares itself instead of paging forever: point the bots path at a directory that exists.
• Binary files cannot ride an override (the store carries JSON text); push refuses them explicitly. A bot with binary attachments keeps its baked form. Content-addressed blob storage is the follow-up.
• File modes are dropped: the store carries path→content only, and materialization writes every file 0644 — an executable helper loses its +x bit, so a step invoking it directly fails with permission denied on the override while the baked bundle worked. push warns per executable file; call helpers through their interpreter (python3 script.py, not ./script.py) in an overridable bot.
• Provisioned webhook projections (CommandMap/BotRules stored on a forge repo integration at provision time) are not rebuilt on a push; an override that changes invocations: routes correctly through live command discovery but the provisioned map stays stale until the repo integration is re-provisioned. push warns when it detects the drift.
• Nexie's catalog skill is regenerated at run time from the runner's filesystem manifests, so other bots' overridden metadata is not reflected in it (the bot's own bundle IS the override). A DB-aware regen seam is the follow-up.
• push derives the slug from the bundle DIRECTORY name; listing/launch key on the manifest name:. Keep them identical (every shipped bot does) — a divergent pair would append a new entry instead of overriding.
• One-time resume hash mismatch: stored bots now compile through the full bundle path, so the workflow hash folds in prompts/ + presets. A run launched from a stored bot BEFORE this change was hashed source-only — resuming it hash-mismatches once; resume it with --force. Runs launched after this change are unaffected.
• The k8s sandbox driver drops host binds, so a bundle devbox.json staged via the host-bind path does not provision inside a k8s sandbox — a pre-existing gap shared with baked bundles.

Rollout note (the queue schema)

bot_bundle and sandbox_image ride the RunMessage, so their rollout is the queue's: a publisher stamps queue.SchemaVersion and a consumer accepts down to queue.MinSchemaVersion, below which a message is not consumable at all. Read those two constants in pkg/queue/types.go before sequencing a mixed-fleet deploy: this page cannot track them, and a stale number here is the exact failure the queue rollout contract exists to prevent. The reverse direction keeps the standard policy: a pre-bump runner rejects a message from a newer server, so roll server and runner from the same release (or runner first) as usual.

Platform settings: bot roles + sandbox image + bot vars

Three more runtime-settings families (ADR-090 doctrine: env/const = default, DB record = override, ≤30 s propagation, super-admin surface), stored in the same platform_settings collection as the usage caps:

# Webhook role → bot-id bindings (reviewer / revi_converse / brancher / implementer)
iterion remote admin roles                        # stored + effective + origin
iterion remote admin roles set --reviewer my-reviewer
iterion remote admin roles set --clear-reviewer   # back to the built-in default

# `sandbox: auto` fallback image — resolved at publish, pinned on the RunMessage
iterion remote admin sandbox
iterion remote admin sandbox set --default-image ghcr.io/…/iterion-sandbox-slim@sha256:…
iterion remote admin sandbox set --clear-default-image

# Bot-var overrides — ${ITERION_X:-default} resolved from the DB before the pod env
iterion remote admin vars                        # stored + origin
iterion remote admin vars set ITERION_VIBE_EFFORT_CLAUDE max
iterion remote admin vars rm  ITERION_VIBE_EFFORT_CLAUDE  # back to env/default

Role overrides apply at every webhook lane that used to read the hardcoded constants (auto-review fan-out, /revi approve, the merge-queue auto-heal, the issue-labeled implementer lane) — a symbol-sweep test keeps new code from re-hardcoding them. The sandbox image is pinned per message so a redelivery or checkpoint re-claim reruns in the same environment; prefer an @sha256 digest ref — the override sits outside the chart's image.requireDigest render guard, so a moving tag here is accepted and warned about (server log + the warning field the CLI prints), never refused. On a store outage both resolvers serve the last-known value (logged) — availability over freshness, the caps posture.