Cloud deployment

This document is the operator runbook for the cloud-mode topology (iterion server + iterion runner + Mongo + NATS JetStream + S3). It covers prerequisites, secret + token lifecycle, NetworkPolicy egress, observability, resume, and migration from a filesystem store.

The Helm chart is published to GHCR as an OCI artifact at oci://ghcr.io/socialgouv/charts/iterion (sources in charts/iterion/). It renders the full stack; values-dev.yaml brings up an in-cluster Mongo, NATS and SeaweedFS for smoke tests, values-prod.yaml expects external dependencies.

Install

bash

helm install iterion oci://ghcr.io/socialgouv/charts/iterion \
  --version <semver> \
  --namespace iterion --create-namespace \
  -f values.yaml

Pick <semver> from the iterion releases; the chart version is kept in lock-step with the binary appVersion, so helm install --version 0.5.3 deploys the iterion 0.5.3 image.

OCI registries do not expose a helm search repo index; to inspect a chart before installing, pull it explicitly:

bash

helm pull oci://ghcr.io/socialgouv/charts/iterion --version <semver>
tar -tzf iterion-<semver>.tgz | head

For chart hacking against unreleased changes, install from a checkout: helm install iterion ./charts/iterion -f values.yaml. task chart:kind exercises this path end-to-end on a kind cluster.

A first install on a cold cluster. helm install --wait waits on the Deployment's progressDeadlineSeconds, and Kubernetes' default 600 s is not a statement about iterion's boot: on a fresh node the kubelet serialises image pulls, so a store whose image waits behind the others fails the install while the server pod turns Ready a minute later (measured on kind: the nats image waited 7m15 behind the pinned mongo and seaweedfs pulls, iterion#2780). server.progressDeadlineSeconds is the knob, next to the runner's — unset by default, so an existing install keeps the behaviour it was deployed with, while values-dev.yaml (the task chart:kind path) sets 1800. It renders in cloud mode only: mode=local renders a StatefulSet, which has no such field. The cost of raising it is worth stating: a genuinely broken rollout (bad image, a crash at boot) then reads as Progressing for as long as the deadline says instead of failing at 10 minutes, so an ArgoCD sync or a CI install step hangs rather than reports. A 0 is not a way to remove the deadline — both templates render the key through a with, which skips a zero.

The install clients follow the same knob through the render: the wait follows the rendered server Deployment's progressDeadlineSeconds. task chart:kind and the cloud-e2e CI job build one argument list — release, chart, -f, --set — and hand it to both scripts/chart-kind-wait.sh and helm install; the helper runs helm template on that list, reads the server Deployment's spec.progressDeadlineSeconds from the render with yq (mikefarah/yq v4, in devbox as yq-go), and prints that deadline plus a 120 s margin as the client --timeout, so the server's deadline — not a fixed client cap — decides whether a slow server rollout is called Failed. It is the one deadline the wait follows: helm --wait watches every workload of the release, so a Deployment whose own deadline outlives the server's — the runner's 30000 s, a lame-duck drain budget values-dev.yaml keeps — is cut by the client cap, not its deadline, if it is still Progressing when the wait ends, and a StatefulSet (NATS) has no deadline, only the cap. And the deadline bounds the whole server rollout for one pod only: Kubernetes measures it from the rollout's last sign of progress, not its start, and each pod that turns Ready is one — with several server pods (server.replicas above 1, or the HPA owning the count) the rollout can stay Progressing past the wait, so the helper prints the wait with a warning naming the count; values-dev.yaml runs one pod. Nothing reads the values file itself: the render already applies what stands between a file and the field (go-yaml's reading of the file, the --set overrides, the with that skips an unset or 0 knob, the StatefulSet of mode=local), and a reader of the file would have to reimplement all of it. That render is the install's own while the templates the server Deployment renders through read no cluster or release state — .Capabilities, lookup, .Release.IsInstall / .Release.IsUpgrade / .Release.Revision, which helm template renders off-cluster as an install at revision 1; none is read today, and TestChartKindWaitRenderReadsNoClusterState (charts/iterion) refuses one on that path, whatever its receiver (the chart hands its includes the root as .root). The helper refuses, naming the cause: a render helm refuses, a render with no Deployment labelled app.kubernetes.io/component=server (mode=local), several such Deployments (the first in render order would be a guess), a Deployment that renders no deadline (unset or 0 — the install would silently watch Kubernetes' default 600 s), a value other than plain decimal digits with no leading zero, and one beyond the int32 the field holds. helm hands the API the render as a YAML 1.1 decoder reads it: a string or a fraction is refused there, an integral float (6e2, or the 1e+06 helm prints for a values number of a million or more — quote such a number) and a signed, hex, octal, binary or underscored spelling are read as the integers they denote, and a leading zero makes octal digits octal (010 is 8) where a YAML 1.2 reader such as yq reads 10 — so the helper sizes the wait from plain digits alone. Its messages print the install as its release, its chart and its values files, never its other arguments — a --set can carry a credential, and CI logs keep stderr — and a URL among them without its userinfo, query or fragment. From the first flag it does not know (--password) on, it prints nothing: only helm knows whether that flag takes the next argument.

Topology

For the fuller control-plane / data-plane view, see cloud-architecture.md.

server publishes RunMessages onto JetStream and serves the studio + run console (REST + WebSocket).
runner pulls RunMessages, claims a NATS-KV lease, executes the workflow, and writes events + artifacts to Mongo + S3.

Sovereign runner pools

runnerPools[] renders one runner Deployment per sovereign pool (sovereign-pools.md): a team whose code must never reach a public LLM vendor runs on its own pool, isolated by the code on the shared install — no dedicated namespace, no NetworkPolicy.

Each pool pod is pinned three ways: ITERION_RUNNER_POOL (the admission stamp — a delivery whose msg ≡ doc ≡ pod stamps do not all name THIS pool is parked on the pool's DLQ, never executed), the pool's own JetStream stream + DLQ + durable consumer, and its gateway credential. The pod carries the gateway credential ONLY in its env — no platform LLM secrets, no run-secrets ring key (the run envelope is self-contained since P4-b). It runs under the shared runner ServiceAccount, whose sandbox-reaper role can read the namespace's Secrets; that residual is the shared runner's own, and scoping the pool pods to a dedicated account is a follow-up.

To bring a pool up:

The pool pod reads TWO keys from the auth Secret by specific reference — ITERION_JWT_SECRET and the authenticated ITERION_MONGO_URI (a pool without either fails at boot; the ring ITERION_SECRETS_KEY* stays out of the pod). On a chart-managed auth Secret, set secrets.auth.mongoURI; on an externally managed one, the key must already be there.
Register the pool in the server's registry first (PUT /api/admin/runner-pools) — the registry reconciler creates the pool's JetStream topology, and the name must match.
Map the team (PUT /api/admin/teams/{id}/runner-pool) — one team per pool.
Create the gateway Secret manually (the chart never renders credentials): kubectl create secret generic "<release>-pool-<name>-gateway" --from-literal=OPENAI_COMPATIBLE_API_KEY=<key> (the Secret key defaults to OPENAI_COMPATIBLE_API_KEY; gateway.apiKeySecretKey overrides it)
Add the pool to runnerPools[] and upgrade:

yaml

runnerPools:
  - name: honorabilite
    replicas: 1
    gateway:
      baseUrl: "https://<regional-gateway>"   # not a secret
      existingSecret: "<release>-pool-honorabilite-gateway"
    # env:                                   # per-pod literals, e.g.
    #   ITERION_LLM_ENDPOINT_ALLOW_PRIVATE: "1"  # a private-address
    #                                       # gateway — never set that
    #                                       # through the SHARED configmap

When networkPolicy.egress is active, allow the gateway host through networkPolicy.egress.extraAllow (the pool pods use the same policy; the sandbox runs' own egress is governed by each run's network: block as for any runner — open when the workflow declares none, unless the run is a repo-defined sandbox, see below).

Draining follows the shared runner contract (lame-duck terminationGracePeriodSeconds): scale the pool's Deployment down and its in-flight run finishes or checkpoints within the window.

Prerequisites

Component	Requirement
Kubernetes	1.28+ for `context.WithoutCancel` semantics + native `Probe.gRPC` (optional)
CNI	NetworkPolicy enforcement enabled (Calico, Cilium, Antrea) when `networkPolicy.enabled=true`
MongoDB	6.0+ with replica set (change-streams require an oplog)
NATS	2.10+ with JetStream enabled
S3-compatible	bucket pre-created with `s3:ListBucket`, `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject` for the IAM principal
Valkey / Redis (optional)	required only for multi-replica servers — shares per-pod state across replicas (see below); a single server replica runs without it
KEDA (optional)	2.13+ if `runner.keda.enabled=true`
Prometheus Operator (optional)	for `metrics.podMonitor.enabled=true`

Auth bundle and access tokens

Every cloud server requires an auth bundle at boot:

Env var	Purpose	Generate with
`ITERION_JWT_SECRET`	Server-side HS256 signing key for short-lived access JWTs (at least 32 random bytes)	`openssl rand -base64 48`
`ITERION_SECRETS_KEY`	AES-256-GCM master key for sealing BYOK, OAuth, and run-scoped credentials (exactly 32 bytes before base64)	`openssl rand -base64 32`

Without those values, cloud-mode validation aborts with an explicit error (use ITERION_DISABLE_AUTH=true only for local smoke tests, not for shared deployments). The server pods need ITERION_JWT_SECRET; both server and runner pods must agree on ITERION_SECRETS_KEY so runners can unseal the credential bundle attached to each run.

Generate + apply the Secret:

bash

kubectl create secret generic iterion-auth \
  --from-literal=ITERION_JWT_SECRET="$(openssl rand -base64 48)" \
  --from-literal=ITERION_SECRETS_KEY="$(openssl rand -base64 32)" \
  --from-literal=ITERION_BOOTSTRAP_ADMIN_EMAIL=ops@example.com \
  --namespace iterion

Reference it from values-prod.yaml:

yaml

secrets:
  auth:
    existingSecret: iterion-auth

On the first boot of an empty users collection, ITERION_BOOTSTRAP_ADMIN_EMAIL creates a super-admin account with a one-time password printed in the server logs. Capture that password, sign in, change it, and remove the bootstrap env var on the next deploy.

Adding ITERION_BOOTSTRAP_ADMIN_PASSWORD to the same Secret switches that account to a declarative one: it is created active with the declared password and reconciled to it on every boot (super-admin flag, active status, password only when it has drifted), and nothing is logged. Rotation is a Secret update plus a restart — and a password changed through the UI reverts at the next restart, which is the trade-off for making the Secret authoritative. Keep both variables set on this path. See cloud-admin-guide.md.

API clients do not send a static deployment token. They authenticate with an access JWT issued by login/refresh, passed as Authorization: Bearer <access-jwt> or via the __Host-iterion_auth cookie (bare iterion_auth on a deployment without TLS). WebSocket clients that cannot set headers may pass the same access JWT as ?t=<access-jwt> on /api/ws/*. Health probes, server info, and auth bootstrap routes remain public.

For rotation details, including JWT signing-key rotation and ITERION_SECRETS_KEY impact, see cloud-admin.md.

Queue connection (NATS JetStream)

Cloud mode routes runs through a NATS JetStream queue: the server publishes RunMessages, the runner pool pulls them. ITERION_NATS_URL is required when ITERION_MODE=cloud — the server refuses to start with ITERION_NATS_URL required when mode=cloud otherwise. The stream / bucket / DLQ names and the JetStream tuning knobs have working defaults (pkg/queue/nats/nats.go); override them only to match an existing cluster (a 0 on a numeric/duration knob inherits the default).

Env var	Purpose
`ITERION_NATS_URL`	JetStream connection string (`nats://[user:pass@]host:4222`) — required in cloud mode
`ITERION_NATS_STREAM`	Runs stream name (default `ITERION_RUNS`)
`ITERION_NATS_KV_BUCKET`	Per-run distributed-lease KV bucket (default `iterion-run-locks`)
`ITERION_NATS_DLQ_STREAM`	Dead-letter stream for max-deliver-exhausted messages (default `ITERION_RUNS_DLQ`)
`ITERION_NATS_STREAM_REPLICAS`	JetStream replication for the runs stream, the DLQ and the lease KV (default `1`). A clustered NATS still serves R1 assets unless this is raised: one node restart then silences the queue while its peers idle. Set it to the cluster size; existing streams migrate in place at the next server boot.
`ITERION_NATS_MAX_ACK_PENDING`	Fleet-wide in-flight (delivered-unacked) ceiling on the durable consumer
`ITERION_NATS_MAX_DELIVER`	Redelivery budget before a message parks on the DLQ (default 8)
`ITERION_NATS_ACK_WAIT`	Per-message ack deadline, refreshed by runner heartbeats
`ITERION_NATS_MAX_AGE` / `ITERION_NATS_DLQ_MAX_AGE`	Runs-stream / DLQ retention
`ITERION_NATS_MAX_PAYLOAD`	Max message size

A broker that is not up yet is waited for, bounded. On a fresh cluster the queue pod routinely starts after the processes that need it — the kubelet serialises image pulls, and a Service's DNS record appears when the Service does — so the boot dial retries for at most 30 s (5 s per attempt, 500 ms between) while the failure means "not up yet": nothing listening (nats: no servers available for connection), an unresolved host, or a container that accepts TCP before it serves the protocol. That is the policy pkg/store/mongo/ping.go already applies to its primary ping, value for value. A failure no retry can cure (a malformed URI, a credentials refusal) is reported from the FIRST attempt, and a broker that never answers ends the boot with connect NATS: queue/nats: connect: no broker after N attempt(s) (deadline …) — never with a connection that looks live but is not (nats.RetryOnFailedConnect is deliberately unused: a readiness probe would pass on it).

Neither wait is an operator knob, because the two of them plus the ~28-collection schema pass share ONE budget: the startup probe's 30 × 5 s = 150 s (probes.startup). Raising either is a decision about that 150 s. And what the wait buys on a cold cluster is one crash-loop backoff, not an arbitrarily late broker — the 7-minute image pull measured in iterion#2780 is far longer than the budget, and what helm install --wait concludes is the Deployment's progressDeadlineSeconds business.

The queue's internal semantics (the MaxAckPending fleet ceiling, AckWait heartbeats, DLQ parking, the orphan sweeper) are covered in cloud-architecture.md § Queue internals.

Shared replica state (Valkey / Redis)

Some server state is per-pod and must be shared when you run more than one server replica: forge OAuth/CSRF/manifest-install state, board-MCP run tokens, and auth rate-limit buckets. Configure a Valkey/Redis backend and every replica reads/writes the same store; leave it unset and the server falls back to in-memory implementations (correct for a single replica, but an OAuth callback or rate-limit check can then land on a pod that never saw the paired request).

Env var	Purpose
`ITERION_REDIS_URL`	Single-node connection string (`redis://[:pass@]host:port[/db]`) — dev/local topology
`ITERION_REDIS_SENTINEL_ADDRS`	Comma-separated Sentinel endpoints for the HA failover topology (wins over `ITERION_REDIS_URL` when set)
`ITERION_REDIS_MASTER_NAME`	Sentinel-monitored master name (required with `ITERION_REDIS_SENTINEL_ADDRS`)
`ITERION_REDIS_PASSWORD`	Password for the data nodes
`ITERION_REDIS_SENTINEL_PASSWORD`	Password for the Sentinels (defaults to `ITERION_REDIS_PASSWORD`)

A Valkey outage degrades gracefully — each operation is bounded by a short round-trip timeout rather than blocking the request path.

NetworkPolicy egress

values-prod.yaml ships with networkPolicy.enabled=true + an empty networkPolicy.egress.extraAllow so the cluster default-denies egress except DNS. Add explicit rules for Mongo, NATS, S3, and the LLM provider (the allowlist is nested under egress):

yaml

networkPolicy:
  enabled: true
  egress:
    extraAllow:
      # In-cluster Mongo (same namespace)
      - to:
          - podSelector:
              matchLabels:
                app.kubernetes.io/name: mongodb
        ports:
          - protocol: TCP
            port: 27017
      # External LLM provider (Anthropic)
      - to:
          - ipBlock:
              cidr: 0.0.0.0/0
        ports:
          - protocol: TCP
            port: 443

The chart synthesises a single egress block from the union of defaults + egress.extraAllow. There is no auto-detection of bundled sub-charts; if you run the dev stores (devStack.*), add the matching egress.extraAllow entry.

Sandbox pods are not selected by this policy. It matches the release's app.kubernetes.io/* labels; a per-run sandbox pod carries iterion.io/* labels only. Its egress is the run's own network: policy (sandbox): open by default, and routed through the CONNECT proxy on the runner pod plus a per-run NetworkPolicy only when the workflow declares an allowlist or denylist sandbox.network mode, the run is a repo-defined sandbox, or the run carries secrets Layer 2 inspects / a gateway credential (then the proxy admits every public host the workflow's own !host exclusions do not refuse). Without the proxy the sandbox pod's egress is whatever the cluster allows. No chart value sets the egress mode for every run.

This policy and the sandbox proxy do not compose. With networkPolicy.enabled on a CNI that enforces NetworkPolicy, the policy's Ingress rules admit only the server port and the metrics port on the runner pod, while the runner's CONNECT proxy (and the per-run MCP listeners) bind an ephemeral port (pkg/sandbox/kubernetes/driver.go ProxyConfig). A sandbox pod cannot reach that port, so a run that needs the proxy — an allowlist or denylist run, a repo-defined sandbox, a Layer 2 run, a run holding a gateway credential — fails at its first egress. No value opens it: networkPolicy.ingress.extraAllow widens the from: of the server port's rule only. On such a cluster, either keep networkPolicy.enabled off for the release that runs sandboxes, or run the sandbox with a no-proxy posture (network: open, no inspected secret, no gateway credential).

The server port's ingress is scoped the same way (#2804): name your ingress controller's namespaces in networkPolicy.ingress.controllerNamespaces (matched on the kubernetes.io/metadata.name label every namespace carries since 1.22) so the rule admits the controller's pods and the release's own — anything else in the cluster that can dial the server port directly can forge X-Forwarded-For (the audit IP, the rate-limit key). networkPolicy.ingress.extraAllow is the escape hatch (an ipBlock for kubectl port-forward's node source, a controller living elsewhere). The chart ships ingress-nginx as the default controllerNamespaces (the common case — override it for your controller). The empty-config verdict is decided (#2804): emptying BOTH lists keeps the historical admit-all render — never a refusal, so an upgrade never breaks — and warns loudly: an iterion.io/ingress-admit-all annotation on the policy, a WARNING comment in the rendered manifest, and a note at install/upgrade.

Whose `X-Forwarded-For` the server believes

The server-side half of the same question is config.auth.trustedProxyCidrs (ITERION_AUTH_TRUSTED_PROXY_CIDRS): the ranges whose forwarded client address the server reads instead of the socket address. It decides the audit IP and the auth rate-limit key, so it also decides whether the limiter holds one bucket per client or one for the whole instance — with no range configured every request behind the ingress carries the controller's own address, and login 5/min, register 2/min and refresh 30/min become instance-wide. The server warns about that at boot, naming the variable; it never refuses to start.

The chart defaults to the private ranges a pod network comes from (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7), so the ingress in front is trusted without a gesture. Narrow it to your controller's pod CIDR: every address inside a trusted range is a proxy you vouch for, which is why the ingress scoping above matters — the two knobs cover one threat from both ends.

yaml

config:
  auth:
    trustedProxyCidrs:
      - 10.244.0.0/16   # your ingress controller's pod CIDR

An entry that is not a CIDR is refused at boot naming it, and so is one with host bits set (10.42.0.7/16 would silently trust the whole /16). Emptying the list is honoured and warned about: the client becomes the socket address everywhere, unforgeable and instance-wide.

The ranges say who may name the client and config.auth.trustedProxyHops (ITERION_AUTH_TRUSTED_PROXY_HOPS) says how many of the chain's rightmost entries are theirs: the chain is read right to left and stops there, one hop by default, which is this chart's shape. That count is what tells a proxy's entry from an in-range client's — with all of private space trusted, an in-cluster pod writes entries too — so raising it past the appending hops really in front makes a client's own entries the identity again. Raise it only for a chained front end (a CDN edge appending ahead of this chart's ingress is 2, with the CDN's egress ranges added to the list above).

The client's address has to reach the controller first

trustedProxyHops: 1 is right only when the address the ingress controller sees is the client's. With the Kubernetes-default externalTrafficPolicy: Cluster on the LoadBalancer Service, every packet is SNATed to the node that received it, so the controller sees a node address — and that is what lands in the chain. Nothing in the ranges or the count fixes it: the client is not in the request at all when the controller replaces the chain (use-forwarded-headers: false, ingress-nginx's default), and it is one entry beyond the bound when the controller appends behind a load balancer that wrote it. Either way the whole internet shares one bucket per node and one audit IP. The shapes that carry the client through:

shape	what reaches the server	settings
`externalTrafficPolicy: Local` on the Service, or PROXY protocol where the cloud LB speaks it (`use-proxy-protocol: "true"`)	the controller's socket peer IS the client	as shipped — `trustedProxyHops: 1`
a managed L7 load balancer that writes `X-Forwarded-For` itself, the controller appending behind it: `use-forwarded-headers: "true"` and `compute-full-forwarded-for: "true"`	`<client>, <the controller's TCP peer>`	`trustedProxyHops: 2`, and in `trustedProxyCidrs` the range of the address the controller's TCP peer carries — the LB's egress range when the LB targets the pods directly (NEG / `ip` targets), the node range (or `externalTrafficPolicy: Local`) behind a NodePort / `instance` target
the same load balancer, the controller resolving the client itself: `use-forwarded-headers: "true"` alone, with `proxy-real-ip-cidr` set to the range of the address the controller's TCP peer carries — the LB's egress range when the LB targets the pods directly, the node range (or `externalTrafficPolicy: Local`) behind a NodePort / `instance` target — never its `0.0.0.0/0` default	`<client>`	as shipped — `trustedProxyHops: 1`

use-forwarded-headers does two different things, and the rows differ by which: alone, ingress-nginx's realip module resolves the client from the incoming header and writes one entry, the resolved address; with compute-full-forwarded-for it appends its peer to the incoming header instead. Behind an L4 load balancer — a SNATing Service with nothing in front writing the header — turn it on for neither reason: nothing on that path puts the client into X-Forwarded-For, so the one entry the controller writes is whatever the client itself put leftmost in its own header (under the default proxy-real-ip-cidr), a fresh bucket per forged value and a chosen audit IP. Only externalTrafficPolicy: Local or the PROXY protocol carry the client through an L4 path; ingress-nginx's own documentation reserves use-forwarded-headers for a controller behind another L7 proxy.

The server says so once per process. When it can name the cause it does — the declared header absent from the requests, present but naming no address, a chain of our own addresses shorter than the count declares, an in-range hop past it — and otherwise at the first request whose identity lands inside trustedProxyCidrs: "the client of a request from … resolves to …, which is inside ITERION_AUTH_TRUSTED_PROXY_CIDRS". The lines are ambiguous on purpose — an in-cluster pod or a LAN client is inside those ranges too, and for them the identity is the right one. Nothing in a request tells the two apart, which is why each is a question and not a verdict.

config.auth.forwardedHeader (ITERION_AUTH_FORWARDED_HEADER) says where, and the server reads that one header and no other. The default — X-Forwarded-For, which this chart's ingress writes — is right for the shipped shape. Name your own when the proxy in front computes the client into X-Real-IP or RFC 7239 Forwarded and leaves the client's X-Forwarded-For alone: without it the client's header is the identity, and no narrowing of the ranges changes that. The server warns once when the declared header is not the one its requests carry.

NATS monitoring endpoint (KEDA)

KEDA's NATS JetStream scaler scrapes /jsz on the monitoring port (8222 by default), not the client URL. The chart helper iterion.nats.monitoringEndpoint resolves to:

.Values.config.nats.monitoringEndpoint if set, else
<release>-nats:8222 for bundled NATS, else fails.

For external NATS:

yaml

config:
  nats:
    url: nats://nats.shared:4222          # JetStream client port
    monitoringEndpoint: nats.shared:8222  # /jsz scrape

Metrics & dashboards

The server + runner expose /metrics on :9090 (configurable via server.metricsPort / runner.metricsPort, or the ITERION_METRICS_PORT env var). Counters/gauges are documented at pkg/cloud/metrics/metrics.go and populated at runtime:

Metric	Pod	Meaning
`iterion_runs_created_total{status}`	server	Every Launch/Resume publish
`iterion_runs_active{status="running"}`	runner	Sum across pods = in-flight runs
`iterion_run_duration_seconds{status}`	runner	Histogram, terminal status
`iterion_ws_connections`	server	Live run-console subscribers
`iterion_mongo_change_stream_lag_seconds`	server	Set on each delivered event
`iterion_nats_pending_messages`	runner	Polled every 15s from JetStream consumer
`iterion_llm_tokens_total{backend,model,direction}`	runner	input/output/cache_read/cache_write
`iterion_llm_cost_usd_total{backend,model}`	runner	Added per claw-priced call; unknown models leave the counter untouched
`iterion_runner_heartbeat_errors_total`	runner	Each KV lease refresh failure
`iterion_bot_override_shadows_newer{slug}`	server	1 while a platform bot override shadows a newer baked bundle (polled every five minutes, platform tier)
`iterion_bot_shadow_check_unavailable`	server	1 when that check could not run — no readable bot root included; the per-slug gauges hold their last values
`iterion_bot_override_unorderable_version{slug}`	server	1 for an override whose version cannot be ordered against the bundle below it; informational, unalerted

Wire a Prometheus PodMonitor:

yaml

metrics:
  podMonitor:
    enabled: true
    interval: 30s

/metrics is ClusterIP-only by design — no ingress should expose it publicly.

Tracing

The server + runner emit OpenTelemetry spans:

iterion.api.launch_run, iterion.api.resume_run (server)
iterion.runner.process_one (runner, root span per run)
iterion.node.execute (engine, child span per node)

Trace context propagates through the W3C traceparent header on the NATS RunMessage so a single trace covers client → server → queue → runner → node graph.

Configure the OTLP exporter via standard env vars:

yaml

config:
  extraEnv:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://tempo.observability:4318"
    OTEL_SERVICE_NAMESPACE: "iterion"
    OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=prod"

When OTEL_EXPORTER_OTLP_ENDPOINT is unset, spans are dropped and the W3C propagator-only path is installed (inbound trace context still respected, but no export).

Spreading the replicas: anti-affinity, spread constraints, disruption budgets

Three replicas on three nodes mean nothing unless something keeps them apart: on ovh-prod, MongoDB and valkeyha held one replica per node by chance alone — no anti-affinity on one, a dead selector on the other (app.kubernetes.io/name In [valkey] against pods labelled valkeyha, the sub-chart's alias) — until a reschedule could have put two on the node the autoscaler drains (#2012). The chart now spreads its own workloads and ships the check that found the defect.

server.scheduling / runner.scheduling (a runnerPools[] entry overrides them key by key): antiAffinity: soft (preferred, the default — a one-node cluster still schedules) or hard (required — a replica with no free node stays Pending) over topologyKey (kubernetes.io/hostname), plus a topologySpreadConstraints entry (maxSkew: 1, ScheduleAnyway by default, DoNotSchedule for a hard spread). The selector is rendered from the workload's own labels, never typed: it cannot name a label the pods do not carry. affinity: stays the free-form block, merged over the generated one.
server.pdb / runner.pdb: the PodDisruptionBudget renders only when the workload is guaranteed more than one replica (replicas > 1, or the HPA / KEDA floor above one). On a single replica, minAvailable: 1 refuses every eviction and holds a node drain forever — a budget that protects nothing and blocks everything — so auto never renders it there; enabled: true|false forces it. maxUnavailable replaces minAvailable when set.
The render gate — task chart:check (in task check) renders the dev, prod and pool values and confronts every anti-affinity, spread and PodDisruptionBudget selector with the labels of its own pods; a selector matching none is refused with the workload and the dead key named (internal/rendergate). The same judge reads ANY rendered stream: helm template … | task chart:gate -- -namespace <apply namespace> — the namespace the objects land in (helm template writes none into them), so a chart's objects and its sub-charts' are judged together — run it on the GitOps repository's render before pushing an affinity or a budget there. The datastores of the cloud deployment (MongoDB, NATS, valkey, SeaweedFS) are other charts: their anti-affinity and budgets live in those values, and this gate is how a dead one is caught before Argo CD applies it.

Pod turnover & in-flight runs

A runner pod goes away for three reasons, and a deploy is the rarest of them:

autoscaling — with KEDA enabled the pool scales down whenever the queue drains. That is continuous, it happens under normal operation, and the ReplicaSet picks a victim without knowing which pods are busy. Measured in production on 2026-08-01: SuccessfulRescale … reason: All metrics below target killed a pod 21 minutes into a campaign, with idle pods available. A PodDisruptionBudget does not cover this — a scale-down deletes pods directly rather than evicting them, so it never consults the PDB. The drain below is what does.
node turnover — a drain, an upgrade, a spot reclaim.
a deploy — a rolling restart of the runner Deployment.

Schema bump? If the deploy changes the queue wire version (pkg/queue/types.go SchemaVersion), pod turnover is the easy half — the mixed-version window is governed by docs/cloud-queue-schema-rollout.md. Read its Deploy ordering section first. The default is server-first, which keeps any parked run on the side a resume recovers first; a bump that leaves MinSchemaVersion alone additionally allows runner-first, which rejects nothing and needs neither a drained queue nor a resume of parked runs.

Pinning the image

image.tag accepts a moving tag (edge, main, latest) and many installs use one to iterate fast. Know what it costs: a tag is resolved per pod, at that pod's own start time, so

a publish landing mid-rollout splits one ReplicaSet across two builds — measured on prod 2026-09-02, three server pods of ReplicaSet 548c4b7f7d serving v3.92.0 and v3.93.0 at once, visible outside as /healthz and /api/server/info disagreeing on commit;
a later HPA/KEDA scale-up creates a pod from an existing ReplicaSet, and what it runs depends on pullPolicy. With Always it is whatever the tag means then; with the chart default IfNotPresent kubelet does not re-resolve a tag it already has cached on that node, so the pod runs an arbitrarily older layer. Either way the ReplicaSet drifts with no rollout entry and no signal.

IfNotPresent has a second consequence worth stating plainly: on a moving tag, kubectl rollout restart is not a reliable way to reach the current build — a node with the tag cached serves the cached layer. Fast iteration on :edge needs pullPolicy: Always at minimum, and even then buys only "current at each pod's start time", not "one build".

That matters beyond cosmetics once the rollout epoch is in play: the epoch rides the PodTemplate, which assumes one ReplicaSet is one build.

image.digest (sha256:…) wins over image.tag and is the only reference that makes that true:

yaml

image:
  repository: ghcr.io/socialgouv/iterion
  digest: sha256:…        # wins over `tag`

# resolve a tag to the digest it points at — the signed release tag for prod,
# the moving tag for a dev install
gh api /orgs/SocialGouv/packages/container/iterion/versions \
  --jq '[.[]|select(.metadata.container.tags|index("v4.7.1"))][0].name'
# the release tags are cosign-signed (keyless, SBOM attested): verify the
# digest you are about to commit against the workflow identity that built
# it — the identity is the reusable workflow AT a release tag ref, so a
# signature the same workflow issued on another ref does not pass. The
# registry path is lowercase (`socialgouv`); the identity is the repository's
# own, case-sensitive spelling (`SocialGouv`)
cosign verify ghcr.io/socialgouv/iterion@sha256:… \
  --certificate-identity-regexp \
    '^https://github\.com/SocialGouv/iterion/\.github/workflows/_build-image\.yml@refs/tags/v[0-9]' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

The trade-off is real and yours: a digest turns each deploy into an explicit bump instead of a rollout restart. That bump is the deploy gesture of SocialGouv's prod: resolve the release's digest per package — iterion for the server (image.digest), iterion-runner-devbox for the runner (runner.image, a …@sha256:… reference) and iterion-sandbox-full for the sandbox: auto image (config.extraEnv.ITERION_SANDBOX_DEFAULT_IMAGE) — commit the three in the GitOps values, let ArgoCD sync. Only the :vX.Y.Z tags are signed — :edge is amd64-only, unsigned and ephemeral, a dev install's tag, never a prod reference.

Once digest is set in a values file, every --set image.tag=… becomes a silent no-op — the upgrade succeeds, no PodTemplate field changes, nothing rolls, and nothing warns. That bites the sequencing recipes in cloud-queue-schema-rollout.md, where the phase-1 command also sets runner.image — which IS consumed verbatim. On a digest-pinned install that phase rolls only the runners, to the old tag, while the server stays put: pods restart, the upgrade reports success, and an operator who reads that as "the server is on the new build" then publishes new-version messages from the old one. Move a digest-pinned install by bumping image.digest (and runner.image to a digest reference), never by --set image.tag.

The chart pins nothing by default, and image.digest pins the shared image only. runner.image — the per-runner-pod override — is empty by default and falls back to that same reference, so a runner pool given its own image keeps the hazard until it is pinned there too. That half matters more under the rollout epoch: a pod created from an old ReplicaSet keeps its literal ITERION_RUNNER_EPOCH while pulling whatever the tag means now.

(SocialGouv's own prod values pin the server, the runner and the sandbox: auto image by digest, the runner for a second reason: a moving tag rolled the runner Deployment on every merge to main and killed in-flight runs. That is a values-level choice, not a chart default.)

image.requireDigest: true is the guard that keeps such an install pinned: the render then refuses — with the values path to fix — any iterion pod image that is not repo@sha256:<64 hex>: the server and runner (image.digest unset, or a tag in runner.image), a runnerPools[].image, and, once runner.sandbox.enabled is on, the sandbox: auto image — a pool's own env.ITERION_SANDBOX_DEFAULT_IMAGE, and config.extraEnv.ITERION_SANDBOX_DEFAULT_IMAGE whenever a pod reads it there (the shared runner, or a pool whose env block does not set its own; a pools-only install with every pool pinned is not asked for it). The helm test pod (busybox) and the devStack images (dev-only) stay out of it. Helm has no warning channel, so the guard is a refusal and opt-in is its escape hatch: the default is false and an install that never set it renders exactly as before. Set it in the GitOps values that already pin by digest, so a hurried hotfix cannot slide them back to a moving tag.

One sibling of that env sits outside the render guard: the platform setting sandbox.default_image (iterion remote admin sandbox set --default-image, platform bots § sandbox) overrides ITERION_SANDBOX_DEFAULT_IMAGE at publish time from the DB, with no chart in the path. The server accepts any non-blank reference there and warns — in its log and in the response the CLI prints — when the value is not repo@sha256:…; on a digest-pinned install keep it unset or a digest reference.

Verifying that an infra-apps push landed (ArgoCD sync liveness)

Autosync usually applies a push within a minute, but it is not guaranteed to: on 2026-09-05 a values push (runnerEpoch: 1 + a SealedSecret change) sat unapplied for ~2h30 with no error anywhere the operator could see, and was applied together with the NEXT push, in under a minute. The failure mode is indistinguishable from "nothing to do", and the next unrelated push silently carries the stalled change in — inside someone else's rollout.

So a push is verified by the Deployments' metadata.generation moving, never by the pods (a drain-safe runner rollout keeps the old pods around for minutes):

kubectl -n iterion get deploy iterion iterion-runner \
  -o custom-columns=NAME:.metadata.name,GEN:.metadata.generation,UPD:.status.updatedReplicas

Generation unchanged after ~5 min: the cheap nudge is an empty commit on the config repo (git commit --allow-empty -m "chore: nudge argocd") — a new revision is what cleared the 2026-09-05 stall.
Still unchanged: read the app's status.sync / status.operationState with a fresh ArgoCD token (argocd app get <app>). An expired operator token is what made that stall unreadable at the time.
A runner rollout that DID apply takes ~5 min end to end (new ReplicaSet ready, then the old one drains); the pods' imageID is the proof that a digest bump reached them.

A scheduled sync-liveness probe (config-repo HEAD vs the app's status.sync.revision age) is tracked in SocialGouv/iterion#733.

Bumping the chart dependency (a template change reaches a deployment)

A deployment does not follow the chart: infra-apps pins the iterion OCI dependency to an exact version in iterion/Chart.yaml, so a template change released on main (a new values key, a new env passthrough) is inert until that pin moves. The chart version tracks the binary's appVersion, so the bump is "to the release that carries the change", never "to latest" — and it drags every template change released in between. The procedure that keeps that honest, as run on 2026-09-06 for the priorityClassName keys (chart 3.102.0 → 3.112.2):

bash

# 1. What changed in the templates between the pinned version and the target?
git log --oneline v3.102.0..origin/main -- charts/iterion/
git diff --stat v3.102.0 origin/main -- charts/iterion/ ':!charts/iterion/Chart.yaml'

# 2. Render the umbrella chart BEFORE touching anything (deps as committed).
cd /path/to/infra-apps
helm template iterion ./iterion -n iterion -f iterion/values.yaml -f iterion/values.ovh-prod.yaml > /tmp/render-before.yaml

# 3. Bump the pin + refresh the vendored archive and Chart.lock.
#    (edit iterion/Chart.yaml: dependencies[iterion].version, with a dated
#     comment naming what the bump brings, in the file's existing style)
helm dependency update ./iterion      # pulls iterion-<v>.tgz, rewrites Chart.lock, deletes the outdated archive

# 4. Set the new values, render AFTER, and read the diff of the RENDERED manifests.
helm template iterion ./iterion -n iterion -f iterion/values.yaml -f iterion/values.ovh-prod.yaml > /tmp/render-after.yaml
diff -u /tmp/render-before.yaml /tmp/render-after.yaml | grep -E '^[+-]' | grep -vE '^(\+\+\+|---)' | sort | uniq -c | sort -rn

The rendered diff is the review: it must contain the lines you meant (here two priorityClassName: lines), the chart labels (helm.sh/chart, app.kubernetes.io/version — on every object, so BOTH Deployments roll), and nothing else. A checksum/config change on the Deployments is expected when the chart label is part of the hashed ConfigMap metadata; an unexplained line in a ConfigMap or Secret is a template change from an intermediate release that step 1 should have named. Commit the four files together — Chart.yaml, Chart.lock, the new charts/iterion-<v>.tgz, the deleted old archive, plus the values — with explicit paths, push, then verify by the Deployments' generation (see the section above), never by pods.

A helm plugin that breaks helm dependency update on this host is side-stepped with HELM_PLUGINS=/tmp/no-helm-plugins helm ….

Generation-aware rollout

The chart uses RollingUpdate with maxSurge: 100% and maxUnavailable: 0 for both server and runner Deployments. This prepares a full replacement capacity before Kubernetes removes old pods. It is an SLO mechanism, not the correctness boundary: HPA/KEDA can scale ReplicaSets proportionally, and JetStream NumPending does not count work already delivered to busy runners.

The correctness boundary is config.rollout.runnerEpoch:

the server stamps the epoch on every launch, resume, webhook, schedule and retry publication;
a runner accepts historical messages (messageEpoch <= selfEpoch) and delayed-Naks a future epoch before metrics, span, lease or running state;
the last rejected delivery is parked on the DLQ and the queued attempt is changed to failed_resumable, using the same recovery path as a schema mismatch;
a persistent no-TTL JetStream KV bucket (iterion-runner-rollout, key epoch.high-water) prevents a newly started lower-epoch process from publishing or consuming. Its /readyz reports 503 superseded while /healthz remains live for diagnosis.

Connect only observes that mark. A process claims (and, if needed, advances) it after all dependencies and listeners have been proven, immediately before serving or fetching. Queue publication and delivery remain fenced until that late claim succeeds. A broken epoch-bump release therefore cannot poison the mark and prevent the healthy previous generation from restarting.

The epoch and epochMismatchDelay are rendered as literal environment values in each PodTemplate, not in the shared ConfigMap. This is essential: if KEDA creates a pod from an old ReplicaSet after the ConfigMap has changed, that pod must retain epoch N instead of impersonating N+1.

Bootstrap in two releases:

Ship wire v12 and all fencing code with runnerEpoch: 0, following the queue-schema runbook. Confirm every server and runner probe reports epoch 0.
In a later release, set runnerEpoch: 1. Thereafter increment it whenever runner execution code changes; server-only changes may retain the current epoch.

epochMismatchDelay defaults to 2m. Maintain delay × (MaxDeliver - 1) above the worst observed cold-readiness time for the replacement fleet. For a latency-sensitive KEDA rollout, temporarily raise minReplicaCount; that buys capacity but does not replace the fence.

An epoch rollback is intentionally rejected. Do not use helm rollback to a release with a lower value. Restore an older, fence-aware image as a new release with a higher epoch. Messages from earlier epochs remain accepted.

Break glass for a mis-set or corrupt mark. The high-water mark is deliberately monotonic, so normal rollback cannot repair a typo such as runnerEpoch: 100 instead of 10; a non-decimal value stored in the KV also makes every process fail while reading it. Quiesce the installation before resetting it: suspend HPA/KEDA reconciliation, scale both the server and runner Deployments to zero, and verify that no pod capable of claiming the mark remains. Then delete only the mark key:

nats kv del <rollout-kv-bucket> epoch.high-water

The default rollout bucket is iterion-runner-rollout. When ITERION_NATS_KV_BUCKET / config.nats.kvBucket is customised, the rollout bucket is <kvBucket>-rollout. Correct runnerEpoch in the deployment source before restoring autoscaling and replicas; otherwise the bad value is claimed again immediately. This procedure intentionally stops new submissions and executions, so use it only when rolling forward with a higher epoch is not possible.

Autoscaling, node turnover and a deploy all arrive as the same signal, SIGTERM, so one mechanism covers them all. What happens to a run a runner is executing is governed by config.runner.drainMode:

complete (default — lame-duck): on SIGTERM the runner stops claiming new runs but lets its in-flight run finish before exiting. New pods (already up) serve new runs; the draining pod holds its NATS KV lease so nothing double-claims. A deploy interrupts nothing — the run runs to completion, even if it takes hours. The bound is runner.terminationGracePeriodSeconds (the k8s hard stop before SIGKILL) and config.runner.drainTimeout (the internal ceiling, default 8h): a run exceeding it is capped — checkpointed and auto-resumed on another pod.
interrupt: on SIGTERM the runner cancels its in-flight run immediately, checkpoints it, and it auto-resumes on a healthy pod from the last completed node. The fast path for an urgent (e.g. security) deploy that must not wait for long runs.

Either way an interrupted run (lame-duck cap, interrupt mode, lost heartbeat, or an eviction the grace window can't cover) is promoted to failed_resumable and redelivered — it auto-resumes with no operator action. Only an operator cancel stays terminal cancelled, and it wins the race: a resumable failure never overwrites it.

Two consequences worth knowing before relying on this:

Resume restarts the interrupted node, not the one after it. A node whose side effects had already landed (a push, a posted comment) runs again. That is the standing resume contract (docs/resume.md), now reached automatically rather than by an operator's decision — so a node that must not repeat needs to be idempotent.
The runner suppresses its own completion notification for an interrupted run, but the usernotify reconciliation sweep re-derives outcomes from the persisted status every 2 minutes and reads failed_resumable as a failure. If the redelivery is slow (a rolling restart is exactly when it would be), a user can still get one "run failed" push for a run that then resumes silently.

Invariants when raising the lame-duck window:

runner.terminationGracePeriodSeconds ≥ config.runner.drainTimeout + a couple of minutes of checkpoint margin (else k8s SIGKILLs a capped run before it checkpoints — it still recovers via the orphan sweeper, just ~10 min slower).
runner.progressDeadlineSeconds above the grace period: a lame-duck rollout stays Progressing until the last old pod drains (possibly hours), and the default 600s deadline would otherwise mark the Deployment degraded while the new pods already serve. ArgoCD users: the sync will likewise show Progressing until the drain completes.

Verified in production, 2026-08-01. The same signal, before and after: a KEDA scale-down killed run 019fbd98 21 minutes into a campaign and it came back cancelled; a kubectl delete pod on the runner holding run 019fbdec left the pod Running with a deletionTimestamp 2h05 out — it finished its run (240.8s, status finished) and only then exited.

What a long window costs. A terminating pod keeps its node slot and its resource requests for the whole drain, so:

a scale-down→scale-up cycle inside the window (KEDA's cooldown is 60s) can hold well above maxReplicas worth of footprint for hours;
kubectl drain and cluster-autoscaler node removal block for up to the grace period per runner pod;
two runner generations talk to one Mongo and one NATS for the length of the window, so a change to run-doc shape, checkpoint semantics or event payloads must stay compatible across it — hours, not the 90 seconds this used to be;
a genuinely broken rollout takes grace + progressDeadline to be flagged.

Shorten config.runner.drainTimeout (and the grace with it) if any of those matter more than never interrupting a long run.

The mechanism lives in the runner's Shutdown (pkg/runner/loop.go); the run's context is decoupled from the fetch-loop context so stopping intake never cancels a live run.

Local contracts and admission maintenance

iterion contracts prints JSON describing only the executable: queue schema_version / min_schema_version, the local run format and Mongo run schema versions, and its build. It opens no connection and disables telemetry initialization. pkg/store has no compiled migration registry; the probe says so explicitly. Server/runner build skew is unknown from one executable: compare the output of the two deployed binaries offline, not a pod count.

A super-admin can read or replace the maintenance record in the existing platform_settings collection through GET / PUT /api/admin/settings/maintenance. Studio's platform settings screens show it read-only and refresh every ten seconds. To stop new launches:

curl -X PUT "$ITERION_URL/api/admin/settings/maintenance" \
  -H "Authorization: Bearer $ITERION_ACCESS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"fence_launches":true,"reason":"Queue schema rollout, change CHG-123"}'

Both fields are required on each PUT. The server stamps updated_at and updated_by; disarm with fence_launches:false and an explanatory reason. An absent record defaults to false. Every new launch admission reads the shared record directly (no TTL cache), including HTTP, board dispatcher, trigger, webhook and schedule admissions; forks are also refused because they create a new run. An armed fence returns HTTP 503 with error: maintenance_fence and a detail naming maintenance.fence_launches, the reason and its timestamp. A failed settings read also refuses new admissions with 503 until the store recovers.

For durable consumers this is a maintenance_hold, not a failed execution: trigger effects stay pending, board cards return to their launch column, and deferred webhooks keep their accepted payload. Bus events (including run outcomes) are materialized before execution: Mongo stores cloud effects; trigger-effects.json beside the native board stores local effects. One-shot label consumption records the owning effect ID in the same card write as the label removal, so a failed outbox marker is recoverable.

Schedules materialize an intention per nominal slot before advancing the calendar. Cloud intentions live in schedule_launch_intents, local intentions and calendar state in schedule-intents.json beside the native board. A hold at 09:00 can resume at 09:01 without waiting for tomorrow, including after a scheduler restart. The intention retains the guard result and payload; each retry re-evaluates overlap. Three held slots with overlap: skip launch only one run while it remains live. Cloud replicas exclude each other per schedule from the overlap check through launch persistence. The renewable lease's owner is checked in the same Mongo transaction as the run write: a paused worker whose lease was taken cannot persist behind its successor's overlap check. This requires a Mongo replica set and the scheduler and run stores to share the bootstrap Mongo client. Queue publication follows the committed run write. Local schedulers hold a file lock across journal reload, overlap, launch and outcome persistence; multiple coordinators cannot replay cached journal state. Repeated holds consume no failure attempts. After disarming, the ordinary worker/dispatcher sweeps launch the preserved work (subject to its other admission rules). A webhook with overlap: supersede selects predecessors, obtains one coherent admission (crediting verified live predecessor slots), then cancels. Parking a debounced payload does not cancel anything; its sweep obtains admission first. A fence armed after admission does not retroactively refuse that replacement.

These journals provide at-least-once recovery: a crash after a launch succeeds but before its completion marker is persisted can still replay the launch.

Existing runs continue: HTTP/WebSocket resumes and automatic retries remain allowed by this fence, while retaining their ordinary authorization and quota checks. Already admitted launches can finish admission; arming is not an atomic barrier against requests that passed the check concurrently, nor a queue/runner drain. Drain and migration orchestration are separate operations. Break glass: an authenticated super-admin bypasses the launch fence and can also disarm it through the same PUT endpoint.

The chart's values.schema.json checks top-level names and types, including subchart values and Helm's reserved global overlay; nested keys remain open. go test ./charts/iterion verifies that schema keys match template references and dependencies, accepts the committed values files and rejects misspelled top-level keys. After editing values or templates, run devbox run -- task chart:render-diff (or BASE=<commit>): it compares HEAD (or BASE) with the working tree, rendering each side with its own values-dev.yaml. Install chart dependencies first with task chart:dep-update; the comparison itself uses local archives and no network. Read the rendered diff: an empty result is explicitly reported and is not proof that an intended values change took effect. Both renders use the same local subchart archives; dependency upgrades require their own review.

Resume from a paused / failed run

Cloud-mode resume goes through the same NATS path as launch. The client passes the inline source of the workflow because the server pod has no operator filesystem:

bash

curl -X POST https://iterion.example.com/api/runs/$RUN_ID/resume \
  -H "Authorization: Bearer $ITERION_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source": "'"$(jq -Rs . workflow.bot)"'",
    "answers": {"approved": true},
    "force": false
  }'

force=true bypasses the workflow-hash mismatch guard (useful after a local fix). The runner reads the flag from the RunMessage and applies it to runtime.New(WithForceResume).

Migration from filesystem store

iterion migrate to-cloud uploads runs from a local .iterion/ directory into Mongo + S3. Idempotent (Mongo upserts + S3 PUT overwrites):

bash

ITERION_MONGO_URI=mongodb://...?replicaSet=rs0 \
ITERION_MONGO_DB=iterion \
ITERION_S3_ENDPOINT=https://s3.amazonaws.com \
ITERION_S3_BUCKET=iterion-prod \
ITERION_S3_REGION=eu-west-3 \
  iterion migrate to-cloud --store-dir ./.iterion --concurrency 4 --tenant <tenant-id> --owner <user-id>

Migration flags:

Flag	Description
`--store-dir <path>`	Filesystem `.iterion/` store to migrate from (default `.iterion`).
`--config <path>`	YAML config file for Mongo/S3 settings; environment variables take precedence.
`--dry-run`	Print what would be uploaded without writing to Mongo or S3.
`--concurrency <n>`	Number of parallel run uploads (default `4`).
`--tenant <id>`	Tenant ID assigned to migrated runs; required for multitenant cloud deployments.
`--owner <id>`	Optional owner user ID attributed to migrated runs.

Re-run safely if interrupted; runs already in Mongo are no-ops.

Smoke test (`task chart:kind`)

bash

devbox run -- task chart:kind

Renders + lints the chart, checks appVersion matches package.json. The install watch follows the rendered server Deployment's progressDeadlineSeconds (values-dev.yaml sets 1800 s; plus a 120 s margin, so 1920 s): a slow first pull of its one server pod exhausts the server's deadline budget, never a fixed client cap; the runner's longer drain deadline is not followed. For a real install + workflow exec, see the cloud-e2e CI job in .github/workflows/tests.yml.

Health endpoints

Path	Behaviour
`/healthz`	200 if the HTTP listener is up — covers liveness (and the startup probe)
`/readyz`	Pings Mongo + NATS + S3 + Valkey with a 1s sub-deadline each. 503 while draining, or when a critical dependency (Mongo) is down; a failing non-critical one reports `status: degraded` and still answers 200

The /readyz JSON response details which dependency is failing so the operator can debug from kubectl describe pod.

Only Mongo gates readiness because every replica pings the same backends: a critical check on a shared dependency converts a 15s blip into a fleet-wide outage — all pods leave the Service at once and the ingress 503s even the routes that never touch it. The runner exposes the same two paths on its metrics port, where /healthz reports the consume loop (busy vs wedged) rather than an open socket.

Sizing the exit: `probes.preStopDelaySeconds + config.server.shutdownDelay

30s teardown ≤ server.terminationGracePeriodSeconds`. Full reference: probes-and-graceful-shutdown.md.

Cloud deployment ​

Install ​

Topology ​

Sovereign runner pools ​

Prerequisites ​

Auth bundle and access tokens ​

Queue connection (NATS JetStream) ​

Shared replica state (Valkey / Redis) ​

NetworkPolicy egress ​

Whose X-Forwarded-For the server believes ​

The client's address has to reach the controller first ​

NATS monitoring endpoint (KEDA) ​

Metrics & dashboards ​

Tracing ​

Spreading the replicas: anti-affinity, spread constraints, disruption budgets ​

Pod turnover & in-flight runs ​

Pinning the image ​

Verifying that an infra-apps push landed (ArgoCD sync liveness) ​

Bumping the chart dependency (a template change reaches a deployment) ​

Generation-aware rollout ​

Local contracts and admission maintenance ​

Resume from a paused / failed run ​

Migration from filesystem store ​

Smoke test (task chart:kind) ​

Health endpoints ​

Cloud deployment

Install

Topology

Sovereign runner pools

Prerequisites

Auth bundle and access tokens

Queue connection (NATS JetStream)

Shared replica state (Valkey / Redis)

NetworkPolicy egress

Whose `X-Forwarded-For` the server believes

The client's address has to reach the controller first

NATS monitoring endpoint (KEDA)

Metrics & dashboards

Tracing

Spreading the replicas: anti-affinity, spread constraints, disruption budgets

Pod turnover & in-flight runs

Pinning the image

Verifying that an infra-apps push landed (ArgoCD sync liveness)

Bumping the chart dependency (a template change reaches a deployment)

Generation-aware rollout

Local contracts and admission maintenance

Resume from a paused / failed run

Migration from filesystem store

Smoke test (`task chart:kind`)

Health endpoints