Usage caps — stop below the provider's wall

An LLM subscription ("forfait") meters two rolling windows, five hours and seven days, and refuses every call once one is exhausted. iterion already survives that refusal: the run parks and a durable retry resumes it when the window reopens (scheduling.md). What it could not do was stop before the wall — and the wall is rarely where an operator wants to be, because the same subscription usually pays for their own interactive work. A fleet of bots that drives it to 100% takes the human down with it.

A usage cap is a percentage the operator chooses, enforced from the provider's own telemetry.

# The recommended posture. Two variables is the whole configuration.
export ITERION_USAGE_CAP_5H_PCT=85
export ITERION_USAGE_CAP_WEEK_PCT=75

Nothing is capped by default: an unset cap leaves runs bounded only by the provider, which is the historical behaviour.

The two postures

The windows fail differently, so they default to different postures.

Window	Default mode	What it does
five-hour	soft	Never interrupts work in flight; no NEW run starts
weekly	hard	Stops the run where it stands, and starts nothing new

A five-hour window refills soon, so killing a half-finished run to save minutes of quota trades a lot for a little. A weekly window that runs out on a Tuesday is a dead week, and the run that would have finished is worth less than the four days of headroom it would have eaten.

Either posture ends a capped run the same way: failed_resumable, with a durable retry armed for the instant the window reopens. A capped run is not a lost run; it is a run that waits. The cap reuses the provider-refusal path wholesale rather than inventing a recovery of its own.

Configuration

Variable	Values	Default
ITERION_USAGE_CAP_5H_PCT	0–100 (0/unset = no cap)	unset
ITERION_USAGE_CAP_5H_MODE	off | soft | hard	soft
ITERION_USAGE_CAP_WEEK_PCT	0–100 (0/unset = no cap)	unset
ITERION_USAGE_CAP_WEEK_MODE	off | soft | hard	hard
ITERION_USAGE_CAP	off disarms both caps	unset
ITERION_USAGE_CAP_TRUST_WINDOW	a Go duration (3h, 90m) — how long a stored dated reading is believed, see A reading is trusted for a bounded time	3h
ITERION_USAGE_CAP_REFUSAL_REST_MAX	a Go duration, or off — the ceiling of the escalating rest an account-level refusal earns, see A repeatedly-refused credential rests longer	6h

A malformed value refuses to start rather than falling back to no cap: every wrong answer here fails open, and a guard silently disabled by a typo is the failure the feature exists to prevent. That includes the trust window, and it holds under ITERION_USAGE_CAP=off too: the window also bounds the credential-skip evidence, which the kill switch does not disarm.

There is deliberately no per-run flag and no DSL field. The cap protects a credential and the deployment that owns it, not a run — a bot able to lift the guard would not be a guard. ITERION_USAGE_CAP=off is the escape hatch, and it belongs to whoever runs the deployment.

Changing the caps at runtime (no restart)

The env vars above are defaults. On a cloud deployment the two percentages are also a platform-scoped runtime-settings record (Mongo platform_settings, same tier as the platform LLM credentials), mutable through the super-admin API — so retuning a cap in production is one call, not a kubectl set env on two deployments plus a rolling restart:

iterion remote admin caps                       # record + env + EFFECTIVE + source
iterion remote admin caps set --five-hour 80 --week 70
iterion remote admin caps set --clear-week      # back to the env default

The raw surface is GET/PUT /api/admin/settings/usage-caps (super-admin, the same guard as the platform LLM-credential routes). The PUT has merge semantics: a field present with a number sets that override, present with null clears it, absent leaves it untouched — a call naming one window can never silently clear the other. Values must be integers 0–100 (0 = no cap); anything else — including an unknown field name — is rejected 400 with the reason. Every update lands in the platform audit log (platform.settings.usage_caps.updated) with old value, new value and the caller.

Propagation bound: ≤ 30 seconds. Both enforcement points — the server's launch-time pre-flight and the runner's claim pre-flight + mid-run guard — resolve the effective policy through a TTL-cached lookup (usagecap.Resolver, 30s TTL), re-read per evaluation. One DB record is the single source of truth for every replica of both deployments, which ends the class of divergence where one deployment rolled with a new env value and the other did not. A run already in flight picks a tightened cap up at its next reading, within the same bound. The pod that served the update is coherent immediately (its cache is invalidated in the handler).

Semantics that stay put:

• Env fallback. No record (or a cleared field) → the env value applies. A deployment that never touches the API behaves exactly as its env vars say.
• Only the percentages are runtime-mutable. The soft/hard modes and the ITERION_USAGE_CAP kill switch stay env-only: they encode the deployment's enforcement posture, and a posture change should be witnessed by a deploy. In particular the kill switch wins — with ITERION_USAGE_CAP=off, a DB percentage stays inert, so a runtime write can never re-arm a guard the operator explicitly disarmed.
• Verification without DB access. /healthz (and /readyz) echo the EFFECTIVE policy plus a usage_cap_source marker (env, db or db+env) — curl it after a change and watch the new number appear within the propagation bound. This is also the FIRST diagnostic for "a scheduled bot posted nothing this morning": every LLM-bearing run failing on usage cap: <window> at N% ≥ M% while zero-LLM runs (collectors) pass is the cap's signature, and /healthz names the ceiling in one curl. Grep the usage cap: substring, NOT rate_limited: a workflow whose every path reaches a model is refused before its first node by the runner pre-flight and carries the bare reason (no rate_limited prefix, no node id), while a workflow with a model-free path is let through and stops mid-run with rate_limited (<backend>): usage cap: …. Measured on the 2026-08-31 Vigie outage, where a 70% DB record was the whole story — and where the morning's first signal, the 04:00 docs-refresh run, was the pre-flight shape.
• Don't wait for the silent morning: wire the operator webhook. With ITERION_ALERTS_WEBHOOK_URL set on the server deployment, every run that parks failed_resumable (usage cap, provider window — with the armed retry's reset ETA) or fails hard produces ONE message on the webhook (Mattermost/Slack {"text": ...} shape), deduped across replicas and backed by a 2-minute reconciliation sweep, with a /runs/<id> deep link. The five silent Monday digests would have been five messages at 06:0x instead of a manual discovery hours later. (This is the cloud alert.OpsDispatcher; the same env var also feeds the in-process alert Manager for local runs.)
• Settings reads fail toward the last-known value (env defaults before the first successful read), retried once per TTL window: a settings-store blip changes nothing abruptly in either direction.

Which windows a cap governs

The provider reports six windows. five_hour is the 5h cap; seven_day, seven_day_opus, seven_day_sonnet and seven_day_overage_included are all governed by the weekly cap — a run refused on the per-model weekly sub-limit is refused, whatever the all-models number says. overage is not capped here: it is metered money, not subscription quota, and --max-cost-usd is what bounds money.

Where the numbers come from

Claude Code emits a rate_limit_event on its stream-json output whenever the provider's usage numbers move, carrying {status, rateLimitType, utilization, resetsAt}. utilization is a fraction (0..1); resetsAt is Unix seconds. That is the only place a subscription's remaining headroom is observable from outside the provider — the metered API returns anthropic-ratelimit-* headers, but a CLI-driven session hides them, and GET /api/oauth/usage requires a user:profile scope that a claude setup-token credential does not carry.

Consequences worth knowing:

• The cap is claude_code-shaped today. Other backends have no equivalent telemetry surface; a run on claw or pi is not capped.
• One event names one window. A rate_limit_event carries a single rateLimitType, emitted when that window's numbers move — so a session refreshes the windows the CLI happens to report, not all of them, and a run refused before its first call (the pre-flights below) refreshes nothing at all.
• A reading expires at its own reset instant. Past it the window has rolled over and the number describes a window that no longer exists, so a stale reading stops blocking by itself — no sweeper.
• A reading is also trusted for a bounded time after it was observed — the next section.
• Recording is not enforcing. Readings are collected whether or not a cap is configured. They are the only input the credential-tier skips have (a fair-usage refusal, a rejected credential — frequency / auth / spend, which no operator cap governs), so a deployment that never asked for a ceiling would otherwise send every run into the same wall. A run carries the observing guard as soon as there is a ledger to publish to; with no cap configured it blocks nothing.

A reading is trusted for a bounded time

The premise "a window's utilization cannot drop before its reset" is false for this provider: it has reset every window early, out of cycle. On 2026-09-04 the seven-day windows of the deployment's forfaits went to 0% at the provider while the ledger still held 93–99% readings taken hours earlier, each carrying a reset instant three to four days out. Every credential walk skipped both forfaits on them, and every claude_code run was refused at admission — including the revi/review merge gate of two PRs, which then read as "review in progress" forever. The lock was self-sustaining: the only writer of a fresh reading is a live session, and the refusal is what prevented one. Nothing could recover on its own before the recorded reset, four days later.

So a reading is authoritative only for ITERION_USAGE_CAP_TRUST_WINDOW (default 3h) after it was observed, whatever its reset instant says. Past that it is suggestive, not binding: the walk lets the credential through, the pre-flight admits the run, and the run's own session re-measures the window within one call. If the wall is really still there, that costs one call and one park — the ordinary wait, with the retry armed for the reset — which is the price of never locking a credential out through a reset the ledger cannot see. Readings with no reset instant (a relayed refusal, a dead credential) are not bounded by the trust window at all — it exists because a dated window can roll over early, which an account-level refusal cannot do — and keep their own 1h staleness bound instead, escalating with the streak (next section).

The same bound governs both consumers of the ledger — the cap pre-flights and the credential-skip evidence — because both read the same readings and must forget a pre-reset one at the same moment.

The launch walk asks the provider instead of guessing. When a forfait's stored readings are stale-but-suggestive — past the trust window, window not rolled over, and saying "closed" when they were taken — the cloud publisher re-measures the credential at Anthropic's OAuth usage endpoint with the forfait's own token (pkg/backend/forfait.FetchWindows), records every window it reports under the same key the runner meters, and decides on that: the forfait is skipped when the wall is real and granted when the window reset early, with no pod spent either way. Best effort and bounded (5s): a credential the endpoint refuses (a claude setup-token lacks the user:profile scope and gets 403), a network error or a malformed body each cost one Info line and fall back to trusting the credential — the trust-window behaviour. Fresh readings and low stale readings never trigger a round trip.

A repeatedly-refused credential rests longer

Three refusals carry no reset instant, because the provider relays them as text rather than as window telemetry: auth (the credential itself was rejected), frequency (a fair-usage limit on the request rate) and spend (the account's own money ceiling). Nothing but the 1h staleness bound ever expires them — so a credential the provider has frozen for days was asked again every hour, forever: one pod and one parked run each time, on a condition only a human can end.

The rest now escalates with the streak. The ledger counts how many times in a row a credential was refused on a window (the store does it, not the caller: one pod sees one refusal, and several pods write the same ledger), and the bound doubles per refusal — 1h, 2h, 4h — capped at ITERION_USAGE_CAP_REFUSAL_REST_MAX (default 6h, so a frozen account is probed four times a day instead of twenty-four). Set it to off for the old flat 1h re-probe.

It stays a rest, not a lock, on three counts: the ceiling is bounded, so a re-probe always happens; the streak resets to zero the moment the credential serves a call; and rotating the credential opens a fresh meter key anyway (see Rotating a credential resets its meter). To cut the wait short after fixing an account in place — same token, same fingerprint — clear its readings by hand, below.

Forgetting a credential's readings by hand

An operator who knows a reset happened does not have to wait out the trust window, and should not raise the global cap to unstick one fingerprint (that lifts the guard for every tenant and every bot). The scalpel clears one credential's readings, under every key it was metered with, and leaves the caps alone:

iterion remote admin usage-readings clear e4ecd2283afb305f
# → {"fingerprint":"e4ecd2283afb305f","deleted":2}

The fingerprint is the credential's audit identity — shown on the key and connection views, and in the server's SKIPPED … fp= / AT ITS CEILING lines. Raw surface: DELETE /api/admin/usage-readings/{fingerprint} (super-admin). Every call lands in the platform audit log (platform.usage_readings.cleared, with the count). Afterwards the credential reads "nothing learned yet": the next run is admitted and re-measures.

What it looks like when it fires

• a usage_cap event on the run's timeline (window, percent, cap, mode, stopped, resets_at) — the only thing that distinguishes "the provider refused us" from "we stopped ourselves";
• a warn line in run.log: usage cap: seven_day window at 76% ≥ 75% (week, hard), resets 2026-08-18T21:00:00Z;
• the run's own error, carrying the same sentence;
• then the ordinary wait: run_retry_scheduled with retry_after at the window's reopening, and run_auto_resumed when it fires.

Two shapes, three situations — and the number is what tells them apart

The warn line comes in two shapes, and the shape alone does not say who refused. Reading it as if it did is what wastes the time this page exists to save.

usage cap: provider rejected on the seven_day window (week cap 85%, hard), resets …

The provider refused, and said so without a number. evaluate picks this wording only when the reading's status is rejected AND it carries no utilization figure. The (week cap 85%, hard) in parentheses names the policy in force, not the reason — which is what makes it easy to misread. Touching the cap changes nothing: the call never got as far as the cap.

usage cap: seven_day window at 76% ≥ 75% (week, hard), resets …

This one is ambiguous, and the percentage resolves it.

• Below 100% — we stopped ourselves. The provider is still serving and iterion refused because its own telemetry crossed the operator's percentage. Raising the cap (iterion remote admin caps set --week …) lets work through immediately.
• At or above 100% — the provider refused, in the same words. A reading can be rejected and still carry a number, and it then renders in this shape; the Anthropic forfait probe marks every window at ≥100% rejected while keeping the figure. A rejected reading is blocked at any cap — the guard reads if !rejected && pct < cap, so even a cap of 100% refuses it. Raise nothing; the reset instant is the only lever.

So the rule to carry is not "which sentence is it" but "is the reading rejected", and the two things that answer it are the missing number and the percentage at or above 100%.

Two consequences worth having in mind before touching anything:

• A refusal can appear while the provider's own dashboard shows headroom on the other window. The weekly and five-hour walls are independent, and a run refused on one says nothing about the other.
• iterion remote admin usage-readings clear will happily forget a refusal too, and it buys nothing: the next run is admitted and then refused at the call instead of at admission. That command earns its keep on the opposite shape — a dated reading trusted past a reset the ledger could not see (see A reading is trusted for a bounded time). Reach for it when the provider's dashboard disagrees with iterion, not when the provider itself is saying no.

Measured on 2026-09-08: fifteen review runs refused over an hour on a deployment whose caps had just been lowered — the caps were not the cause, and every one of the fifteen resumed and delivered by itself once the window reopened.

What a cap does NOT stop

A run is only refused in advance when it could not possibly avoid spending. The cap governs a model subscription, and it blocks at launch only if EVERY path from the workflow's entry to a terminal passes through something that can call a model — an agent, a judge, an llm router, a model-answered human node, an agent recovery rung, a subbot, a supervisor.

If any model-free path exists, the run starts and the mid-run guard stops it at the actual call. That costs a pod and a clone in the worst case, and it is the price of not refusing work that would never have been billed.

The distinction is not cosmetic. A zero-LLM run is often the half of a bot that gathers — and gathered material is not recoverable by retrying later. Vigie's collect mode polls feeds into a queue; a feed serves a short window and does not remember what nobody fetched, so every refused collect is material permanently gone, while the digest half it feeds waits on a queue that stays empty. Between 2026-08-17 and 2026-08-18 that is exactly what happened.

Why "every path" and not "contains a model node". A two-mode bot carries both halves in ONE .bot: Vigie's collect polls feeds with tool nodes, digest synthesises with an agent, and a router picks between them from a field the plan node produces at RUNTIME — not from a var, so no launch-time analysis can predict it. A predicate asking merely "does this graph contain an agent?" answers yes for both halves and refuses the collect half too. That is exactly the defect that silenced the production veille, and shipping the weaker predicate first did not fix it.

The predicate is ir.Workflow.AlwaysReachesLLM, walking forward from the entry and treating a model-calling node as a wall: reaching a terminal without hitting one proves a model-free path exists. It stays conservative in the direction that matters — an unwalkable graph, a missing entry, a dangling edge or a supervisor all answer "true", keeping today's refusal rather than opening the gate. (UsesLLM still exists for the plain "does this graph contain one?" question; the two deliberately disagree on a two-mode bot, which is the whole point.)

Both pre-flights apply it: the cloud runner's (which has the compiled workflow in hand) and the local launch path's (which compiles only when the cap is blocking, so the common case pays nothing). The mid-run guard stays armed in both cases, so a workflow that turns out to spend anyway is still stopped at the call.

And only when it could spend the wire the cap meters. The readings come from the claude_code delegate's session telemetry and nowhere else, and the pre-flight key is built from the run's Anthropic-wire credentials (or the platform's). A run whose every route is pinned off that wire — both LLM nodes on claw + openai/…, a codex bot — cannot spend the capped subscription, and parking it for the anthropic weekly reset strands it for nothing: a fully pinned two-node rite froze for five days that way while its single-node sibling on the identical pin ran (#668). The cloud runner's pre-flight asks model.AnthropicWireReachable under the launch's own overrides, and lets the run through when the answer is no. Every uncertainty answers "reachable" and keeps the guard armed: an empty, auto or claude_code backend, claw/pi with a provider it cannot resolve, any anthropic/zai hint on any backend, a model-answering node with no LLMFields.

Primary routes only. A fallbacks: route or a run-level --fallback stage onto the wire does not arm the pre-flight: the primary can carry the whole run without ever touching the wire, and the rescue route fires only on a failure the mid-run guard and the delegate's own usage-window classification already refuse at dispatch. Refusing such a run in advance would park work that could not possibly spend the capped subscription — the one thing the pre-flight promises not to do. The credential the rescue route needs is still sealed into the run: the wants derivation widens on the chain, it is only this guard that ignores it.

Cloud

Every pod sees only its own session, so readings are shared through the usage_windows collection (one document per credential and window, newest wins). A claimed run consults it before cloning a repo or starting a container and parks for free when there is no headroom — otherwise each pod would rediscover the ceiling by spending against it.

The ledger is keyed per credential: a tenant that brought its own subscription is never blocked by what another tenant spent, and runs falling back to the deployment's own credential share one meter, which is correct — they really are one subscription.

Rotating a credential resets its meter — on purpose

The key names the CREDENTIAL, not the slot it sits in:

claude_code|tenant:<id>              # legacy, still valid, expires in place
claude_code|tenant:<id>|fp:aaaa1111  # the meter of ONE credential

fp: is the audit fingerprint of the credential the run actually spends, stamped when a human connects it and preserved across the automatic token refreshes (which rewrite the tokens of the same subscription). Without it, posting a fresh token over an exhausted one inherited the old account's reading — legitimately fresh until its own reset instant, up to seven days out — and parked every run of a credential with a full week available. That happened.

So: connect a new credential and the pre-flight finds nothing, fails open, and republishes the new account's real readings at the first call. The old readings expire in place under their orphaned key (the collection has a 14-day TTL), and nothing needs migrating. This applies to every tier — a tenant's forfait, a pool donor's lent subscription, and the deployment's own platform forfait, where the meter is fleet-wide.

Two boundaries worth knowing:

• Re-connecting the SAME Anthropic subscription also opens a fresh meter. A credentials.json carries no account id, so iterion cannot tell "the same subscription again" from "a different one" (Codex's auth.json does carry account_id, and is metered per account). The reflex of re-pasting credentials when a token looks broken therefore forgets what was measured. It fails open — one run rediscovers the wall and republishes it — and the mid-run guard is the backstop.
• A local CLI/studio run is metered per machine, not per credential. Rotating your own credential inside a long-lived iterion studio process keeps reading the replaced account's window until it resets; restart the process to clear it.

The pre-flight fails open on every uncertainty (no ledger, an unreadable ledger, nothing measured yet, a rolled-over reading). A cap exists to protect a subscription from a fleet, not to strand the fleet on a bookkeeping outage; the in-run guard still stands behind it, so failing open costs one call.

A local CLI or studio run keeps the same policy on a process-local ledger, and a second run in the same process starts already knowing what the first one measured. There the launch is refused outright (429, with the reason and the reopening instant) rather than parked: the operator is present, so an immediate answer beats a queued one.

Emergency brake

The cap governs iterion. To stop everything on a cloud deployment, including runs already claimed, freeze the runner instead — queued work piles up in NATS and resumes intact on unfreeze:

kubectl -n iterion annotate scaledobject iterion-runner \
  autoscaling.keda.sh/paused-replicas="0" --overwrite   # freeze
kubectl -n iterion annotate scaledobject iterion-runner \
  autoscaling.keda.sh/paused-replicas-                  # thaw

In-flight runs survive the freeze: a scaled-down pod SIGTERMs into the lame-duck drain (ErrRunInterrupted → failed_resumable, auto-resumed when capacity returns), and even a SIGKILL leaves the run to the orphan sweeper, which flips it to failed_resumable within minutes. Cancel first only when you do NOT want the run to come back after the thaw — a cancel is terminal until an explicit resume.