ADR-155: reserved capacity is a durable allocation — one ledger, one atomic acquire, a settle and a sweep
Status: accepted — the decisions are taken: the operator decided on 2026-11-03 that the shape stands as written and that a reservation's identity is tier-qualified (decision 11). Design only — nothing below is built; the units are tracked under #950. Date: 2026-11-03 Issue: SocialGouv/iterion#950
Context
On 2026-09-08 three teams held what read as three credentials and were one Anthropic account. When the shared five-hour window closed at 06:14Z every claude_code run was refused — eight parked in six minutes, fourteen within the hour. No cap had been exceeded and no quota breached: every individual run stayed under its own ceiling all the way down. Review simply had nothing held for it.
Two units have landed since, and both are accounting:
- #1069 made the repository an accounting dimension.
credusage.KeycarriesRepoID;docIDleaves an empty one out entirely, so no stored credential-month is re-identified by the deploy that adds it;ListByRepois the read the dimension adds, and every listing that predates it sums the repositories back together. - #1099 shipped
pkg/budgetfloor— a measured PREFERENCE. The package doc states the bound first: nothing is allocated.WindowCeiling,OtherReserved,OtherReservedSlotsandOtherReservedUSDare arithmetic over a ceiling the operator already set, and the gates consult them to lower that ceiling for every bot no reservation names.
What a preference cannot do, read on 70b5b1972:
- Admission acquires nothing.
gateConcurrencysubtracts the held slots, callsCountActiveRunsByTenant, compares, returns: two replicas both readcap - held - 1and both admit.gateMonthlyCapssubtracts the held dollars and hands the remainder toorgusage.AllowRun, whose run arm is atomic ($inc, post-increment read, rollback on refusal) and whose cost arm readscost_usd_millis— a counterAddSpendaccumulates after an attempt ends. A reservation is therefore judged against spend already reported, never against spend already promised. - A run admitted under the bias is not held to it. The publisher computes the bot's lowered policy (
capPolicyFor→botCapPolicy{effective, deployment, heldOut}) and keeps it to itself; the runner'susageGuardForre-readsr.cfg.UsageCapSource, the deployment-wide one.grep -rn budgetfloor pkg/runner/returns nothing: the runner cannot reconstruct the fact, it has no bot→reservation resolver at all. - The pool is unreachable for a workload that holds a credential.
walkCredentialPlandeactivatescredentialTierPoolwhenhasCredentials()—holdsDefaultLLMCredential(bundle)in live resolution,len(x.api)+len(x.oauth) > 0in the preview — and the tier's own BODY then re-asks the question aslen(bundle.APIKeys) == 0 && len(bundle.OAuthCredentials) == 0, a second guard the preview does not carry. A floor-HELD credential is never sealed, so the bundle is empty at both and an unreserved bot the reserve held out does reach the pool — deliberate, and the restore step refuses to hand the held credential back. What stays inexpressible is the ticket's own ask: Revi on a pool of forfaits while Revi's team holds a healthy credential. - Execution capacity is not reservable at all.
MaxAckPendingis one process-wide int applied verbatim to the shared durable and to every pool consumer (poolConsumerConfig); its only knob isITERION_NATS_MAX_ACK_PENDING, andgrep -rni maxackpending charts/returns nothing — the chart cannot express it. KEDA is oneScaledObjectfor the shared runner deployment; pool lanes are fixed-replica. - The meter has no workload dimension. A repo quota expressed as a share of
review-pr's reservation (ReserveSharePercent+ShareOfBot) is consumed by any workload, becausecredusage.Keycarries no bot. - Policy absent and policy unavailable are one answer. The
platformcfgresolver is process-local — one TTL cache per replica,Invalidatelocal — and a cold-start read failure leaves the cache nil, whichbudgetFloorPolicymaps to the zero policy: reserves nothing.
D31 widened the ask past the credential budget: execution capacity — NATS MaxAckPending, the KEDA replica ceiling — is reservable too.
The 2026-09-10 cross-family plan review (14 findings) is the other input, and the numbers below are the comment's own "Follow-ups recorded from the review" list — not its three verified-in-source bullets. Closed on main: the first verified bullet (the walk's restore step handing a held credential back — closed at the class in skipped_creds.go, keyed by fingerprint), which is not follow-up 1; finding 2's first half (an inline source's bot label is warned about and ignored, and an unverified label is judged as naming no bot); finding 5 (accepted and documented as by-design on forge.ProjectPathFromCloneURL). Live, each against what closes it: finding 1 — the pool does not participate in arbitration — decision 6 and unit 5; finding 3, decision 9; findings 4 and 6, decisions 7 and 10; finding 7, decision 8 and unit 10; finding 8, unit 9; finding 2's second half, decision 11 and unit 11.
Options considered
Where the allocation lives.
- A — generalise
pkg/credpool.Ledger. It is the closest thing the repo has:Reserve/Renew/AddSpend,LiveCommitmentderived from the open leases,decide()judgingsettled + outstanding,ReleaseExpiredswept every two minutes. Rejected, and the reason is the keystone of this record: it closes the overshoot, not the multi-replica race. Its concurrency axis is a TOCTOU —live LiveCommitmentis read in a separate round trip by the broker and passed in as a parameter, so two replicas both seeRuns = N-1and both admit — and theLedgerdoc says so outright: "the daily bucket is the atomic one … the weekly cap and the concurrency reading are checked around it and are soft by nature".decide()has norequestedterm either. Cloning that shape faithfully would leave the two-replica witness red after the fix. - B — grow
pkg/budgetfloorinto a stateful package. Cheapest diff, and it breaks the property its own doc makes load-bearing: the package is a stdlib leaf precisely so the launch gate and the credential walk can import it without inverting the graph. Rejected. - C — a new leaf
pkg/capacityledger, withcredpool.Ledgerandorgusage.Counterbecoming adapters over it. One tracker, thepkg/errtrackrule generalised, and the Nth-variant test of philosophy 2: the next reservable subject implements the seam instead of growing an engineif. It costs two adapters and a conformance suite that pins both twins. Chosen.
How the acquire becomes atomic.
- A — derive
Outstandingfrom the live allocations. Whatcredpooldoes, and for a good reason (an abandoned run must never leave a slot permanently consumed). Rejected as the admission path: a derivation is a second query, which is the window this record exists to close. - B —
$incan outstanding counter in ONE write, read the post-increment document, compensate the increment on refusal. The only atomic admission pattern this repo has (orgusage.AllowRun,credpool.MongoLedger.Reserve's day bucket), so an operator reasoning about one quota system reasons about all three. The cost is that crash-safety moves out of the data shape and into a sweep. Chosen. - C — insert an allocation document under a unique index and count in the same operation's result. Keeps per-allocation provenance, and asks Mongo for a count whose cost grows with the subject's history on the hot path. Rejected for admission; the per-allocation record is kept as the settle and sweep subject, not as the counter.
Who runs the reconcile net.
- A — elect it through
pkg/lease. Rejected as a correctness claim:pkg/lease's own doc calls it load reduction, "not a mutual-exclusion guarantee", and tells the caller that work which must never run twice needs a claim on the item itself. - B — tick on every replica, CAS on still-open per item. What
credPoolSweeperalready does, deliberately: "closing a lease is conditional on it still being open, so two replicas sweeping the same instant produce one close". Election would only reduce cost. Chosen, with the per-item CAS named as the correctness carrier.
How the pool becomes reachable for a workload that holds a credential.
- A — automatic for any floor-held credential. This is already today's side effect (a held credential is never sealed, so the bundle is empty at the pool tier). It does not serve the reserved workload itself, which is the ask. Rejected as an answer.
- B — an explicit per-reservation opt-in, default OFF. Never changes an existing deployment's posture, greppable, and the donor's audience, limits and ledger still decide. Chosen.
- C — a deployment-wide switch. One dial, and it hands every workload the donors' capacity rather than the one the operator named. Rejected.
The concurrency rule, once allocation is exact.
- A — switch to the exact rule (count the holders' own active runs, subtract only the UNUSED part of each reserve) and keep today's conservative rule as the fallback when the ledger is unreachable.
gateConcurrencydocuments the over-refusal it accepts today — cap 3, reserve 2, the reviewer running 2, the third slot stays unused — and names the missing input: a per-bot active count, which the allocation provides. Chosen. - B — keep the conservative rule. No behaviour change for anyone, and a deployment capped at 3 with a reserve of 2 wastes a slot forever. Rejected.
- C — a policy flag. A fifth dial on a surface that already has four axes, for a rule nobody wants to be wrong. Rejected.
What a reservation names (finding 2's second half).
- A — keep the bare bot name as an accepted bound. No migration, and the escape hatch is the operator re-pointing the reservation. Rejected: the name is chosen by whichever tenant registers it — botsource rows are unique per
(tenant_id, slug)(tenant_slug_unique), and resolution serves a team's own row over the platform's by design — so re-pointing moves the reservation to another name any team can register next. - B — refuse a team bot whose name a reservation holds. Closes the homonym at registration, and takes away the fork the resolver is built to serve (a team that forks a bot gets its fork on every launch surface), on a deployment that may have configured no reserve worth the refusal. Rejected (philosophy 1).
- C — qualify the identity with the tier that serves it, and resolve before judging. A reservation names the bot AND the tier; the gate judges the identity the server's own resolution produced, never the request's label. The vocabulary exists already:
launchBot.Tier()maps the resolution ontostore.BotSourceTier*, whichRunMessageand the run document already carry (Run.BotSourceTier). Chosen by the operator on 2026-11-03.
Decision
- A reservation is a durable ALLOCATION on a named capacity subject. The subject is
(kind, id, axis)— a credential window, a tenant's monthly dollars, a tenant's concurrency slots, a runner pool's in-flight ceiling — and admission judgessettled + outstandingagainst it, never a reporting counter, whereoutstandingis the POST-ADMIT value and so already carriesrequested: the repo's ownrunsAfterAdmitconvention (Limits.Denyinpkg/credpool/store.go— "the day's run count INCLUDING the admission being judged" — reached post-increment by the Mongo ledger andorgusage.AllowRun), compared strictly, so a cap of N admits exactly N. An explicitrequestedterm belongs only whereoutstandingEXCLUDES the asker, ascredpool's derivedLiveCommitmentdoes throughexcludeRunID. Adding it on top of a post-increment counter counts the asker twice, and the subject's usable capacity shrinks by one ask: a cap of N admits N-1 unit allocations, a $100 reserve stops $r short where $r is the next ask, and a resume whose own allocation is still outstanding is refused by itself — the regressionexcludeRunID's own note records. "Reserved" stops meaning "the others' ceiling is lower" and starts meaning "this much is held, and something holds it". pkg/capacityledgeris a new leaf package — leaf by its POSITION in the graph, not by an empty dependency set. Its judgement core imports stdlib pluspkg/clockand nothing else, the rulepkg/budgetfloorholds literally (context,fmt,sort,strings,time), while the Mongo twin beside it addspkg/internal/mongoutiland the driver — exactly whatpkg/credusage,pkg/orgusageandpkg/leasealready are (pkg/usagecapis that shape pluspkg/secrets, for its env half). So the launch gate and the credential walk import it without inverting the graph, and this decision does not contradict the Mongo twin it asks for below.Reserve(subject, requested) → (allowance, AllocationID, DenyReason, error),Renew(a resume re-checks spend without consuming a second unit, the rulecredpool.Ledger.Renewalready states),Settle(AllocationID, actual),Release(AllocationID),Outstanding(subject),Expire(now, limit). ONE judgement function, over the POST-ADMIT total: each caller makes its count post-admit before calling it — the Mongo twin by reading its own$incback, the memory twin by addingrequesteditself, a DERIVED reading by adding it to an outstanding that excludes the asker — which is how bothcredpoolledgers reach the one copy ofLimits.Deny. The error is the last value on every call, as oncredpool.Ledger.Reserveandorgusage.Counter.AllowRun: an unreachable ledger is an error, never aDenyReason— encoding it as a refusal would invert decision 8's fail-open doctrine and is the silent fallback the conventions forbid. A capped subject with nothing left denies rather than granting zero: zero on the wire means "no ceiling". Memory and Mongo twins behind one conformance suite, the pattern ofpkg/credusage(NewMemoryCounter/NewMongoCounter),pkg/usagecap(MemStoreinstore.go,MongoStoreinmongo.go, onerunStoreConformanceover both) andpkg/lease.- The acquire is ONE write whose own post-state yields the count:
$incthe subject's outstanding counter byrequested, read the post-increment document, compensate with the mirror$incon refusal — the shape oforgusage.AllowRunand ofcredpool's day bucket. One round trip to admit, two to refuse, and the allow/deny decision itself is atomic:AllowRun's own words.requestedis named PER AXIS, because neither precedent has one to copy (orgusage.AllowRuntakesmaxRuns/maxCostMillisand no ask;credpool.decideDERIVES the allowance throughshareAcrossSlots): one slot on the countable axes, the run's clamped budget ask on the monthly dollars, the reserve's share on the window percentage. Incrementing by zero on the money axis would leave the counter holding none of the promise and hand two concurrent admissions the same remainder — the overshoot this record exists to close. Where the ledger grants LESS than the ask, the same path corrects the counter to what it promised, by the mirror$incofrequested - allowance: outstanding is never a different quantity from the allowance handed out. It is deliberately NOT a conditional (filtered) update —{outstanding: {$lte: cap - requested}}yields no count on a miss and cannot tell a full subject from an absent one, which is why the two precedents filter on_idalone.credpool.Ledgerandorgusage.Counterbecome adapters over it, so there is one tracker and not three; unit 2 carries them. - The sweeper is a correctness carrier, not a cost optimisation. Because outstanding is a counter and not a derivation, a pod killed mid-run holds its allocation until something expires it. The sweep runs on every replica, closes an allocation conditionally on it still being open (so two replicas produce one close), charges no invented spend, and is bounded per pass —
credPoolSweeper's shape, and its reasons. - The runner is held to the lowered policy. The publisher stamps
capPolicyFor'seffectivepolicy on theRunMessage;usageGuardForenforces the stamp. The fact travels with the result, written by the side that knows. The live re-read of the DEPLOYMENT cap stays, so a cap tightened mid-run still bites, and the run is held to the TIGHTER of the two: a stamp never LOOSENS what the runner's own source already caps. Without that rule the stamp has an uncapping arm —capPolicyForreturns a ZERObotCapPolicywhen the publisher has no cap policy wired, andusagecapreadsMaxPercent0 as "unenforced", so an empty stamp would be indistinguishable from a real lowered policy of nothing. Both ends have an unstamped arm and both fall back to the DEPLOYMENT policy — the UNLOWERED ceiling (capPolicyForlowerseffectiveonly, over adeploymentthat staysbase;usageGuardForreadsr.cfg.UsageCapSource) — so for that run the reserve is not held: the lax direction, and neither arm may be silent. A message with no stamp is one queued by the previous image of the single binary that publishes (cmd/iterion/server.go), and its window is RETENTION, not a drain:DefaultStreamMaxAge(24h) for a queued message,DefaultDLQMaxAge(7d) for a parked one, whichhandleDLQReplayre-publishes verbatim — the frozen payload, stamp absent, reached by an operator gesture rather than by a drain. That handler already loads the run doc and decodes the message, so it is where a re-stamp belongs; until it carries one, the seven days are an accepted bound, logged and counted. A STAMPED message on a runner that predates the reader is the long window — a separately rolled image — and it is decision 10's case: the reserve counts as unheld on the runner side until the fleet's floor clears the release that reads the stamp. Bothclawandclaude_codelanes read the stamp, or it is typed-refused for the one that does not (backend parity). - The pool becomes an arbitrated capacity tier. A reservation may be marked pool-eligible (default OFF), and then
credentialTierPoolstays active for that workload even though it holds its own credential. Eligibility is ONE predicate —poolEligible(bundle, botID)— called by the plan arm, by the tier body's own guard and by the preview. The body's guard is why flipping the plan arm alone changes nothing: the tier is reached and the grant refused at the acquire, while the preview reports the tier ACTIVE — the disagreement this decision exists to prevent, and a guard copied into the body is how it returns.budgetFloorWarningsgains its third advisory shape: a reservation marked pool-eligible on a deployment where no enabled pool admits its tenant holds nothing. - The meter carries the workload.
credusage.KeyandSpendgainBotIDand, beside it, decision 11's reservation tier (platformorteam; ateamtier's team is the key's ownTenantID, because the team tier serves a bot to the team that owns it alone) in a field of its own — neverKey.Tier, which already names the CREDENTIAL's resolution tier and shares the wordsteamandplatform.docIDleaves an empty workload out exactly as it does forRepoID, andaggregateReposbecomes ONE fold over both additive dimensions — the workload cleared beside the repo. The write half alone is not additive: it splits a credential-month into rows no pre-dimension listing can tell apart, sinceList,ListByFingerprintandListByTierall read through that fold (summed) andMonthlyUsageexposes no bot at all — the reporting outagedocID's own comment describes, in the lax direction.ListByRepokeeps its per-repository rows and folds the bots inside each. A repo's share of a reservation resolves against that reservation's consumption. Enforcement of the new dimension is armed only on a fleet verified to write it, by decision 10's predicate. A fleet that is not cleared keeps the pre-dimension reading — the repository's total — never the bot-attributed subset, because the subset is the lax direction: money leaks and nothing alerts (finding 6). - Policy unavailable is not policy absent. The resolver keeps its process-local TTL cache, and BOTH readers of that record stop reading an unreadable policy as "no reservations":
budgetFloorPolicyat the launch gate andcapPolicyForat the credential walk each map a nil record to the zero policy today, so fixing the gate alone would leave the window axis silently un-reserved on the very same cold start. One readiness accessor serves both. The two arms differ, and only one of them has a last-known value to hold a reserve on. A WARM cache already serves one: the resolver warns "settings read failed — serving last-known values" and returns the cached record, so the reserve is held on it and the conservative concurrency rule stands in while the ledger is unreachable. A COLD start that never resolved has NO policy:cachedstays nil,readystays false, andGetreturns a bare*T, so todaybudgetFloorPolicymaps nil to the zero policy and cannot tell "nothing written" from "never read". Unit 10 exposes that distinction —Resolver.readybecomes readable — and takes the admit direction: refusing every launch on a deployment that may have configured no reserve at all is the worse failure (philosophy 1), so the launch is admitted with no allocation and the fact is logged AND counted under its own reason detail. Fail-open stays the doctrine for a degraded store read —gateLaunch's doc comment owns it — made observable instead of silent. - Execution capacity is reservable per pool.
RunnerPoolgains an in-flight ceiling that overridescfg.MaxAckPendingfor that pool's consumer (poolConsumerConfigis the seam;0inheritscfg.MaxAckPending— the operator'sITERION_NATS_MAX_ACK_PENDING, whichapplyDefaultshas already resolved toDefaultMaxAckPendingonly when nothing was set, so no pool is re-capped below a global the operator raised, and the historic pin of1that serialised the whole fleet is the lesson this must not re-pay). The chart carriesITERION_NATS_MAX_ACK_PENDINGand the per-pool value, and renders oneScaledObjectper pool, gated on that pool's ownkeda.enabledso a cluster without KEDA still renders. The prod numbers are the operator's gesture, not this record's:values-prod.yamlkeepskeda.enabled: false, and the infra-apps commit lands on its own lane. - A rollout asymmetry is judged on the fleet's verified floor, and the unverified arm takes the CONSERVATIVE direction, never the lax one. Decisions 5 and 7 each add a field one component writes and another reads, across components that roll separately — the publisher ships in the server binary, the runners in their own image — so both cross this one rule rather than carrying a guard each. Cleared is a predicate, not a helper: at least one runner build observed inside
engineFloorLookback, every observed build orderable, and all of them at or above the release that ships the field — EXCEPT where the engine executes in this very process (cfg.Mode != "cloud", the marker the server already keys local mode on): there the writer IS the reader,appinfoanswers for both halves, and the arm is CLEARED. Reading "no observer wired" as unverified would unarm decision 7 forever on the one topology that knows its own build with certainty.s.engineFloordoes NOT answer the predicate — with no observer wired, on anObservedRunnerBuildserror and on an EMPTY observation it returns this server's OWN build as the floor, which on a current server reads as cleared, andsourcesdoes not separate those three either: each returns["server <self>"]byte-identically, so it tells observed from not-observed and nothing finer. Its one existing consumer PASSES an unorderable floor with a warning (requiresFloorGap'sEngineUnknownarm), the polarity this gate inverts. So the unit owes a readiness accessor besideengineFloorthat carries the ARM itself, with each unverified one witnessed. An unverified fleet is a WARNING and never a refusal (philosophy 1), and the conservative reading it falls back to is logged and counted, so a rollout is observable rather than silent. - A reservation's identity is tier-qualified, and the gate resolves the bot before it judges. This record first kept the bare name as an accepted bound:
Reservation.BotIDcarries no tier,sameIDfolds the name only, and the launch handler's firstgateLaunchtakes the request's label (launchBotLabel) beforeresolveBotTieredruns — so a tenant that registers a team bot namedreview-pris judged as the reserved workload, and consumes or is refused on the platform bot's band. The operator decided on 2026-11-03 to close it, and this decision replaces the bound. A reservation names its tier beside the bot:platformfor the deployment's bot — served by a platform override or the baked catalog, the two tiers the resolver falls to when the launching team has no row of that name — andteamfor a team's bot, which also names that team, because a team bot's name is unique only inside its tenant. A stored reservation that names no tier is read asplatform: every reservation written before this decision protects the deployment's bot, and that is the migration the operator decided.RepoQuota.ShareOfBotnames a reservation too, and is qualified by the same rule, andPolicy.Validate's one-reservation-per-bot check, the share's reservation lookup andPolicy.Bots()key on the qualified identity, so a platform reservation and ateamone of the same name coexist. The gate's subject is the RESOLVED identity — the canonical name and the tier of the row that served (launchBot.BotID,launchBot.Tier()) — and a continuation's is its run document's (Run.BotSourceTier; empty before the stamp, it is read throughBotSourceTenant:botsource.PlatformTenantIDisplatform, any other tenant isteamfor that tenant, empty isplatform). Only the arms that judge the LAUNCH against a reservation read that identity: concurrency and the monthly dollars at the launch gate, and the window reserve at the credential walk. The repository quota reads a reservation too, but the quota's own:gateRepoQuotahandsRepoCapnothing butsubj.Repo, andRepoCapresolves the quota'sShareOfBot— qualified above — whatever is launching, so it never reads the launch's identity. The record reads the operator's "before it judges" as before it judges the bot — its reading, put to him when #2980 is picked up: the arms that read no launch identity — the maintenance fence, the teamless and member-rung refusals, the team read and its suspend checks, the repository quota, and the launch-rate token, which therefore moves ahead of concurrency — keep their place ahead of the resolution, so a caller they refuse costs no resolution. The rate token does not bound the work of a launch refused AFTER the resolution: that refusal hands it back throughlaunchAdmission.rollback, as a monthly refusal does today (#1726, pinned byTestGateLaunch_RateTokenRefundedOnRollbackand the floor test's "a monthly refusal refunds only its rate token"), and a concurrency refusal never reached the token at all before this decision. What bounds that work is what the reservation arms are fed: the RAW resolution (resolveBotTieredRaw— the tier walk and, for a stored row, its materialization, which the refusal'slaunchBot.Cleanupremoves), never the cloud snapshot:resolveBotTieredis that call followed bysnapshotLaunchBotin cloud mode, and the snapshot runs only once the launch is admitted. The same qualified identity is what the exact concurrency rule's per-bot count and decision 7's workload dimension key on, so a team fork's runs and spend never read as the reservation's own.
Consequences
- The word "guarantee" becomes accurate on three axes and stays qualified on the fourth. Window, monthly dollars and concurrency slots are allocated; the provider's own capacity is still learned, not owned. The forward consult that exists (
usagecap.Latest+Preflightat resolution time) is conditional —meteredWindowrequires a metered backend and a fingerprinted slot — andStore.Latest's contract makes an unknown key read as "not blocked". So a credential with no reading in the current window has no forward estimate, only a bounded, self-healing read of the last observation. - Per-axis tenancy is unchanged. The window axis is deployment-wide while monthly dollars and concurrency are per-tenant; a reservation still never creates a cap that is not configured, and the holder still faces the plain cap. Both invariants are pinned by the existing floor tests.
- One quota key across forge hosts stays by design (finding 5): the slug is the meter's whole key, so two deployments'
acme/widgetsshare one quota identity. The escape hatch is the one the derivation already names — a caller holding the forge's own project path must prefer it. - A team's fork of a reserved bot is the team's workload (decision 11): launched under the reserved bot's name, it is judged as other work and faces the ceiling the reservation lowers, while bot resolution keeps its team-over-platform precedence unchanged. An operator who means to protect a team's own bot writes a
teamreservation naming that team. A label that resolves nowhere is refused before any arm judges it against a reservation, so it never borrows a band on the way to its refusal. - An opted-in workload that receives a pool grant loses the platform tier:
poolGranted()deactivatescredentialTierPlatformin the same plan. That is the existing rule and it is not changed here — it is named in the opt-in's warning text and indocs/credential-pool.md, because an operator who opts a workload in is changing which tier serves it. - The pool tier is visited before the restore tier, which is why a floor-held credential leaves the bundle empty at the pool and why the pool is reachable at all today. The opt-in changes that tier's precondition, not the walk order.
- #2643 (a hard per-key concurrency cap that parks instead of refusing) consumes the same primitive. This record defines the ledger; it does not pre-empt that ticket's parking semantics.
- Observability. A refusal caused by a reservation is indistinguishable today on the series an operator alerts on:
LaunchDeniedTotal(declared inpkg/cloud/metrics) is labelled with the reason tokens that are consts inlaunch_gate.go, and only the DETAIL names the reserve. An additive token for a held-out refusal is part of the last unit; the unreserved wording stays byte-identical, because it is quoted indocs/quotas-and-limits.mdand read by operators who configured no floor. - What is deliberately left: the provider's real remaining capacity (no API exposes it); fork, rewind and the resumes a run's own machinery drives stay outside the gate, as they are today; and the
platformcfgresolver stays process-local — two replicas can judge the same launch under two policies for up to a TTL, which the ledger's conformance suite pins rather than hides.
The units, in order — each with the witness it owes
Nothing here is built. The first unit rebases on the landing of leap/credpool-fingerprint and leap/usagecap-ambient-key, which hold pkg/credpool, pkg/runner/usage_cap.go and the publisher.
Amended 2026-11-04 (ADR-161, debt-leap #112). Units 4 and 6 wait on ADR-161's U2, not on the bench landing; the other units start from main.
pkg/capacityledger— the leaf, both twins, the conformance suite. Witness: a table test on the judgement over the POST-ADMIT total, and the suite run against both twins. Three rows carry it, because the overshoot direction alone leaves the other open —pkg/orgusage's own "concurrent launches never overshoot the cap" assertsallowed > capand nothing else, so it stays green when N-1 are admitted: the boundary from ABOVE (N+1 refused), the boundary from BELOW (a cap of N admits exactly N), and a resume whose own allocation is still outstanding, whichRenewmust not refuse. Three mutations: drop the outstanding term, and N allocations each receive the same remainder; addrequesteda second time on top of the post-increment read, and the from-BELOW row reddens; count the asker's own live allocation in aRenew, and the resume row reddens. CI's-raceandmongo-conformancejobs are where the Mongo twin is judged.- The launch gate allocates, over the two adapters (
pkg/server,pkg/orgusage,pkg/credpool) —gateConcurrencyandgateMonthlyCapsacquire instead of counting;launchAdmission.rollbackreleases beside the run unit and the rate token; the terminal path settles. Witness: two*Servervalues over ONE memory ledger, both admitting into the last reserved slot — it must redden on HEAD (both admitted), and it is built in BOTH completion orders, because a test that can only redden in one order is flaky rather than a witness. Second mutation: drop the release from the rollback, and the slot stays consumed. The existing floor tests run unchanged (pkg/server/launch_gate_floor_test.goon the gate's repo, concurrency and monthly-dollar axes,pkg/server/cloudpublisher/budget_floor_test.goon the credential walk's window reserve). The two adapters are decision 3's whole cost and land here, not later:gateMonthlyCapsreaches the atomic arm THROUGHorgusage.AllowRun. Their witness is the EXISTING suites run unchanged against the adapter —runCounterSuiteover bothorgusagetwins (orgusage_test.go,mongo_test.go) and the broker tests that drivecredpool.NewMemoryLedger, withLimits.Denystill the single copy of the donor rule; the mutation delegates one axis and leaves the old counter beside it, after which the two disagree and the cap is enforced twice.credpool's Mongo ledger has no test of its own today (grep -rln NewMongoLedgernames only the two commands andmongo.go), so the conformance suite is also its first twin-level coverage. - The reconcile sweep (
pkg/server,pkg/capacityledger) — every replica, per-item CAS, bounded batch, no invented spend. Witness: acquire, abandon the holder without settling, one pass returns the slot andOutstandingdrops to zero; the mutation drops the expiry filter. A second test asserts two concurrent passes produce exactly ONE close, and its observer takes a mutex. - The runner honours the lowered policy (
pkg/queue,pkg/server/cloudpublisher,pkg/runner). Witness: a run stamped with a lowered five-hour ceiling stops at the LOWERED percentage; the mutation reverts the stamp read. Plus the unstamped arm, asserted on its SIGNAL and not merely on its fallback: the run is judged on the deployment policy AND the fallback is logged and counted, and the mutation drops that counter — a test that asserted only the fallback would pin the leak as intended behaviour. Plus the EMPTY-stamp row: a stamped but zero policy on a runner whose own source caps leaves the run CAPPED, and the mutation takes the stamp instead of the tighter of the two. The other direction — a stamped message on a runner too old to read it — cannot be exercised from inside the new binary; it is decision 10's reading, owed by unit 6's arms, and this unit only states that the reserve counts as unheld until the floor clears. - The pool as an arbitrated tier (
pkg/budgetfloor,pkg/server/cloudpublisher, the CLI,pkg/server/platform_settings.go,docs/credential-pool.md). Witness: a run holding a healthy credential whose reservation is pool-eligible receives a grant and the preview reports the same tier — the GRANT it receives, never the plan bit. Two mutations: revert the plan arm; revert the tier BODY's guard, after which the walk hands out nothing while the preview still reports the tier active andTestBudgetFloor_ThePreviewAnswersWhatTheWalkDoesreddens. A second test pins the default: no opt-in, no pool. - The meter carries the workload (
pkg/credusage,pkg/runner,pkg/server,pkg/budgetfloor, the studio's typed client). Witness: a repo capped at N% ofreview-pr's reserve is NOT refused byfeature-dev's spend; a credential-month written under two bots reads as ONE row throughList,ListByFingerprintandListByTier, the mutation dropping the bot from the fold; a platformreview-prand a team's fork namedreview-prstay two workloads in the share's consumption, the mutation keying the dimension on the bare name; the conformance suite extended for both twins; and the finding-6 case — an unverified fleet keeps the repository's total, so a runner that writes no bot cannot make the share read unconsumed. The mutation drops the engine-floor gate and the share reads as free — and one row per arm of decision 10: no observer wired in CLOUD mode, anObservedRunnerBuildserror and an EMPTY observation each keep the repository's total and emit the warning, while the LOCAL row (cfg.Mode != "cloud") ARMS the dimension — and every unverified one reads as CLEARED if the gate asksengineFloorfor a build instead of asking the readiness accessor. - Per-pool in-flight ceiling (
pkg/queue/nats,pkg/config,pkg/platformcfg). Witness: on a config whose global is not the compiled default (MaxAckPendingset asITERION_NATS_MAX_ACK_PENDINGwould set it), two pools' consumer configs carry their own ceilings while the shared consumer and an override-less pool both carry that configured global — assertingDefaultMaxAckPendinginstead would pass under either reading and witness nothing.TestPoolConsumerConfigalready pins a non-default global (MaxAckPending: 42): extend that shape rather than write a new one, and add one row whose global is ABOVEDefaultMaxAckPending(512, the valuepkg/config/config_test.goalready sets the env var to), since 42 is below it and cannot exhibit a pool re-capped under a global the operator RAISED. Two mutations: drop the override, and resolve an absent override toDefaultMaxAckPending— the second reddensTestPoolConsumerConfigtoday, which is why the spelling matters. - The chart exposes it (
charts/iterion,docs/agents/ops/runbooks.md) — the configmap key, the per-pool value, oneScaledObjectper pool gated on its ownkeda.enabled. Witness:helm linton dev and prod values, the chart render-diff gate, and a render test asserting the per-poolScaledObjectnames the pool's consumer and that a KEDA-disabled pool renders nothing. Operator gesture: the prod numbers and the infra-apps commit. - A named refusal and the end-to-end witness (
pkg/server,pkg/cloud/metrics, the alert rule,docs/quotas-and-limits.md) — finding 8. Witness: one test crossing publisher → runner for a reservation, allocated at admission and settled at the end; the mutation reverts the new reason token and the series can no longer tell the two causes apart. A deliberately false assertion must fail first: a test never collected proves nothing. - Policy unavailable is not policy absent (
pkg/platformcfg,pkg/server,pkg/server/cloudpublisher) — decision 8: the readiness accessor, and the two arms at BOTH readers. Witness: one row per arm — a WARM cache whose next read fails holds the reserve on the last-known policy; a COLD start that never resolved admits with no allocation, logged and counted, and does NOT read as "no reservations written"; and the same pair oncapPolicyFor, whose nil arm hands every bot the UNLOWERED ceiling. The mutation collapses the arms back to one nil check, after which the cold-start row stops telling them apart. - The reservation's identity is tier-qualified (#2980;
pkg/budgetfloor,pkg/server,pkg/server/cloudpublisher, the CLI,pkg/server/platform_settings.go,docs/quotas-and-limits.md) — numbered last so no other unit's number moves, and landing second, right after unit 1. Decision 11: the tier beside the bot onReservationandRepoQuota.ShareOfBot, a match that compares both, an untiered stored reservation read asplatform,Policy.Validate'sseen/byBotmaps, itsShareOfBotlookup andPolicy.Bots()keyed on the qualified identity, and a gate whose reservation arms judge the resolved identity on every launch surface — the twelvelaunchSubjectliterals that name a bot at70b5b1972go through ONE constructor fed by alaunchBotor a run document, and a static sweep in the shape ofbot_resolver_sweep_test.gorefuses a literal outside it. The credential walk's window reserve takes the same identity from theLaunchSpec'sBotSourceTier, or from the prior run on a resume, threaded throughresolveAndSealCredentialstocapPolicyFor— the walk runs before theRunMessagethat carries the tier is built. The operator doc moves with it in three places: the gate's "six limits, in this exact order" puts the launch rate ahead of concurrency; "The workload is a bot id" becomes a tier-qualified identity, an untiered reservation read asplatform; and "What is not covered" saysPOST /api/runsis judged on its resolved bot, no longer on the request'sbot_id. Witness: a team bot homonymous with a reserved platform bot, launched by its team through the REST launch, is NOT judged as the reserved workload, and the same name launched by a team holding no row of it IS — both directions, or a gate that refuses everything passes. Two mutations: match on the folded name only, which reddens the homonym row; judge the reservation arms on the request's label before the resolution, which reddens whichever row the tierless label misreads — both rows are owed. A policy holding a platform and ateamreservation of one name validates, and each matches only its own launch; the mutation keysValidateon the folded name. A viewer, a client out of launch-rate tokens and a launch over its repository quota are refused with no resolver call; the mutation moves the whole gate behind the resolution. A cloud launch refused on its concurrency arm takes no snapshot and gets its rate token back; the mutation resolves throughresolveBotTieredahead of the reservation arms. The migration's witness is the existing surface tests that write an untiered reservation on the baked catalog'sprobebot and drive it through the resolution (TestHandleLaunchRun_CarriesTheRequestsBotToTheGate,TestProcessBoardCard_CarriesTheCardsBotToTheGate): they run unchanged, and the mutation that reads an untiered reservation as naming no tier reddens them.
Units 4, 5 and 6 sequence after the branches that hold their files; unit 7 and unit 8 sequence against #2029, whose revised plan targets the same NATS and chart surface. Unit 10 lands with unit 2 — the same gate's other arm, split out because its witness is the resolver's, not the ledger's. Unit 11 lands ahead of unit 2 and unit 6: the per-bot count and the meter's workload dimension key on the identity it qualifies, and either one written under a bare name would carry the homonym into the ledger and the meter.
