BYOK API keys (cloud) — per-org, per-user, per-webhook

How iterion-cloud resolves the LLM provider API keys a run uses. The short version: keys are owned by the org, sealed at rest in Mongo, and resolved per-run with a precedence chain (per-webhook override → user default → org default → deployment env fallback). Nothing here is a global plaintext secret the agent can read.

This document exists so we don't reverse-engineer the resolver again. Every claim is anchored to a file:line; if the code moved, fix the anchor in the same change.

Mental model

A run launched by a webhook has the synthetic owner webhook:<id> (no real user), so the user-scoped tiers are empty for it — its chain collapses to per-webhook override → org default → env fallback. That is exactly the "default per org, overridable per webhook" model.

Storage — the api_keys Mongo collection

One document per key, sealed at rest. pkg/secrets/byok.go:

field	meaning
_id	key id (secrets.NewApiKeyID()) — what a webhook override references
tenant_id	owning org; every store call is tenant-filtered (fail-closed)
scope_team	the team the key belongs to
scope_user	set ⇒ user-scoped (personal); empty ⇒ org-wide
provider	anthropic | openai | bedrock | vertex | azure | openrouter | xai | zai (byok.go:50-63)
name	human label
last4 / fingerprint	shown in UI; the key itself is never returned. fingerprint is FingerprintSHA256(plaintext) — the audit identity the run document, the GRANTED log line and the metering bump all key on; indexed (sparse) by EnsureSchema
sealed_secret	the ciphertext (SealAPIKey(sealer, keyID, plaintext)); JSON-hidden (json:"-")
is_default	the default for its (team, user, provider) tuple — ClearDefault keeps it unique
last_used_at	best-effort observability: bumped by id at resolution (MarkUsed, detached off the launch path) and by fingerprint at the START and END of every runner attempt (MarkFingerprintUsed). The start stamp lands once the run is ADMITTED, after the usage-cap pre-flight: a run parked on a ceiling never held the key and never dates it. Nothing moves it during a turn, so a long attempt shows its start until it ends. Scope follows the key's tier: a tenant's own key is bumped under the run's tenant only, so a different tenant that stored the byte-identical secret never reads its own key as "in use" (the studio shows this field as exactly that, before a rotate or delete); a platform-tier or pool-lent key (RunBundle.PlatformSourced / PoolSourced) is bumped across tenants, because its row lives under the platform sentinel or in the donor's tenant and it serves every tenant
alive_runs (view only)	how many runs count against this key's ceiling right now — the same query the launch walk asks (what counts). Present whatever max_concurrent_runs is, so "is this key in use?" has an answer; absent (not zero) when there is nothing to count with (no fingerprint on a legacy row, no run store, a logged store error). The run side of the same audit is cred_fingerprints / llm_idle_since on GET /api/runs/{id}
refused_until / refused_reason (view only)	set when the PROVIDER is currently turning this credential away — a dead token, a fair-usage refusal, a spent org ceiling, an exhausted window — folded from the shared usage ledger by usagecap.RefusedUntil, the same reading the launch walk acts on (usage-caps.md). Absent when nothing is refusing the key, and on a fingerprint-less row (which names a slot, not a credential). This is the only place a pinned refused key is visible: a webhook key_overrides pin bypasses the skip by design, so the walk leaves no skip log
max_concurrent_runs	optional ceiling on how many alive runs may hold this key at once (0 = uncapped) — the operator-side answer to providers whose fair-usage limits publish no numeric bound. What counts is defined in Concurrency ceiling — what counts
expires_at	optional

• Interface: ApiKeyStore (Create/Get/GetOwned/Update/Delete/ListByTeam/ListByUser/MarkUsed/MarkFingerprintUsed/ClearDefault) — pkg/secrets/byok.go. GetOwned is the credential pool's cross-tenant read, bounded by ownership; MarkFingerprintUsed the runner's metering bump.
• Ingestion gate: the create and rotate routes refuse (400) a value whose shape could not authenticate — secrets.ValidateAPIKeyShape: a bearer token with any white-space, control or invisible character for the bearer providers; anything but a JSON object for bedrock / vertex, whose credential is a document. See the ingestion-gate section of cloud-llm-credentials.md.
• Backings: MongoApiKeyStore (prod) + MemoryApiKeyStore (tests).
• Wired in the server at cmd/iterion/server.go:193 (NewMongoApiKeyStore(st.DB()) + EnsureSchema), handed to both the HTTP server (ApiKeys: config) and the cloud publisher.

The plaintext is sealed with the server's Sealer before it touches Mongo, and is only unsealed transiently inside resolveAndSealCredentials to be re-sealed into the per-run bundle. It is never written to logs, events, artifacts, or returned by the API.

Concurrency ceiling — what counts

max_concurrent_runs is enforced at resolution (cloudpublisher.apiKeyUsable): a key at its ceiling is passed over like a refused one and the walk hands the next key of that provider over (or the next tier). The count is CountAliveRunsWithCredFingerprint on the run store — both twins, one conformance row — and a run counts against a key only when all three hold:

1. It is alive — queued or running (RunStatus.HoldsCredentialSlot). A parked or paused run spends nothing while it waits, and its resume re-resolves against the ceiling like any claim.
2. Its stamp names the key. The stamp (run.cred_fingerprints, written at launch and re-written at every resume) is narrowed to the credentials the run's resolved routes can spend — the same walk and vocabulary the credential pool's wants derivation uses (model.EffectiveProviders). The sealed bundle keeps every credential; the stamp does not: a run whose every node pins provider: anthropic executes on the forfait and holds no slot on the z.ai facade key it also carries. A run with an unpinned or unresolvable route keeps every fingerprint (it takes whatever the process holds — fail open toward protection).
3. It is not model-idle. The runner sets run.llm_idle_since the moment the run's last model-calling node finishes and clears it the moment one starts (concurrent branches are counted, not flipped), so a sixty-minute tool-only verify gate between two agent passes holds a slot for nobody. A run counts from claim until it proves idle, and every re-stamp starts it over. Explicit toggles, not a lease: a pod that dies mid-node leaves a run the orphan sweeper parks, and rule 1 releases it.

Everything uncertain counts (over-protection), never the reverse: a count error leaves the ceiling unapplied for that resolution and is logged.

Resolution — secrets.Resolve

pkg/secrets/byok.go:168:

Resolve(ctx, store, teamID, userID string,
        providers []Provider,
        keyOverrides map[Provider]string,   // provider → key_id
        sealer) (map[Provider]Resolution, error)

Two passes over the keys visible from (teamID, userID):

1. Pass 1 — explicit overrides. For each provider → key_id in keyOverrides, pin that exact key (must be visible + the right provider). This is the per-webhook override hook.

2. Pass 2 — priority walk. For any provider not already pinned, take the first key in keyRank order (byok.go:234):

   rank	key
   0	requesting user's default (scope_user==me && is_default)
   1	requesting user's other key
   2	org default (scope_user=="" && is_default)
   3	org other key
   99	another user's personal key — never applies

The publisher calls it for allKnownProviders (publisher.go:138) and seals whatever resolved into the run bundle.

Where the publisher uses it

pkg/server/cloudpublisher/publisher.go:167resolveAndSealCredentials, step 1 ("BYOK API keys", L189):

resolved, _ := secrets.Resolve(ctx, p.apiKeys, tenantID, ownerID,
                               allKnownProviders, nil /* keyOverrides */, p.sealer)
for prov, r := range resolved { bundle.APIKeys[prov] = string(r.Plaintext) }

The bundle is sealed under a fresh secrets_ref; the runner unseals it and stamps bundle.APIKeys into ctx (pkg/secrets/credentials.go). The in-process claw backend and the claude_code, pi, and Codex delegates read their applicable keys from that credential context. Kimi and Grok instead rely on their CLI's own environment/config and do not consume the sealed BYOK map.

Env fallback. When the bundle has no key for a provider, the resolved bundle is empty for it and the runner falls back to the pod env (ANTHROPIC_API_KEY, OPENAI_API_KEY, …). That is the only role of the deployment-level iterion-llm Secret — a fallback for orgs that haven't entered their own keys, not the primary path.

REST API

pkg/server/byok_routes.go. All under requireAuth; key values are write-only (never returned).

verb + path	role
GET /api/teams/{id}/api-keys	list org + my keys visible from the team
POST /api/teams/{id}/api-keys	create an org-wide key
GET/POST /api/me/api-keys	list / create a personal key
PATCH /api/teams/{id}/api-keys/{key_id}	rename / promote to default
DELETE /api/teams/{id}/api-keys/{key_id}	revoke

Create body (byok_routes.go:132): { "provider": "anthropic", "name": "...", "secret": "<key>", "is_default": true }. The server seals secret and stores only the ciphertext + last4.

Studio UI: Settings → API Keys (studio/src/views/SettingsDialog/ApiKeysTab.tsx, studio/src/api/byok.ts). Cloud accounts use the sibling account API-key page.

Per-webhook key override

Goal: a webhook can pin a specific key per provider, overriding the org default — and you can have several webhooks for the same bot, each on a different key (e.g. billing/quota separation per integration).

Built — engine + wiring. Resolve's keyOverrides (Pass 1) is the mechanism; the wiring threads a webhook's pinned keys through to it:

1. webhooks.Config.KeyOverrides map[string]string (provider name → key_id) — pkg/webhooks/types.go.
2. Threaded: webhook handler → runview.LaunchSpec.KeyOverrides → persisted on store.Run.KeyOverrides (so cloud resume re-resolves the same keys) → resolveAndSealCredentials(…, keyOverrides) → secrets.Resolve(…, overrides, …).
3. Set via the webhook create/PATCH API — key_overrides on webhookConfigReq. validateKeyOverrides rejects a key_id from another tenant or the wrong provider at config time (the resolver is already tenant-scoped, so this is a fail-fast UX guard, not the security boundary).

A pin bypasses the refusal skip, and says so. The evidence predicate is consulted in Pass 2 only: an operator who names a key gets that key, even when the provider is freshly refusing it — honouring the pin over the optimisation is what keeps the predicate an optimisation. Because the walk then logs no skip, a pinned dead key used to be visible only as the absence of a line. It now emits one warn per launch naming the key, its fingerprint, why the provider is refusing it and when that lapses, and the key's own view carries refused_until / refused_reason. The pin still wins; it is no longer silent.

Example: PATCH /api/teams/{id}/webhooks/{wid} with {"key_overrides": {"anthropic": "<key_id>", "openai": "<key_id>"}}. The studio webhook-editor field for it is the remaining follow-up (the API is functional). Covered by TestResolve_OverrideWins (pkg/secrets/byok_test.go) + TestGitLabWebhook_HappyPath threading assertion (pkg/server/webhooks_gitlab_test.go).

Multiple webhooks per bot — already supported

The webhook spine keys configs by _id, not by bot; nothing stops N webhook_configs in one org all targeting the same bot_ids. Combined with per-webhook overrides, that yields "same bot, different key per webhook." No work needed beyond the override field above.

Per-webhook secret override (e.g. a distinct forge token)

The same per-webhook idea applies to the bot's stored secrets (the secrets: block), not just LLM keys. A bot like review-pr declares forge_token and the org binds it to one stored secret (bot-secret bindings; ResolveGenericWithBindings precedence user → binding → team). A webhook can override that binding per workflow-secret name via webhooks.Config.SecretOverrides (name → secret_id), threaded exactly like KeyOverrides (handler → LaunchSpec → store.Run → ResolveGenericWithBindings Tier 0, which wins over the binding). Set it on the webhook create/PATCH API as secret_overrides; validateSecretOverrides rejects a secret_id that isn't an org-scoped secret of the webhook's tenant. Use it to post under a different GitLab token / bot identity per webhook (webhook A → bot-1's token, webhook B → bot-2's). The override carries no binding-level allowed_hosts, so egress falls back to the workflow's own secrets.<name>.hosts declaration.

Plugging many repos into auto-review with one token

Two knobs make "one token, every repo" work at the org level (no per-repo setup), with no instance-wide secret:

1. Token scope. Use a GitLab group access token (covers every project in the group) as the org's forge_token, instead of a single-project token. One token authenticates posting on all repos.
2. Webhook scope. Register the GitLab webhook at the group level — GitLab fires it for every project in the group — pointing at one iterion webhook_config with a broad/empty project_allowlist.

So 1 group token (org binding) + 1 iterion webhook + 1 GitLab group webhook = the whole group auto-reviewed. An instance-wide default forge token (shared across orgs) is deliberately not a concept — secrets are per-tenant for isolation; the org + group-token model gives "one token, all repos" without crossing the tenant boundary.

Deployment guidance (cloud)

• Proper model: each org enters its keys via POST /api/teams/{id}/api-keys (sealed into Mongo, per-tenant). The publisher then resolves that org's keys per run.
• Bootstrap shortcut (what ovh-dev used first): the iterion-llm sealed Secret holds ANTHROPIC_API_KEY + OPENAI_API_KEY as pod env — a single instance-wide fallback. Fine to start, but it is not multi-tenant; once orgs bring their own keys it should shrink to (or be removed in favour of) the per-org store. Reseal/rotate playbook for the fallback lives in the k8s-deploy notes.

Security invariants

1. A key's plaintext is sealed before it reaches Mongo and is only unsealed transiently to seal into a per-run bundle; it never lands in logs/events/artifacts/API responses (json:"-" on sealed_secret).
2. Resolution is tenant-scoped and fail-closed (teamID required; another user's personal key ranks 99 = never applies).
3. The agent never selects a key — selection is a server/publisher authz decision, mirroring the file-secret rule in secrets.md.
4. A webhook override may only reference a key the webhook's tenant owns.

See also: backends.md (backend/provider selection), secrets.md (file/env/generic secrets), and the cloud control-plane epic for the webhook spine.