Architecture

buildkit-operator is a control plane over stock buildkitd. It owns two things — routing (send builds that should share a cache to the same daemon) and lifecycle (keep that daemon warm, scale it to zero, snapshot it, clone it). It deliberately owns nothing at the storage layer: the daemon's content store, snapshots, and bbolt metadata are vanilla.

flowchart LR
    ci["CI runner<br/>GitHub Action / build CLI"]
    subgraph op["ns: buildkit-operator (control plane)"]
        buildd["buildd (Deployment ×2, HA)<br/>reconciler (leader) + /route /prewarm (all)"]
        gw["gateway<br/>shared SNI router (1 LB)"]
    end
    subgraph builds["ns: buildkit-builds (daemons)"]
        sts["StatefulSet-of-1 per (project, arch)<br/>buildkitd (vanilla) + companion<br/>+ Cinder gen2 PVC (warm cache)<br/>Service :1234 — always ClusterIP"]
    end
    buildd -- "reconciles / scales / snapshots" --> sts
    gw --> sts
    ci -- "POST /route (endpoint)" --> buildd
    ci -- "buildx remote (TCP + mTLS)" --> gw

Namespaces. The control plane (buildd + gateway) runs in buildkit-operator; the per-project daemons, their certs/config and the BuildProject CRs live in buildkit-builds; the Kata node plumbing (when sandboxing untrusted forks) is in a third namespace, buildkit-system. The split is by trust/role so each namespace carries only the admission exemption it needs — see ADR 0006. buildd creates daemons in the builds namespace but takes its leader-election Lease in the operator namespace (via the POD_NAMESPACE downward API).

The routing key — the one invariant that matters

All builds that must share a cache must resolve to the same key ⇒ the same StatefulSet ⇒ the same daemon. This is the heart of the design and lives in internal/router as a pure function, shared verbatim by the CLI and the control plane so they can never disagree.

Function	Formula	Purpose
`ProjectKey(repo, name, target, arch)`	`"p" + sha256(normRepo [\x00 n:name] \x00 normTarget \x00 normArch)[:16]`	the canonical cache identity
`ForkKey(canonicalKey)`	`"fork" + key`	an ephemeral, isolated daemon for untrusted/fork PRs
`CloneKey(canonicalKey, i)`	`"c" + i + key`	the i-th CoW clone for fan-out
`CachePVCName(key)`	`"cache-buildkitd-" + key + "-0"`	the StatefulSet's `volumeClaimTemplate` PVC
`EndpointHost(host, port)`	`tcp://host:port`	the deterministic `<daemon>.<gateway-host>` endpoint for off-cluster CI via the shared SNI gateway

Design points:

The key is coarse on purpose — no branch, no commit, no build context. Two concurrent builds and a build an hour later all converge to the same daemon, so they share layers and cache mounts. A finer key (e.g. per-branch) would fragment the cache and defeat the whole point.
target is part of the key because different Dockerfile target stages have genuinely different caches; folding them together would thrash.
arch is part of the key because a daemon is single-arch (it builds natively; cross-arch is a separate daemon, not QEMU-in-one).
The optional name segments a monorepo into per-component daemons — one daemon + cache per image, so unrelated components in the same repo don't share (or thrash) a cache. It is omitted from the hash when empty, so a single-image repo keeps the exact key it had before name existed (migration-safe). Wired end-to-end through RouteRequest.Name, BuildProjectSpec.Name, DeriveChild, and the CLI --name flag (env BUILDKIT_OPERATOR_NAME) / build.sh NAME.

Example: SocialGouv/buildkit-operator-example + amd64 (empty name) → pa081c22c974da132 (the daemon name you see running on the cluster).

The reconcile loop

The BuildProject reconciler (internal/controller) is a standard controller-runtime loop. Per object it converges:

StatefulSet-of-1 + Service + PVC — rendered by internal/builder with the rootless security profile and the gen2 volumeClaimTemplate. The Service is the stable mTLS endpoint :1234.
desiredReplicas — tier- and idle-aware scale-to-zero. When a warm project goes idle past idleTimeoutSec (and no build is in flight), it scales the StatefulSet to 0 but keeps the PVC — so waking up is an attach, not a restore. hot never scales to zero.
maybeSnapshot — periodic in-use VolumeSnapshot on the snapshotEverySec cadence, using OVH's in-use snapclass so the daemon does not need to scale to zero to be snapshotted. Old snapshots are pruned to --keep-snapshots.
reconcileFanout — when spec.fanout > 0, materializes N CoW clone daemons (CloneKey) from the latest snapshot — vertical-first scaling for a saturated project. The clone spec comes from the shared derivation policy bkov1.DeriveChild(parent, parentSnapshot, CloneChild, key) — the same function the /route fork path uses (with ForkChild), so a fan-out clone and a fork daemon can never drift in how they inherit storage/security and seed from the parent snapshot.
Status — phase (Pending/Warm/Idle/Scaling/Failed), replicas, endpoint, lastSnapshot, and inflight. Status is only written when it actually changes — unconditional status writes would re-trigger reconcile in a loop.

Nothing takes a daemon away from a running build

status.inflight holds one timestamped entry per routed build: /route adds one, /complete removes it, and an entry whose release never arrives expires on its own clock past --max-build-seconds. That set is what every destructive decision consults, each re-reading it from the API server rather than the informer cache, because /route runs on every replica while the reconciler runs on the leader:

scale-to-zero and fork reaping hold off while a build is in flight;
a pod-template roll (any chart upgrade that changes the rendered daemon — a companion tag is enough) is withheld until the daemon drains. Every build gets at least an hour before a forced roll can cut it; --max-build-seconds caps the wait so a project that builds back-to-back still takes new images. A wedged daemon is rolled regardless — the roll is usually the repair;
/route stops advertising a daemon mid-roll, so a build is never handed the endpoint of a pod Kubernetes is about to delete;
a PodDisruptionBudget exists exactly while the daemon serves builds, so a node drain waits for them — the one disruption the control plane cannot otherwise veto — and is deleted once idle, since a zero-minimum budget would block drains on an unhealthy pod instead of permitting them;
lowering fanout leaves a clone that is still serving builds for a later reconcile.

Metrics emitted: buildkit_operator_routes_total, buildkit_operator_route_duration_seconds, buildkit_operator_coldstarts_inflight, buildkit_operator_scale_events_total, buildkit_operator_snapshots_total, buildkit_operator_daemon_rolls_held.

Control-plane HA

buildd runs replicas: 2 with leader election (--leader-elect, a coordination.k8s.io Lease). Two roles, split deliberately:

The reconciler runs on the leader only — exactly one writer of cluster state, no double reconcile.
The /route HTTP API runs on every replica — the route server sets NeedLeaderElection() = false, so a routing request is served whether it lands on the leader or a follower. Routing is read-mostly (ensure-or-wait); only the leader mutates.

buildkit-operator-buildd reports 2/2 ready and the buildkit-operator-buildd.buildkit-operator.socialgouv.github.io Lease is held by one of the two pods. Kill the leader and the follower takes the Lease; /route never stops serving.

The shared SNI gateway (off-cluster CI)

Daemon Services are always ClusterIP (in-cluster clients only). An external CI runner reaches every daemon through one shared SNI gateway — a new cmd/gateway binary (image ghcr.io/socialgouv/buildkit-operator-gateway) fronted by a single LoadBalancer, instead of a public LB per daemon (which doesn't scale with project count).

How it routes, without terminating TLS:

buildd is started with --gateway-host <domain>, which makes /route return a deterministic endpoint tcp://<daemon>.<gateway-host>:1234 — computed straight from the key, no polling (no waiting on an LB ingress IP).
The runner dials that hostname. Its TLS ClientHello carries SNI <daemon>.<gateway-host>.
The gateway peeks the ClientHello's SNI (it terminates no TLS), maps <daemon> to that project's daemon ClusterIP Service <daemon>.<ns>.svc:1234, and pipes the still-encrypted bytes through. mTLS stays end-to-end to the daemon — client-cert auth is intact; the gateway never sees plaintext and rejects any SNI outside its domain or not a buildkitd- daemon name.

Requirements: a wildcard DNS record *.<gateway-host> → the gateway LB, and the daemon certificate's SAN must cover *.<gateway-host> (create-certs.sh honours GATEWAY_HOST=…). The Helm chart renders the gateway Deployment + its one LoadBalancer Service from gateway.{host,image,service.type,resources}, gated on gateway.host. See ci-integration.md and security.md. This is the same end-to-end mTLS shape the existing buildkit-service uses — buildkit-operator just fans one LB out to many daemons by SNI instead of allocating one LB each.

Backends — Kubernetes or a single host

The reconcile loop above is the Kubernetes provisioner. The control plane is substrate-agnostic: the routing key, the build CLI, the CI Actions and OIDC are all backend-neutral, so the only Kubernetes-bound layer is the provisioner — the code that turns a key into a running, addressable daemon. It sits behind a small Provisioner interface, and a single-host implementation (Incus + ZFS) slots in with --backend local: one buildkitd per project on a retained ZFS dataset, with an in-process scale-to-zero + snapshot loop instead of a controller (no CRD, no etcd). See Single-host backend and ADR 0007.

What stays vanilla (non-goals)

No fork of BuildKit or containerd; no custom snapshotter; no merging bbolt stores between daemons.
No concurrently-writable cache between daemons — that does not exist in BuildKit. Across daemons we share layers (via S3, see storage-and-cold-cache.md); RUN --mount=type=cache mounts stay per-daemon by design.