Merge policy — main is protected by a merge queue

main is guarded by a GitHub merge queue (ruleset "main protected — merge queue", enforcement active). This closes the semantic inter-PR conflict class: two PRs that are each green against main-at-branch-time can still break main when combined (observed 2026-07-12: #120×#121 didn't compile combined; #128×#130 conflicted). The queue rebuilds each PR against the queue head — main + every PR already ahead of it in the queue — and merges only if that combined tree is green.

How it works

• A PR is merged through the queue, not directly. Click "Merge when ready" (or gh pr merge <n> --auto --squash) to enqueue it.

• The queue creates a temporary branch = main + earlier-queued PRs + this PR, runs the required checks on it, and squash-merges only if green. Grouping is ALLGREEN (a failing entry drops out; the rest still merge).

• A queue entry is a full CI cycle, and the organisation shares 20 concurrent jobs across every repository — so how many entries build at once is what decides how long a merge takes, far more than how long any test runs. Measured 2026-09-08: nine jobs totalling ~31 min of compute took 44 min of wall clock, 19 of them waiting for a first slot, while five entries built in parallel — so the queue was set to max_entries_to_build: 2. Since 2026-09-29 every required check from tests.yml runs on the organisation's arc-runners scale set (below), which the cap does not bound, and the queue builds three entries at once (max_entries_to_build: 3); four once a week at three shows the scale set keeps up — runner pods not left pending, the queue's latency in task ci:queue-stats. min_entries_to_merge: 3 (merges land in batches of 3-5, so the workflows that fire on every push to main — Tests, Container Image, Trivy, and through Container Image's completion Runner Image and Sandbox Base Images — run once per batch). A lone PR still merges after min_entries_to_merge_wait_minutes (5).

• How long the queue waits for a check — check_response_timeout_minutes (60), read from the ruleset on 2026-10-29 with gh api repos/SocialGouv/iterion/rulesets/18857412 (that GET is also the rollback of any change to it, since the PUT below wants the complete representation). An entry whose required check never reports is dequeued checks_timed_out after that long, and so is every entry behind it. Measured 2026-10-15 (SocialGouv/iterion#2377): a required job sat 70 minutes queued on arc-runners with no runner while its siblings ran, and the queue ejected the pull request. The pool had capacity — the Actions service was holding the job on a phantom runner, the ghost class .github/workflows/arc-canary.yml exists to detect.

  The alert and the dequeue currently race, by arithmetic. The canary pages on a probe nothing picked up within STALL_SECONDS (1500 s, 25 min, scripts/ci/arc-canary-verdict.sh), it probes every 30 minutes, and its watchdog gives the period it judges a 5-minute head start — so dispatch can stop being trustworthy up to 30 + 25 + 5 = 60 minutes before anyone is paged, exactly the queue's own patience. The page is supposed to land while a human can still set CI_SELF_HOSTED to off (below), not after the queue has given up. Closing that gap is one field of the ruleset — check_response_timeout_minutes to 90 — and an admin's PUT, which is what is left on #2377; internal/ciguard holds the three numbers above to the ceiling this file claims, and holds every site that writes the timeout to one value.

• Required checks (the fast, reliable ones): test, race, vendor-check, mongo-conformance, golangci, brand, revi/review — and nats-conformance once an admin adds it to ruleset 18857412. test and race are AGGREGATOR CONTEXTS, not test-running jobs: the work lives in the gated legs (test-unit, test-e2e, studio, the race-shard matrix) that run in the queue behind them, and each aggregator re-verifies every leg's conclusion under if: ${{ !cancelled() && <tap skip> }} — the run-state clause combined with the one verified skip (below), so an aggregator skips exactly the entries its legs skip and runs on every other one. The required NAMES stay stable — a PR that renamed one could never pass its own queue build — while work scales sideways (shards, new legs). The legs themselves carry the tap skip and nothing else, a contract internal/ciguard's gatedJobs enforces: a leg any other condition skips reports Success on the pull request while reddening its queue entry forever, and a leg without the skip spends the heavy minutes the skip exists to save. fmt-check is STAGED: it reports on the pull request AND in the queue, so the context is already there the day the ruleset names it, and internal/ciguard's requiredChecks carries it. Editing that ruleset from the API needs PUT /repos/{owner}/{repo}/rulesets/{id} with the complete representation (name, target, enforcement, bypass_actors, conditions, rules); a PATCH, or a PUT missing any of those, answers 404 — which reads exactly like a permission ceiling and is not one. Read the ruleset first and send it back with the one field changed. The nats-conformance job runs the JetStream schema-rollout integration tests (#481); until it is required, a regression there merges green. The slow container-image build is intentionally NOT required — it builds on merge to main and would stall the queue 12 min/PR. A required check carries no if:, with one verified exception: the Homebrew tap update (below) — which the two aggregators carry combined with their !cancelled() clause.

  Promoting a check to required is a two-file change. The eight advisory jobs — nats-conformance, cloud-e2e, helm-lint, govulncheck, desktop-vet-linux, desktop-vet-cross, docs-links, docs-build — carry if: github.event_name != 'merge_group' in .github/workflows/tests.yml: a job that cannot block a merge should not hold a runner slot the queue needs. (fmt-check is not listed above and carries no merge-queue skip: it is STAGED to become required — named in internal/ciguard's requiredChecks, not yet in the ruleset — and stays in the queue so its context exists the day it names it.) Adding one to this ruleset without deleting its skip is worse than a stalled queue: a job skipped by a job-level if: reports Success, so the required check is satisfied by a job that never ran — every entry merges green on a check that did not execute. (A workflow-level filter is the opposite: the check never reports and the queue hangs until check_response_timeout_minutes. Same word, opposite failure, and the silent one is the one this file's skips produce.) Nothing in the repository can catch it — the required list lives in the ruleset — so the two edits go together, by hand.

  An advisory check that nobody reads is not a check. docs is deliberately not required — blocking a hotfix on a documentation build would be the wrong trade — but the consequence is that a broken publish rides main in silence: the queue never looks at it, so every following merge inherits a red main without having caused it. Measured 2026-09-10: one wrong file extension in a doc link (monaco.ts for monaco.tsx) kept the site from building across six consecutive commits and ~100 minutes, found only by reading the run list by hand. .github/workflows/docs.yml now reports on itself — a failed publish on main opens ONE issue labelled ci:docs-build (later failures comment on it rather than pile up), and the next successful build closes it. The alert is a state, not a stream. Any other advisory job whose failure nobody would notice wants the same treatment rather than promotion to required.

• Every required check from tests.yml runs on self-hosted runners — test, race, vendor-check, mongo-conformance, golangci and brand route to the organisation's arc-runners scale set on merge_group, because the 20-job cap above is what makes a cycle slow. That scale set is outside this repository, and it was dead and unnoticed for over a year before 2026-09-08.

  If nobody can merge and the checks never report, this is the first thing to try. Set the repository variable CI_SELF_HOSTED to off (Settings → Secrets and variables → Actions → Variables) and every job goes back to GitHub-hosted runners on the next run. A variable rather than an edit to tests.yml, deliberately: repairing by merging does not work when merging is what is broken. Unset means on. Alerting on the scale set itself is still missing — SocialGouv/iterion#983. Since the mega-job split, one queue entry holds ~12 concurrent GitHub-hosted jobs on this lane (the gated legs plus the race-shard matrix): the escape hatch still works, but it now spends 12 of the org's 20 shared slots per entry — one fork PR from another repository can starve it, so expect slowness, not silence.

• No required human approval (required_approving_review_count: 0) — the bot factory's own adversarial review + the checks are the gate; a reviewer still merges deliberately.

The one verified skip: the Homebrew tap update

brew-update.yml opens one pull request a night, chore(brew): update tap to vX, that moves the version and sha256 values of Formula/iterion.rb and Cask/iterion-desktop.rb and nothing else. No heavy check's verdict depends on those values (internal/ciguard's tap tests rewrite them), and running the checks anyway cost ~57 minutes of compute per pass — on the pull request, then again in the queue, where the entry held one of the three build slots for ~25 minutes (test 24.1, race 22.3, mongo-conformance 4.0, golangci 3.2, fmt-check 1.8, vendor-check 1.0 min, measured on #2314).

So the gated legs (test-unit, test-e2e, studio, race-shard), the aggregators test and race that speak for them (combined with their !cancelled() clause), mongo-conformance, golangci, vendor-check and fmt-check carry one condition, the only if: a required check may carry here: they skip a pull request whose title, or a queue entry whose commit message, starts with chore(brew): update tap to (the queue squashes a single-commit pull request under its commit subject — the repository's COMMIT_OR_PR_TITLE — so brew-tap-pr.sh puts the prefix in both). A title is only a claim and a skipped job reports Success, so the claim is verified where nothing can skip the verification:

• brand — required, never skipped, and already a reader of both tap files — runs scripts/ci/tap-entry-check.sh with the same predicate, evaluated (never re-matched: GitHub's startsWith ignores case). It fails unless the diff touches the two tap files alone, matched exactly and with rename detection off, and moves only their version and sha256 values, each from one value update-brew-tap.sh writes to another (a dotted numeric version, a 64-hex digest) — a line added, a comment, a desc, a Ruby interpolation or an empty digest is refused: the pull request's test merge against its base, the queue entry against the entry it was built on.
• In the queue it also reads the live grouping strategy (GET /repos/{owner}/{repo}/rules/branches/main) and fails unless it is ALLGREEN. Entries stack — each is built on the one ahead of it — so a tap entry's skipped checks vouch for nothing the entries ahead brought. ALLGREEN merges an entry only on its own checks; under HEADGREEN a green tail would merge everything ahead of it. Switching the queue to HEADGREEN ejects every tap entry until the skip leaves tests.yml — by design.
• internal/ciguard holds the pieces together: the expression byte for byte on each job, brand without it and without needs: (a job whose dependency is skipped is skipped too, and passes), the verifier step unconditional (no if:, continue-on-error, shell: or defaults:) and fed the same predicate, the tap pull request's title and commit subject starting with it.

The verdict that counts is the queue's. On the pull request the verifier judges the test merge against the base it was built on, and retargeting a pull request (an edited event) re-runs nothing; the queue judges every entry again against the entry ahead of it, and a direct merge bypasses every check anyway. The workflow file is the pull request's own, so a pull request that also edits tests.yml can drop the verifier with the rest — the class every check here shares, which review and revi/review answer, not this check.

A code pull request titled like a tap update does not merge: its heavy checks skip, brand refuses it, and the entry leaves the queue. A push to main evaluates the predicate false, so main still runs everything.

The tap pull request also opens at night. brew-update.yml runs when version.yml's nightly run completes (00:03–02:21 UTC on its 27 runs since 2026-09-11), not on a cron of its own: GitHub fired its 04:37 UTC cron between 10:46 and 12:21 UTC, inside the queue's working hours. Over 2026-09-12..10-12 each hour from 00:00 to 05:59 UTC saw 2-7 merges in all, against 19-37 for each hour from 06:00 to 21:59. It taps the newest tag whose CLI artefacts are complete, so the nightly's own tag, still building, waits for the next night; a cask left behind is caught up once its release holds the desktop ZIP; and nothing walks the tap back — update-brew-tap.sh never writes a version below the one a file carries, and brew-tap-pr.sh opens a pull request only for what main does not carry yet, never from a branch whose files sit below main's (tap branches outlive their merged pull request here). A desktop-only release (its CLI build red) reaches the cask only through a dispatch of brew-update.yml with its tag, as long as the formula has not moved past that tag.

Direct pushes (admin bypass)

Repository admins bypass the queue (bypass_mode: always) — a hotfix can still be pushed straight to main (e.g. un-break a red main fast). Use it sparingly; the queue is the default path. Non-admins must go through a PR + the queue. For reviewed chantier work, the operator direct-landing lane generalises this bypass — and owes it what a hotfix does not: the train, the locally reproduced gate and the greppable trailers.

The fast lane for an incident fix, and its exact procedure

The queue is a single writer with one lane: a green incident fix waits behind every comfort PR enqueued before it. Measured on 2026-09-06, a fix that closed a production incident class waited 4 h 42 at ~1 PR/h, with the head group rebuilt every ~15 minutes (each rebuild restarts the ~35-minute test workflow for every member), while estimatedTimeToMerge reported ~2 h throughout.

There is no automatic fast lane, by decision: a label-driven direct merge would bypass the very rebuild that closes the semantic inter-PR conflict class the queue exists for. The escape hatch is a human admin, deliberately, and this is how it is done — the shape matters, because two of the three obvious invocations do the wrong thing:

# 1. `gh pr merge <n> --squash` WITHOUT --admin does NOT merge: it enqueues.
# 2. `--admin` on a PR already in the queue answers "already queued".
# So: dequeue first, then merge with --admin.
gh api graphql -f query='mutation($id:ID!){ dequeuePullRequest(input:{id:$id}){ clientMutationId } }' \
  -f id="$(gh pr view <n> --repo <owner>/<repo> --json id --jq .id)"
gh pr merge <n> --repo <owner>/<repo> --squash --admin

Before using it, the same proofs the queue would have demanded must already be green on the PR head: every required check (test, race, vendor-check, mongo-conformance, golangci, brand) and revi/review. The bypass skips the rebuild against the queue's other members, nothing else — so it is legitimate when the PR is small, or touches files no queued PR touches, and reckless when it is a wide refactor.

After an admin merge, verify what actually landed: the queue has merged a stale head before (see the note below), and a bypass has no group build to catch it.

git fetch origin main
git diff --stat <pr-head-sha> origin/main -- $(git diff --name-only "$(git merge-base <pr-head-sha> origin/main)" <pr-head-sha>)

An empty diff means the merge carries the head you reviewed.

Two consequences worth stating plainly:

• The autonomous pilot cannot use this. An agent has no admin rights and must not be given them; an incident fix launched by a bot waits in the queue like everything else, or a human takes it through the hatch above.
• The durable fix is queue throughput, not exceptions. Every direct push to main invalidates every merge group in flight: each one rebuilds from scratch on the new base. That is why the brew-tap update lands through the queue as its own pull request, and why the release waits for the queue to drain (below); an exception buys the rebuild back for one PR only.

Releases and the queue

A release is a direct push to main — version.yml runs release-it, which commits chore: release vX.Y.Z and its tag, the token-bureau App bypassing the ruleset — so it invalidates every merge group in flight like any other direct push. Releasing after every merge discards the build of whatever is queued behind the merged PR while it runs. Measured with task ci:queue-stats over 2026-09-22..28: 58 releases for 102 merges, and 20 queue builds discarded by a direct push while they ran — 15 by a release commit, 5 by a pull request merged directly (31 of the 102 skipped the queue through the admin bypass, and each of those is a direct push too). Over 2026-09-01..28: 387 of 1096 queue builds, 242 of them by a release.

So version.yml holds a merged pull request's release while a merge group is building — any gh-readonly-queue/main/* branch exists — and the merge that drains the queue releases everything merged since the last tag. The nightly (22:27 UTC) and a workflow_dispatch release regardless, without reading the queue at all: the nightly is the backstop for a queue that drains without a merge (its last entry ejected) or never drains, and a dispatch is an operator's explicit request. A release delayed by a busy queue therefore waits at most until the next drain or the next nightly; to ship now, dispatch it.

GitHub sometimes leaves a gh-readonly-queue/main/* branch behind after its group merged (seen on mastodon, cilium, zed, flutter). Counted as a live group, one such orphan would hold every merged PR's release until the nightly. A branch is named after the pull request at the tail of its group (pr-<n>-<base>), and a group only exists while that pull request waits in the queue, so the hold counts a branch only while its pull request is open — and, as a backstop for one dequeued but kept open, while its head commit (dated when the entry was queued; rebuilds keep the date) is less than a day old; from their last enqueue to their merge, the 221 queue merges of 2026-09-15..29 waited 16 minutes at the median and 80 at most. The job's log names every branch it ignored with its pull request's state: check it, then delete it — gh api -X DELETE repos/SocialGouv/iterion/git/refs/heads/gh-readonly-queue/main/pr-<n>-<sha>.

The hold is best effort, not a lock. The branch list is read just before the run moves to main's tip — in that order, so a group that merges between the two holds the release rather than leaving release-it on a stale tree — and release-it still builds for a few minutes before it pushes; a group that starts between the read and the push rebuilds once. What is left is measured, not assumed: task ci:queue-stats counts the queue builds discarded by a release (Measuring the queue) — which also counts the nightly and dispatched releases that invalidate on purpose; the script cannot tell them apart from a merged PR's release that slipped through the window.

Two release runs never overlap, whatever triggered them: version.yml has one concurrency group for every trigger, with queue: max, and each run that releases moves to main's tip before release-it — a run queued behind a release finds it already tagged and releases nothing (a nightly or a dispatch checks for new commits first, since a dispatch's explicit increment would otherwise cut an empty release). Overlap is what must never happen — when a push is rejected, release-it rolls back by deleting the remote tag of the version it computed, and if the other run had just pushed that same version, that is the other run's tag. queue: max is there because a group otherwise keeps a single pending run and silently replaces it with the next one: a merged pull request's run could replace a pending nightly or dispatch, then hold its release while the queue builds.

A rebuilt group's older runs are cancelled

GitHub rebuilds a queue group whenever its base moves — a push to main, an entry ahead failing or leaving — and leaves the discarded group's runs to finish. They hold arc-runners pods for a verdict nobody reads, which is the churn half of the queue's cost: over 2026-10-21/22, 14 of 38 completed merge_group runs were red and the five pull requests involved were rebuilt repeatedly, each rebuild leaving a full Tests run behind it.

.github/workflows/merge-queue-churn.yml cancels them, and only them. The branch of a queue entry is gh-readonly-queue/<base>/pr-<n>-<base sha>: the pr-<n> prefix survives a rebuild, the base sha is the generation. A run is cancelled when it carries the same prefix as the run doing the cancelling, on a different branch, is not completed, and was created strictly before it. So the newest generation kills the ones it replaced; every job of one generation shares its branch and cannot touch its siblings (the three race shards included); an equal timestamp cancels nothing, since a tie is not an order and the cost of guessing is a required check reported cancelled — a red entry.

Required check NAMES are untouched: the janitor is its own workflow, nothing requires it, and the ruleset never waits for it. It runs on ubuntu-latest on purpose — queued behind the pod shortage it exists to relieve, it would arrive after the runs it was meant to kill — with actions: write as its only scope and no checkout. internal/ciguard/queue_churn_test.go runs the shipped jq filter against crafted run listings, in both list orders.

Measuring the queue

devbox run -- task ci:queue-stats -- --since 2026-09-22 --until 2026-09-28 (scripts/diagnostics/queue-stats.sh, read-only, needs an authenticated gh) answers the three questions a queue change has to be judged on, over a fixed window in UTC so two periods compare:

• how long a merge takes — minutes from a pull request's first enqueue to its merge, p50/p75/p90/max, per author (the brew-tap bot's PRs are counted apart), for the pull requests the queue merged; the ones merged directly — the admin bypass, a dequeue followed by an admin merge included — are counted apart, since each is a push to main. Also the removals from the queue that were not a merge (failed_checks, merge_conflict, manual), among the pull requests merged in the window;
• what the window paid for — the Tests runs on merge_group, and for every rebuild, its cause, read from the new group's base — the sha gh-readonly-queue/main/pr-<n>-<base sha> carries: a base that moved to a direct push to main (a release, a direct merge, any other push) IS that push, whatever the delay, while an unchanged base, a queue merge or a speculative queue branch is the queue itself (an entry ahead failed or left, a dequeue and a re-enqueue) — and the state of the build it replaced: for a push, whether it was already green or still running when the push hit (GitHub does not cancel the run of a group it discards — the janitor below does, so a discarded build now reads as cancelled rather than as a full run nobody waited for; a build that had already finished red had ejected its pull request, so its rebuild is a re-enqueue, counted as the queue's). Queue builds discarded by a direct push is the line the release hold above exists to shrink. The attribution is queue-rebuild-cause.awk, its own program so a harness can run it: internal/ciguard/queue_stats_test.go holds each of its rules to a crafted table — the previous rule blamed a push only within 90 seconds of the rebuild, and GitHub rebuilt #1967's group 114 seconds after v3.217.0's commit, so the discarded build was counted as the queue's own churn. Runs are read day by day and checked against the count the API announces: a query answers at most 1000 runs, and paging has returned a run twice;
• what failed the queue — for each failed build, the failing jobs and steps and, unless --no-logs, the --- FAIL: tests in their logs, ranked. A build is named after the pull request at the tail of its group, so a failure that an entry ahead caused repeats in every group behind it. A test that fails the queue once and never again is a flake candidate; one that fails it with a generated-file freshness message is two PRs editing the same generated artefact.

For the bot factory

A bot opens a PR → the operator/agent reviews (the adversarial in-loop review already ran during the bot loop) → gh pr merge <n> --auto --squash enqueues it. Overlapping bot PRs (e.g. two per-run store features) are then serialized and rebuilt-combined automatically — no more manual rebase-and-fix.

scripts/merge-guard.sh remains as a local pre-flight (rebuild the combined tree before enqueuing) when you want to check overlap before spending queue CI.

Prerequisite that made this viable

The race job was flaky (timing ceilings too tight under the CI -race full-parallel load). A flaky required check stalls the whole queue, so the dispatcher test deadlines were raised to generous ceilings (commit 5368500f5) before enabling the queue.

Reverting

The ruleset is instantly reversible by an admin: gh api -X DELETE repos/SocialGouv/iterion/rulesets/18857412.

A job reports inotify “too many open files” (#1198)

inotify_init can report EMFILE for the process descriptor limit or the real user's inotify-instance limit. The latter can be shared by several runner containers with the same host UID. Read the job's Watcher resource limits step and the failing process's watcher resources error before choosing a remedy. The error retains its original errno and reports the real UID, descriptor ceilings, visible process descriptors, an ordinary file-open observation and the inotify sysctls. It neither changes a limit nor skips the watcher test. Counts are observations at failure time, not an atomic inventory of every container on the node.

An ordinary file opening successfully while inotify initialization returns EMFILE, with descriptors well below nofile_soft, points toward the per-user instance ceiling. A failed ordinary open and a reached descriptor ceiling point toward process FD exhaustion. Confirm with the runner/node operator before changing shared limits or concurrency. The configuration in SocialGouv/infra-apps/arc-runners/values.yaml controls the shared scale set; the repository's CI_SELF_HOSTED=off switch remains the emergency fallback, not a silent automatic response to this error.

Bounded reproduction on the ARC worker

On 2026-09-13 an ARC CI job reported real UID 1001, an identity UID map, RLIMIT_NOFILE=1048576, max_user_instances=128 and max_user_watches=228254. A separate diagnostic Job on worker-nodepool-node-e6dab0 (Linux 5.15.0-134-generic) reproduced that 128-instance boundary under the previously checked unused UID 2147480000:

{"uid":2147480000,"max_user_instances":128,"nofile":[1048576,1048576],"child":{"held":128,"errno":24},"other_process_inotify_errno":24,"ordinary_fd":"open_ok"}

One process held all 128 instances; the next call in it and a second process failed with EMFILE, while /dev/null still opened. Closing the descriptors immediately restored capacity. The Job added no watches, changed no sysctl and was deleted after completion. No runner remained during inspection, so this is a reproduction of the shared resource boundary, not a measurement of the historical CI peak.

The exact Job manifest is retained for operators. Before an approved repeat, choose the node explicitly and verify the dedicated UID has no processes/inotify instances on that node; never substitute the runner UID. The script refuses ceilings above 512, has no retries, and is bounded by a 120-second deadline and 128 MiB memory limit. It requires Linux and the pinned image's Python 3. It is an operator diagnostic, not a CI step or an automatically deployed resource.

A proposed configurable floor is tracked separately in the infrastructure PR linked from #1198. Keep the ticket open until approved activation and representative concurrent CI runs demonstrate the result.

The operator direct-landing lane (decided 2026-10-22)

Operator-led chantiers may bypass the queue entirely: no PR, a train (the branch rebased onto origin/main, the full required-check set reproduced locally ON THE INTEGRATED TREE), then a direct push to main. The lane's contract — the exhaustive local gate, the converged adversarial loop, the Adversarial-Rounds: / Adversarial-Model: / Landing-Train: trailers, the post-hoc CI witness, the red-main priority, and the duty to warn sessions with queued PRs or unpushed branches — lives in the review-and-merge page. Every existing PR keeps the queue and its revi/review blocking gate; the lane is the admin bypass generalised from hotfixes to reviewed chantier work, and its bypass is greppable in the trailers by design.