Scheduling recurring bot runs (iterion schedule)

Some bots are meant to run on a clock, not on demand — a weekly security audit, a nightly docs-refreshment pass, a periodic dependency sweep. iterion schedule wires those into the host's own cron so they fire on time without keeping an iterion process resident. This is the complement to iterion dispatch: the dispatcher is an always-on loop reacting to a tracker; the scheduler is a set of cron triggers reacting to the clock.

Model

A declarative manifest is the single source of truth. schedule install materialises it into a managed block of the host crontab; each cron line calls iterion schedule run <name>, which re-reads the manifest and executes the run in-process. Because the host scheduler is the trigger, nothing iterion needs to stay running between firings.

~/.iterion/schedules.yaml          # the manifest (override with --manifest or $ITERION_SCHEDULES_FILE)
~/.iterion/logs/schedule-<name>.log  # per-schedule stdout+stderr, appended each run
host crontab                       # a managed block, markers below, regenerated by `schedule install`

The manifest is host-wide (a host has one crontab), and every entry carries its own workdir, so a single manifest can schedule bots across several repositories.

Manifest format

version: 1
schedules:
  - name: sec-audit-source-weekly      # unique; used in the crontab line + log filename
    cron: "0 2 * * 1"                   # standard 5-field expression, passed opaquely to host cron
    bot: bots/sec-audit-source/main.bot
    workdir: /home/jo/lab/ai/iterion    # cd here before running; bot path resolves against it
    store_dir: ""                        # optional override; empty uses project-store resolution
    sandbox: ""                          # optional --sandbox override (none|auto)
    timeout: "2h"                        # optional max run duration (guards a hung run)
    vars:                                # optional --var overrides (commas kept verbatim)
      label_source: sec-audit-self
    description: Weekly SAST self-audit  # optional; emitted as a crontab comment
    disabled: false                      # keep in the manifest, leave out of the crontab
    overlap: skip                        # optional: skip (default) | allow | keepalive — see "Overlap policy"
    max_concurrent: 0                    # optional, with overlap: allow — cap on live runs (0 = unlimited)
    guard: ""                            # optional pre-launch sh -lc gate — see "Guard command"
    guard_timeout: "30s"                 # optional guard subprocess timeout
    guard_var: guard_output              # optional var name receiving the guard stdout

With an empty store_dir, the run uses the normal resolution anchored at the bot path: an existing managed project .iterion, otherwise the deterministic slot under $ITERION_HOME/projects/ (normally ~/.iterion/projects/). Set an explicit <workdir>/.iterion when scheduled runs must appear in a studio bound to that workspace store.

Commands

Command	What it does
iterion schedule add <name> --cron … --bot … [--workdir …] [--var k=v]… [--store-dir …] [--sandbox …] [--timeout …] [--description …] [--disabled] [--overlap skip|allow|keepalive] [--max-concurrent N] [--stale-after 5m] [--guard …] [--guard-timeout 30s] [--guard-var guard_output]	Add or update an entry (upsert by name). Overlap, keepalive (--stale-after), and guard flags mirror the manifest fields — see "Overlap policy", "Always-on agents", and "Guard command".
iterion schedule list [--json]	List manifest entries.
iterion schedule remove <name>	Delete an entry from the manifest.
iterion schedule run <name> [--dry-run]	Execute one entry now — what cron invokes. Applies the overlap policy and the guard before launching. --dry-run prints the resolved iterion run command without executing.
iterion schedule audit [--name X] [--surface host-cron|trigger|cloud] [--since 24h] [--tail 50] [--json]	Show the tick-decision history: every fired / skipped / guard outcome with its reason — the deterministic answer to "why didn't my scheduled bot fire last night?".
iterion schedule install [--print] [--tz UTC]	Sync the manifest into the host crontab. --print renders the block to stdout without touching the crontab (works even where crontab is absent).
iterion schedule uninstall	Remove the iterion-managed block from the host crontab (manifest left intact).

--manifest <path> is available on every subcommand.

add/remove only edit the manifest — run schedule install afterwards to push the change into the crontab.

The managed crontab block

schedule install reads the current crontab, replaces (or appends) the block delimited by:

# >>> iterion schedules (managed by `iterion schedule install`) >>>
…
# <<< iterion schedules <<<

User-authored crontab lines outside the markers are preserved untouched, and re-installing is idempotent (the block is replaced, never duplicated). Two details make scheduled runs actually work in cron's minimal environment:

• CRON_TZ=UTC (override with --tz) — honoured by cronie/Vixie cron so schedules fire in the intended zone regardless of host local time; on other cron implementations it is a harmless env var, so verify the firing time against your cron's timezone semantics.
• PATH=<install-time PATH> — captured when you run schedule install, so docker, git, and any scanner binaries the bot needs are reachable from cron.

A rendered block looks like:

# >>> iterion schedules (managed by `iterion schedule install`) >>>
# Managed by iterion — edit the manifest then `iterion schedule install`.
# Remove with `iterion schedule uninstall`.
CRON_TZ=UTC
PATH=/usr/local/bin:/usr/bin:/bin:…
# Weekly SAST self-audit
0 2 * * 1 cd /home/jo/lab/ai/iterion && /usr/local/bin/iterion schedule run sec-audit-source-weekly --manifest /home/jo/.iterion/schedules.yaml >> /home/jo/.iterion/logs/schedule-sec-audit-source-weekly.log 2>&1
# <<< iterion schedules <<<

Example — weekly self-audit

iterion schedule add sec-audit-source-weekly \
  --cron "0 2 * * 1" --bot bots/sec-audit-source/main.bot --workdir "$PWD"
iterion schedule add sec-audit-deps-weekly \
  --cron "0 3 * * 1" --bot bots/sec-audit-deps/main.bot --workdir "$PWD"

iterion schedule run sec-audit-source-weekly --dry-run   # sanity-check the resolved command
iterion schedule install                                  # write the crontab block
crontab -l                                                # verify

The audit bots label their findings source:sec-audit-self on the native board (see Security in CLAUDE.md). They pin the iterion-sandbox-sec image via sandbox.image, so the host needs that image present (CI publishes it; for a local loop, docker tag your build to ghcr.io/socialgouv/iterion-sandbox-sec:edge).

Note: sec-audit-source (SAST) is production-ready. sec-audit-deps (SCA) now has a real CVE floor — run_generic_heuristics runs trivy fs --scanners vuln over the workspace from a bare checkout, matching pinned versions against the OSV/GHSA/NVD DB. The per-ecosystem npm/pip-audit and code-pattern/typosquat malware signals remain partial, so a run still self-labels with a "⚠ Coverage" banner — but it is no longer a zero-finding scaffold.

Overlap policy — skip is the default (behavior change)

Since the schedgate integration, iterion schedule run does not fire while a previous run of the same schedule is still live (any non-terminal status: running, queued, paused_*). Before, every tick launched unconditionally — a nightly bot that overran its window silently piled up concurrent runs on the same repo. Skip-by-default fixes that latent bug; the trade-off is deliberate and loud:

• Every decision is recorded in the tick audit (<manifest-dir>/logs/tick-audit.jsonl, read with iterion schedule audit). A skipped tick writes skipped_overlap with the blocking_run_id, so you find out which run held the slot instead of wondering why nothing ran.
• A stale run stuck in running (crashed host) blocks its schedule until it is reconciled or cancelled. iterion does not guess a staleness cutoff: the audit names the run — inspect it, then iterion resume or cancel it. The studio/server's orphan reconciliation flips flock-releasable orphans automatically.
• Opt out per entry with overlap: allow (unbounded) or overlap: allow + max_concurrent: N (fire while fewer than N runs of this schedule are live, counting the one about to start).
• After upgrading, watch iterion schedule audit --since 48h once: a schedule that used to rely on implicit overlap shows up as skipped_overlap there.

Overlap counting keys on run provenance: schedule-launched runs are stamped source.kind: schedule + source.schedule_id in run.json, which also makes them attributable in the studio and queryable via the store.

Always-on agents — overlap: keepalive

overlap: keepalive runs a bot continuously: every tick relaunches it, but with at-most-one-live semantics and a staleness cutoff. It is how you keep an agent (a watcher, a poller, your own long-lived bot) alive as a stream of fresh, individually-budgeted runs rather than one immortal run that fights max_duration, per-node deadlines, and the cloud budget ceiling.

The only difference from skip: a run that is silent past stale_after (default 5m; a running run whose last progress is older than the cutoff) stops counting as live, so the next tick relaunches a fresh run and the zombie is reaped (running → failed_resumable, so it is still inspectable/resumable). This closes the "crashed run stuck in running blocks the schedule forever" gap above — a dead always-on agent recovers on its own within one tick.

# schedules.yaml — an always-on watcher, relaunched every minute,
# recovered if it goes silent for 2 minutes.
- name: watcher
  cron: "* * * * *"           # host crontab floors at 1 minute
  bot: bots/my-watcher/main.bot
  overlap: keepalive
  stale_after: 2m

• Sub-minute cadence is not expressible on the host crontab (its floor is 1 minute). For a faster pulse, run the resident scheduler (iterion studio / iterion server) and author the bot with a kind: keepalive invocation carrying an interval: (see below) — the in-process scheduler ticks every 15s by default (ITERION_SCHEDULER_INTERVAL).
• max_concurrent is invalid with keepalive (at-most-one-live is the point).
• stale_after should be ≥ the bot's max_duration so a long-but-alive run is never falsely reaped. Staleness only ever applies to running runs — a paused_waiting_human run is legitimately idle and never reaped.

Authoring an always-on bot (kind: keepalive)

A bot declares its always-on capability in its manifest.yaml; the cadence lives on the invocation (not baked into the bot's logic):

invocations:
  - kind: keepalive
    keepalive:
      interval: 30s            # >= 5s; sub-minute needs the resident scheduler
      stale_after: 5m          # optional (default 5m)

Enable it one-click from the bot's studio home (the /bots/{name}/triggers/from-invocation route), or toggle Always-on on a schedule in the studio Schedules tab. To also steer each launched run mid-flight (correct it, nudge it), give the bot a supervisor: block — the engine auto-attaches a supervisor coordinator to every keepalive run, no extra wiring.

Guard command — fire only when there is work

guard: is an optional sh -lc snippet executed before any launch, in the entry's workdir, with ITERION_SCHEDULE / ITERION_SCHEDULE_BOT in the environment and a hard guard_timeout (default 30s) on its own context:

• exit 0 → the run fires, and the guard's stdout becomes the run's vars[guard_var] (default guard_output) — a cheap way to hand the run "the work found" (an issue list, a diff summary).
• exit non-zero → the tick is skipped (guard_blocked in the audit, exit code + stderr tail captured).
• guard breaks (spawn failure, timeout) → guard_error in the audit — deliberately distinct from guard_blocked, so "the guard said no" never masks "the guard is broken".

Example — only run the fixer when ready-labeled issues exist, and pass them in:

- name: fix-ready-issues
  cron: "*/30 * * * *"
  bot: bots/feature-dev/main.bot
  workdir: /home/jo/proj
  guard: 'out=$(gh issue list --label ready-for-agent --json number,title); [ "$out" != "[]" ] && printf %s "$out"'
  guard_var: issues_json

Idempotence stays with the workflow (mutate the state you poll — e.g. the bot relabels the issue), exactly like the dispatcher's tracker loop: the guard is a gate + input source, not a dedup engine.

Retry — a provider quota window is waited out, not re-attempted

A scheduled run that dies because the LLM provider's quota window is exhausted (the Anthropic forfait 5h / session / daily / weekly caps) is not a failure of the run: nothing about it is wrong, it just arrived while the door was shut. Retrying it inside a node's retry budget, or handing it back to the work queue, cannot help — a weekly reset can be seven days away, and each attempt costs a fresh pod against a wall that will not move.

So iterion waits for the reset and resumes, and the wait is durable (cloud mode: the intent lives in the run document, a server-side sweeper acts on it). The schedule itself is unaffected: it keeps firing on its own cadence, and the retry is about the run that already failed.

retry:
  usage_window: resume   # resume (default) | off
  max_attempts: 5        # over the run's WHOLE lifetime; never reset
  max_wait: 192h         # cap on how far ahead a retry may be scheduled (8d)
  jitter: 10m            # spread runs that share one reset instant

Where to declare it. The same block is accepted on four layers, and each one overrides only the fields it actually sets:

priority	layer	where
1	per-run override	launch API / CLI
2	launching surface	a cloud schedule row, a trigger subscription, a webhook config, a schedules.yaml entry — each exposes the same four retry_* fields
3	the bot	retry: in the bundle's manifest.yaml
4	machine default	ITERION_RETRY_USAGE_WINDOW / _MAX_ATTEMPTS / _MAX_WAIT / _JITTER

A platform ceiling (ITERION_CLOUD_RETRY_MAX_ATTEMPTS, ITERION_CLOUD_RETRY_MAX_WAIT) is applied last and can only lower a resolved policy, so a tenant cannot reserve a hundred attempts over thirty days on their own schedule.

Local vs cloud — the wait is held differently. In cloud mode the intent is durable, so nothing extra is needed. On a host-crontab schedule the wait is held in-process by the bounded auto-resume loop, which stays opt-in: the retry_* fields on a schedules.yaml entry shape that wait, they do not enable it. Add --auto-resume N (or ITERION_AUTO_RESUME) to turn it on. The reason it is not on by default is that a local iterion run blocks your terminal, and silently sleeping it for 33 hours is not a default anyone would want.

Choosing a value. The question a bot author is answering is is this output still worth having late? A weekly digest is (resume); a "what changed in the last hour" report is not (usage_window: off — let the next tick produce a fresh one). max_wait bounds one scheduled sleep, not the run's total lateness: if the parsed reset is farther away, iterion schedules the retry at that horizon and re-evaluates when it wakes. For example, 36h means "wake no later than 36 hours from now and check again"; max_attempts is the separate whole-lifetime bound on repeated waits.

What you see. The run stays failed_resumable while it waits, with a run_retry_scheduled event naming the instant, the attempt number and how the instant was derived (reset_source). When the window reopens the run emits run_auto_resumed and continues from its checkpoint — the same pair the CLI's in-process --auto-resume loop writes, so the timeline reads identically local and cloud. A run that stops retrying records why (budget spent, admission denied, source no longer resolvable) rather than going quiet.

Which reset the wait is armed on (cloud). The credential a run fails on is the one the launch's credential walk fell through to — and the one it passed over often reopens first: a team key refused on its five-hour window reopens the same afternoon, while the platform forfait the run landed on is walled until Monday. The publisher therefore stamps the run with the earliest reopening among the credentials it skipped (skipped_cred_reopens_at, re-stamped on every resume), and the retry arms on the earlier of that and the failed credential's own reset — the event then reads reset_source: skipped_credential and carries skipped_cred_reopens_at. The resume re-resolves the whole chain, so coming back at that instant lands on the reopened key. A skipped credential that has already reopened by the time the run parks means "re-resolve now": the retry lands at the five-minute floor.

That earlier wake is a guess — the skipped credential may be refused too — and it spends an attempt of the same max_attempts budget as a wake on the real reset. So the last attempt the budget allows is reserved for the failed credential's own reset whenever that reset is still ahead and inside max_wait: the event then reads reset_source: typed_error+last_attempt_pinned (or runtime_code+…, whichever evidence named the instant). Without it, five wakes on a five-hour cycle cover 25 hours of a seven-day window and the run is abandoned days before the wall it waits on falls. Earlier attempts still take the early wake, so the recovery above is unchanged; and when the reset lies past max_wait, or the provider named no instant at all, nothing is reserved — there is no reachable wall to reserve for.

Not covered by this: a budget cap (max_cost_usd and friends) still needs a human to raise the cap and resume — retrying the same cap would re-fail instantly. Nor an auth failure, which is a credential problem time does not fix.

Retention — pair recurring schedules with iterion runs prune

Every scheduled run persists forever under <store-dir>/runs/<run_id>/; the store has no built-in retention. A weekly bot alone adds ~50 run directories a year, and a fleet of hourly/daily bots pushes that into the low thousands per month. Cap disk usage by running iterion runs prune on its own crontab line — the schedule manifest only runs bots, not arbitrary commands, so this one belongs outside the managed block:

# Weekly retention sweep — keep the last 100 runs and prune anything
# finished/failed/cancelled that is older than 30 days.
30 3 * * 1 /usr/local/bin/iterion runs prune --store-dir /path/to/workspace/.iterion --older-than 720h --keep-last 100 >> "$HOME"/.iterion/logs/runs-prune.log 2>&1

Flags mirror the semantics of the shipped statuses — see iterion runs prune --help for the full list. --dry-run is the safe way to preview what a candidate retention policy would delete before committing to it in the crontab. The command only removes <store-dir>/runs/<id>/ directories; it never touches <store-dir>/worktrees/ or anything else.

failed_resumable runs are excluded by default (they are recoverable); opt in with --status finished,failed,cancelled,failed_resumable when you have accepted their loss.

Notes & limits

• Cron expressions are passed through verbatim. iterion only checks the field count (5); range validity is the host cron's job.
• Host cron only. schedule install/uninstall shell out to crontab. On a host without it (e.g. a minimal container, Windows), use schedule install --print and wire the block into whatever scheduler you run (systemd timer, Task Scheduler, a CI cron). The manifest + schedule run <name> work everywhere.
• No TTY at firing time. Scheduled runs are non-interactive; a bot that pauses for human input will pause and persist a resumable checkpoint (resume it later with iterion resume), not block the cron job.
• A repo-bound schedule mints its clone token at the tick. A schedule whose repo_url matched a team repo integration at creation carries repo_integration_id; each tick resolves that integration's connection and mints/refreshes the MANAGED secret (EnsureManagedSecret — the same path a studio/API launch uses), pinning it as the run's forge_token. A pinned integration that cannot resolve fails the tick loudly rather than limping to a doomed clone. Only a schedule with NO pinned integration falls back to the bot's forge_token secret resolution — per-bot binding first (POST /api/teams/{id}/bots/{bot}/bindings), else any team secret named forge_token — where a hand-set token that expires kills every tick at clone (Invalid username or token) while manual launches keep working. If a schedule is stuck in that state, recreate it with the repo attached (or bind the bot to the connection's managed secret).
• A repo-bound schedule survives a re-provision. Re-provisioning the repo integration (enabling another bot, POST …/forge/repo-bots) rebuilds its schedule rows from the manifests, but each surviving bot's row keeps its id (runs and audit entries point at it), the vars the operator set — merged over the manifest's schedule.default_vars, operator keys winning — its last fire, pause state, overlap/guard policy and customised cron. A bot whose scheduled behaviour hinges on a var should still declare it in default_vars, so a row created from scratch delivers too.

Implementation

• CLI wiring: cmd/iterion/schedule.go
• Logic + manifest + crontab splice: pkg/cli/schedule.go
• Tests (manifest round-trip, splice idempotency, install/uninstall via injected crontab seam, dry-run resolution): pkg/cli/schedule_test.go