Iterion

Appearance

Like what we’re building? Star on GitHub

← Bot runs

product-docs (Prody) — run bilans ​

Functional, business-audience product documentation generated and maintained in a dedicated docs repository from a multi-repo product catalog. Newest run first.

2026-09-17 — profile 2: a one-product catalog documented in French from a local source snapshot, converged (run 01a0afa6-aafb) ​

  • Status: validated end to end — catalog_ingest → scan_hints → campaign (cut once by the wall, then complete) → scope_check → page_lint → gate converged → mr_gate (off) → publish_gate (off) → done.
  • Versions: bot product-docs 1.1.1 (dsl: 2, wave 3b of #1344) · iterion 5fa790d82 for the bots; the engine the branch binary v3.157.0+c31559517 (built on the wave-3a branch from main at v3.157.0; main is at v3.159.0 as this is written, one PR of engine work ahead), served through the host's Anthropic-compatible facade (z.ai) as the runs' provenance records.
  • Method: launched FROM a scratch DOCS repository holding catalog.yaml (one product greet, docs.product_dir: produits/greet, one source repository given as a local path to the scratch greeter, ref: main), a skeleton produits/greet/index.md and an editorial override .product-docs/ton-et-style.md; --var catalog_path=catalog.yaml --var product_id=greet --var max_passes=1, --store-dir the operator's workspace store, --sandbox none, --merge-into none, caps --max-cost-usd 1.5 --max-duration 10m, every LLM node on claude_code/claude-opus-5 (the GPT forfait is closed until 2026-09-20). The first campaign died at the 10-minute wall with three pages already committed; resumed with --max-duration 30m --max-cost-usd 3.
  • Result: $1.70 recorded (the cut attempt's spend is not), 20 min 10 s from launch to done. The campaign loaded the docs repository's own editorial rule before the bundle defaults, grounded every claim by reading the whole source AND running the CLI (help text quoted verbatim, the three refusals reproduced), wrote the pages in French for the non-technical reader — a glossary, a help page, the greeting journey, a blocking case (a composed name written without quotes) found by execution, not by reading — and reported drift_remaining and unread_sources empty; the deterministic scope and lint gates passed, converged: true. Storage branch iterion/run/01a0afa6-aafb-7a59-a87f-1ea7948a9d00 → e3490f3 (four docs commits across the two attempts).
  • Value: the six prompts' paragraph breaks reach the models (proven in the run's events.jsonl for campaign_system and campaign_user, twice each; the MR and publish prompts were not rendered, both gates off), and the bot's grounding contract held on a source it could only read as a snapshot: catalog_ingest copies sources without .git, the agent's first git log failed with exit 128 and it fell back to reading files — by design.
  • Findings / misses: none bot-side.
  • Engine hardening: none needed.
  • Lessons for next run: --max-duration 25m for one full pass; the cheapest fixture is a catalog entry whose url is a local path; pass --merge-into none again on resume.

2026-09-04 — first run on the prod instance: the datalab URL brought back, then re-published by the bot (run 01a06baa) ​

  • Status: validated on prod — first product_docs run ever on the cloud instance (every earlier run was local), scope ✓ lint ✓ converged, publish tail end to end, verify_publish http 200 from outside.
  • Versions: bot 1.0.0 (baked catalog) · iterion prod v3.100.0 runner / v3.100.1 server · claude_code / claude-opus-5 on the team forfait.
  • Method: repo-targeted launch on DNUM-SocialGouv/documentation_produits @ demo/prody-sirena (the only branch carrying .prody/catalog/), max_passes=0, open_mr=false, publish=true, publish_base_url = the RELAUNCHED datalab service, publish_s3_bucket=devthejo, publish_tools_ref=v3.100.0. Team secrets onyxia_s3_* (renewed STS tokens) + forge_token (the PAT connection's managed secret — the docs repo has no repo integration, so the clone credential comes from the bot binding) bound to product-docs.
  • Result: 44 min, $25.5 — campaign $25.0 (one verification pass that fanned out 49 sub-agents in two modalities, page-first then source-first), publish $0.5 (105 s). 3 commits on 2 pages, all real: the fourth distribution and two national key figures of the indicators dashboards, a third filter that silently removes requests (Inclure les EIG), the silent merge and the non-shared reopening of a request. is_product_bug: true (filed in the summary — the board MCP is unavailable in the sandboxed runner session, as for Doki). The site was rebuilt (21 pages) and mirrored; the truth gate polled the new host and got http 200, title present.
  • Delivery: with open_mr=false the 3 commits are BANKED on iterion/run-01a06baa… in the DNUM repo (the engine's death/finish bank works on a foreign repo too) and nothing reached the draft PR #24 — a deliberate choice for a "verify + publish" run, but it means a human cherry-pick or a follow-up run with open_mr=true.

What the run says ​

  • The URL had died with the platform, not with the bot. On 09-02 the STS tokens expired and the datalab culled prody-docs-serve; the bucket still held the site. Relaunching the service (Vscode-python, init script pinned at v3.100.0, SITES_BUCKET=devthejo, user port 5000) brought /sirena/ back to 200 within a minute, BEFORE any run — a new release id, hence a new host. The launcher takes its whole config as URL parameters (init.personalInit=«…», extraEnvVars[0].name=«…», networking.user.enabled=true); the brackets must stay literal, a percent-encoded %5B0%5D becomes a bogus top-level key.
  • A manual launch runs on the launcher's OWN forfait first. The first attempt (01a06b9f, POST /api/runs with the operator token) died in 6 s on usage cap: seven_day window at 97% — the operator's personal Claude connection, at its weekly ceiling, sits ABOVE the team key in the credential tiers. Relaunching through a one-shot schedule row (a system actor) resolved the team key and ran. Worth knowing before any manual prod launch on a shared team.
  • Scope notes are read by the campaign, not by the tail. The notes said "verification pass + publication"; the campaign agent took "publication" as its own job, mirrored the site to the bucket itself and probed the OLD hostname it found in the repo (404 — the culled service), then asked for an operator action the publish tail had already made moot. Harmless here, but a pass spent on the tail's work: keep scope notes about the DOCS.
  • The verification pass is no longer a $5 pass. The campaign self-orchestrates (49 sub-agents) at the bot's default effort; the local recipe ran it at high. On a converged corpus that is the price of breadth — the two findings Round 2 surfaced were in areas a page-first reader never opens.
  • Two credential leaks, both mine, both in the transcript: a full accessibility snapshot of the datalab launcher renders the helm values preview WITH the service's temporary credentials, and a value-stripping filter written for key=value lines printed the key = value ones. The tokens were renewed and the bot only ever saw the new ones; the old ones expire on 09-11. Extract by filtered evaluate, never by snapshot, and redact by printing token COUNTS, never fields.

Lessons for next run ​

  • The datalab is best effort by design (7-day STS, 7-day cull): the demo survives only as long as someone relaunches. Prody 1.1.0 makes the tail platform-agnostic (image + deploy-target); the durable target is fabrique.
  • Launch prod dogfoods through a one-shot schedule row, delete it after the tick; keep open_mr=true when the pass is allowed to write, or the commits stay on the bank branch.

2026-08-26 — first real product + first publication: Sirena live on the SSP Cloud datalab (run 01a03ed3) ​

  • Status: validated — the whole chain ran once on a real product, including the opt-in publication tail shipped the same day (#533).
  • Versions: bot 1.0.0 (publication tail on branch, merged as #533 the next morning) · iterion v3.64.1 (release binary) · skill deploy-onyxia-sspcloud v0.1.0 → v0.1.3 during the session.
  • Method: local run from the DNUM docs repo (DNUM-SocialGouv/documentation_produits, branch demo/prody-sirena), claude_code / claude-opus-5, max_passes=0 (one verification pass: the corpus had converged on 08-23/25, 22 pages), open_mr=true, publish=true, publish_base_url = the standing datalab service's user-port host, publish_s3_bucket=devthejo. Onyxia S3 STS credentials in the local sealed store (three file secrets).
  • Result: scope ✓, lint ✓ → publish_gate ready → the publish agent built the site (GitBook blocks → MkDocs Material through deploy/gitbook_to_mkdocs.py), mirrored it to s3://devthejo/sites/sirena and reported deployed: true → verify_publish http 200 from outside → surface_site_link. ≈ $17, 25 min. The six verification commits of the pass were cherry-picked onto the draft PR #24 in the DNUM repo (kept in draft: merging is the product owners' act).
  • Value: the DNUM deliverable — a business-readable Sirena documentation generated from the sources, verified, served on a URL the bot itself put live, with the RGPD finding (three retention periods documented, one implemented) kept in the promises ledger instead of aligned down.
  • Findings / misses:
    • Two runs before this one finished WITHOUT opening their PR because a file secret that was not mounted stays silent (optional: true); fixed engine-side (#531), and the bot-probe recipe (a tool node testing {{secrets.X.path}}) stays the way to check a mount.
    • MC_HOST is parsed as a URL: an STS secret carrying / silently truncated the authority. Percent-encode every part (skill v0.1.3).
    • mc's table renders on failure too; only the exit code is truth (a | tail -1 had swallowed it once).
    • The datalab bucket does not pre-exist (mc mb --ignore-existing first), diffusion/ is authenticated-read only (public exposure goes through a datalab service on the user port).
  • Engine hardening: #531 (unmounted file secret fails loudly), #526 (sandbox default image falls back to :latest between releases).
  • Lessons for next run:
    • The platform is best effort by design: STS credentials expire in ~7 days and the datalab may delete the serve service on the same horizon. Both happened on 2026-09-02 — the URL 404'd by itself. A bot that publishes weekly needs a durable target (fabrique) or a rotation.
    • A plugin-source manifest is not validated at registration: an unquoted : in plugin.yaml broke EVERY webhook launch of the instance for 2h20 (#536–#538). Validate the YAML before registering.

2026-08-25 — first dogfood, fixture product (run 01a03a6a) ​

  • Status: validated (converged), with one real bot bug found and fixed.
  • Versions: bot 1.0.0 · iterion fd9c5e611 (worktree build).
  • Method: claude_code / claude-opus-5, two campaign passes. Fixture: a docs repo (/tmp/prody-dogfood/docs, empty product tree, one local editorial override) and one source repo (subvention-app: a grant request service — FR i18n catalog, two role-scoped route files, one form template, a .env). Catalog catalog/demo.yml naming a single repo by local url:. Vars: catalog_path=catalog/demo.yml product_id=demo max_passes=2 scratch_dir=/tmp/prody-dogfood/scratch. Flags: --merge-into none, --sandbox none (no container runtime on the host), open_mr left false.
  • Result: converged on pass 2. 21 commits, one page per commit, all carrying both Bot: product-docs and Product-Docs-Sources: subvention-app@e7bb04f1. 8 pages: product home, glossary, two role hubs (espace-demandeur, espace-instructeur), two step pages each. page_lint green over all 8 (0 violations); scope_check green; gate.converged = true. ≈$20.6 (13.6 + 7.0), ≈50 min wall clock. Commits landed on iterion/run/01a03a6a… (merge deliberately skipped).
  • Value: high, and specifically the value the bot was built for. From an EMPTY product directory the campaign produced a role-then-journey page set a business reader can follow, in French, grounded file by file: statuses from fr.json, the SIRET/50 000 €/10 Mo constraints from the route validators, the "réponse sous 30 jours" promise from the mail body, the mandatory refusal motive from the refuse handler. Unprovable facts were marked [à confirmer] in-sentence rather than invented or dropped — pass 2 spent all 5 of its commits doing exactly that re-grounding.
    • The docs repo's own editorial rule was obeyed over the bundle default: .product-docs/modele.md demanded that hub pages open with ## À qui s'adresse cette page, and both hubs do — while the product home (not a hub, per the bundle's own taxonomy) correctly does not. Editorial sovereignty (ADR-092) works end to end.
    • It found a genuine source defect and said so instead of documenting a promise the code does not keep: instruction.js comments that accepting "notifies the applicant", no mailer call exists, and fr.json has no acceptance/refusal message body. Reported as is_product_bug: true and filed on the board rather than papered over.
    • It refused to resolve an ambiguity it could not prove (the worklist reads deposee, both decision handlers call transition(id, 'instruction', …), and transition is defined in none of the cloned files): it documented both ends and marked the passage between them.
  • Findings / misses:
    • The redaction is honest but blind by construction: the inventory reported "1 secret-bearing file redacted" and the campaign flagged, unprompted, that it therefore could not know what functional area that file covered. Right behaviour; worth remembering that a product whose config carries functional truth will be under-documented by design.
    • catalog_ingest failed LOUDLY and usefully three times before the run started (missing YAML parser path, unwritable scratch root) — each message named the cause and the remedy. No silent degradation observed.
  • Engine / bot hardening: one real bug, found only because this was a live run. iterion mirrors the bundle's skills into <workspace>/.claude/ at run start and again on every resume; scope_check collected untracked paths from git status and judged that engine-owned tree against the writeable set, so a pass that wrote 8 pages and 16 clean commits was failed on .claude/. Worse than a false positive: the agent then deleted the engine's own skill mirror to satisfy the gate. Fixed by excluding .claude/ in scope_check (surgically — an untracked tree that is not the mirror still fails), with both halves pinned in bots/product_docs_gates_test.go. docs-refresh escapes the same class only by accident: its writeable set is "any .md", which happens to admit the mirrored skills.
  • Lessons for next run:
    • Budget the campaign properly: --max-cost-usd 15 killed pass 1 at scope_check after the agent had already spent $13.57 and committed everything (the 90% hard limit blocks the next node). A single Prody pass on a one-repo product costs $7–14; budget max_passes × ~15 or leave the bot's own 300 ceiling alone. Resuming with --max-cost-usd 45 picked up exactly at scope_check and lost nothing.
    • iterion resume has no --sandbox flag (only iterion run does) — use ITERION_SANDBOX_DEFAULT=none on a host with no container runtime.
    • Two passes is the natural shape on a bootstrap: pass 1 authors the page set, pass 2 re-grounds it. max_passes=2 was enough here; a multi-repo product will want the default 4.
    • The fixture is worth keeping: an empty product tree + one small role-shaped source repo exercises hub/step authoring, glossary, [à confirmer] discipline and the editorial override in under an hour.