iterion bench asymptote measures the inter-session quality stabilisation curve of a workflow: rerun the same task in N independent sessions and watch the per-iteration judge verdict converge. The shape of that convergence is the asymptote — the empirical reliability ceiling of the (model + recipe) on the task.
Run the same task through the same workflow in independent sessions, plot the final judge verdict per session, and the curve climbs across early sessions then stabilises. That stabilisation is a positive signal: it proves the recipe is doing reproducible work, and the height of the plateau is the (model + recipe)’s reliability ceiling for the task. A flat tail means a single session is a trustworthy delegate; a noisy tail means run twice and merge. Multi-family alternation (Claude ↔ GPT) can raise the plateau further on critical or complex tasks (security, cryptography), but it is optional refinement, not the default thesis. The asymptote is fundamentally about consistency of one recipe across sessions.
This page is the operator’s guide.
For each run in the input set, the bench:
events.jsonl from the store and walks it.EventEdgeSelected.Data["iteration"] (the engine emits this on every loop-back edge).EventNodeFinished and reads the configured verdict field (default: output.approved).[0..1] score (booleans become 1.0/0.0; numerics pass through clamped).The output is markdown (written to --output <path> or stdout) with a side-by-side comparison when a --variant-runs group is also supplied.
# Canonical: stabilisation curve over N sessions of the same workflow.
iterion bench asymptote \
--runs run_a,run_b,run_c,run_d,run_e \
--judge-node final_judge \
--output report.md
# Numeric judge field with a non-default approval threshold.
iterion bench asymptote \
--runs run_a,run_b,run_c \
--judge-node review_judge --judge-field score --approval-threshold 0.8 \
--include-per-run --output -
# Compare a baseline against a multi-family alternation variant on a security-
# critical task. Multi-family is *optional refinement*, not the default thesis —
# only reach for it when the cost of a missed defect outweighs the extra spend.
iterion bench asymptote \
--runs base1,base2,base3 --label single-family \
--variant-runs alt1,alt2,alt3 --variant-label multi-family \
--judge-node final_judge \
--output report.md
Required flags:
--judge-node <node-id>: the IR node whose verdict is read each iteration. Must be a real node ID from the workflow (commonly reviewer_claude, judge, final_judge, etc.).Common optional flags:
--judge-field <key>: the field on the judge’s output carrying the verdict. Default approved.--approval-threshold <0..1>: score threshold for the boolean “approved” mapping. Default 0.5.--loop <name>: pin scoring to one loop name when the workflow has several.--label, --variant-label: human-friendly group names for the report.--include-per-run: append a per-run iteration list at the end (useful for spotting outliers).--store-dir <dir>: explicit store path. Defaults to .iterion/ under cwd.--output -: write to stdout instead of a file.A healthy asymptote on a calibrated workflow looks like this:
| Iter | asymptote mean | asymptote pass-rate | asymptote n |
|---:|---:|---:|---:|
| 0 | 0.42 | 40% | 5 |
| 1 | 0.74 | 80% | 5 |
| 2 | 0.92 | 100% | 5 |
| 3 | 0.94 | 100% | 5 |
| 4 | 0.94 | 100% | 5 |
iter: 0 1 2 3 4
asympt: 4 8 * * *
Read this as: across 5 independent sessions of the same workflow, iteration 2 is the first time every session converges to “approved”, and iterations 3-4 confirm the stabilisation. The asymptote sits at iteration 2; the small lift to mean=0.94 in iter 3-4 is residual noise on the verdict prompt, not real quality progress.
A broken curve looks like one of these:
| Symptom | Likely cause | What to fix |
|---|---|---|
| Curve flat at low values across all iterations | Judge prompt approves nothing, or looks at the wrong field | Re-prompt the judge; verify --judge-field matches the workflow’s schema |
| Curve climbs then drops | Implementer regresses on later iterations (often: budget pressure, dropped context) | Check events.jsonl for budget_warning; raise the iteration cap or compaction |
| Curve climbs then plateaus below 1.0 | Judge prompt is over-strict on minor non-blocking issues | The verdict prompt is the load-bearing piece — re-prompt to drop nits / false positives |
| n column drops from 5 → 1 across iterations | Most runs hit the loop cap or terminated early; only outliers reach late iters | Increase the loop budget, or interpret late iters as “the difficult cases only” |
The bench supports two modes:
--runs ...): the canonical asymptote — a single recipe re-run N times. The report shows one column set and the per-run series.--runs ... + --variant-runs ...): comparison. The report adds a Δ pass-rate column. Use this for A/B-ing recipe variants — most usefully, comparing a single-family baseline to a multi-family alternation variant on security-critical or complex work, to quantify whether alternation lifts the asymptote enough to justify the extra cost.Multi-family alternation is not the default thesis. The asymptote thesis is fundamentally about consistency of one recipe across sessions; alternation is an optional refinement when the failure cost is high.
--loop <name> explicitly..bot source. To distinguish single-family vs multi-family runs, label them on the CLI (--label/--variant-label) rather than expecting the bench to infer it.events.jsonl and artifact schema the bench parses.