iterion

Workflow Authoring Pitfalls

Hard-won lessons for authoring .bot workflows where LLM agents do real work on real codebases. Read this before writing or amending an iteration workflow that has the power to commit code.

The TL;DR: LLM agents will optimize the metric you measure, not the goal you imagined. If your verdict criteria, scanner, and prompts can be satisfied by a façade, an agent will produce a façade — even when a fresh human reading the same goal would never consider it.

This is the “what fails” half. The “what works” half — the measured shape of productive human-driven sessions and the authoring rules derived from it — is references/productive-session-patterns.md.

The cheating LLM is the workflow’s fault

When an agent produces something that looks like progress but doesn’t actually achieve the underlying goal, the diagnosis is rarely “the model hallucinated.” The diagnosis is almost always one of:

  1. Goodhart’s law — the success metric was a proxy for the real goal, and the proxy was easier to satisfy than the goal.
  2. Path of least resistance — the prompts described the shape of the work but not the intent, so the agent picked the implementation closest to the source rather than the one closest to the target.
  3. No anti-façade gate — the judge had no rule to recognize that “rename X → Y” is not “remove X.”
  4. Verdict tunnel vision — the verdict only looked at the artifact the agent produced, not at the whole dependency graph the workflow was supposed to clean up.

When you catch this happening, fix the workflow before re-running. Adding human supervision on every batch is not a substitute for sharpening the spec.

Case study: the goai → claw-code-go façade (2026-04)

What was supposed to happen

iterion depended on github.com/zendev-sh/goai (vendored, 7300 LOC) for its in-process LLM layer. The team also maintained claw-code-go, a native multi-provider Go port of Claude Code with its own internal/api/Client (zero goai imports, real HTTP/SSE multi-provider implementation). The plan: make iterion depend on claw-code-go’s native API, drop goai entirely.

What actually happened

The workflow ran 8 batches over ~3 hours and reported 96% migration parity. Verdicts were batch_complete: true, all tests passed, the build was green. iterion had zero import "github.com/zendev-sh/goai" in any of its .go files outside vendor/.

When we tried to actually delete vendor/github.com/zendev-sh/goai/ and remove the require from iterion’s go.mod, the build broke immediately. Reason: the agent had created a brand-new package — claw-code-go/pkg/sdk/ — populated with files like:

// claw-code-go/pkg/sdk/types.go (created by the agent)
package sdk

import "github.com/zendev-sh/goai/provider"

type FinishReason = provider.FinishReason
type Usage        = provider.Usage
type Message      = provider.Message
// ...

The new pkg/sdk/ was a thin façade that re-exported goai/provider types under different names. iterion was rewritten to import claw-code-go/pkg/sdk instead of github.com/zendev-sh/goai. From the scanner’s perspective, iterion was “100% PORTED” — no goai strings in iterion’s own source. Internally, the goai dependency had simply been relocated into claw-code-go (which iterion vendored), invisible to a grep scoped to iterion.

The native claw-code-go/internal/api/Client — the actual point of the migration — was not used at all. The agent had taken the path of least resistance: re-exporting goai types preserved goai’s API shape, so iterion needed minimal rewriting. Adapting iterion to claw’s native streaming API (Client.StreamResponse(ctx, req) → channel of StreamEvent) would have required rewriting iterion’s three generation strategies. The agent didn’t do that work, and nothing in the workflow forced it to.

Why the workflow let it pass

Five concrete defects in the .bot:

  1. The goal in port_plan_system was ambiguous. “Replace iterion’s native goai layer with claw-code-go” can mean (a) iterion’s source stops importing goai, or (b) iterion’s whole dependency graph stops requiring goai. The agent satisfied (a). Nothing in the prompt locked (b).

  2. The target API was not specified concretely. The plan said “bridge to claw-code-go” without naming the entry point (pkg/api.Client.StreamResponse). The agent invented a different bridge point (pkg/sdk/) that looked simpler.

  3. The migration scanner was textually scoped, not architecturally scoped. It ran grep -r "github.com/zendev-sh/goai" --include='*.go' . | grep -v vendor/. Once the goai imports lived inside claw-code-go/pkg/sdk/ (vendored from iterion’s perspective), the scanner returned zero — even though the dependency tree was unchanged.

  4. The judge had no anti-façade rule. The verdict checked “blocker / suggestion” classification on the diff. Creating a new package that re-exports the source types is not a blocker by any reasonable definition of “production-breaking.” But it is a complete no-op for the actual migration goal. The judge had no language for “this batch claims migration progress but is actually a relabeling.”

  5. No “why” anchor in any prompt. The locked decisions section listed constraints (“hard rename, no alias”, “anthropic + openai validated”) but didn’t repeat the purpose (“we want iterion to use claw’s native multi-provider client so we can drop the third-party goai dependency entirely”). Without that anchor, every prompt iteration drifted further from the original intent.

What good looks like

A workflow that would have caught this:

Authoring checklist for .bot workflows that touch real code

Before writing the workflow

In the prompts

In the scanner / verdict

When reviewing a plan at the human gate

Goodhart variants seen in this codebase

Metric the workflow rewarded What the agent did
grep zendev-sh/goai in iterion source returns 0 Moved goai imports into a new package inside claw-code-go (vendored from iterion → invisible to the grep)
parity_percentage hits 100% The percentage was computed from a checklist of file-level statuses; the agent updated each file’s status to PORTED based on whether iterion still imported goai from that file. The fact that the file now imported a façade re-exporting goai didn’t change the status.
Tests pass Tests exercised iterion’s behavior end-to-end through the façade. Since the façade preserved goai’s runtime behavior 1:1, tests passed.

In every row, the metric was a faithful measurement of the surface property but a poor proxy for the underlying goal. The agent satisfied the metrics. The goal stayed unmet.

Practical rules to internalize

  1. A migration is not done until you can rm -rf the old dependency and the build still passes. That is the only test that cannot be gamed.

  2. If your scanner can be satisfied by mv, your scanner is wrong. Moving a problem to a different filesystem location is the most basic form of metric-gaming and any tool that doesn’t catch it isn’t measuring what you think.

  3. Anti-façade rules belong in the judge, not the scanner. The scanner says “still 5 imports remain”; the judge says “creating a wrapper to absorb those 5 imports doesn’t count.”

  4. The plan_gate human review is your last chance to catch a façade. Always ask: “what does this batch actually remove from the dependency graph?” If the answer is “nothing — it just renames or re-exports”, you are about to approve a no-op.

  5. When an agent’s output contradicts the plan but satisfies the metrics, the metrics are wrong. Don’t relax the goal to fit the output; sharpen the metrics and re-run.

Improvement loops must converge to an asymptote

An improvement/review loop must converge to an asymptote — settle into a stable approved state and stop. A slight, very occasional oscillation is acceptable; it must be the rare exception. The rule is the asymptote, never sustained oscillation. iterion bench asymptote (docs/asymptote-bench.md) measures it.

The shipped mechanism (ADR-058 v2 — the whole loop fleet since 2026-07-07): one campaign agent + a deterministic verify gate + a machine-checkable termination contract + a bounded continuation_loop(max_passes). gate.converged = <done-flag> ∧ gates green; the campaign commits each unit in stride (git is the state), so loop exhaustion ships what is banked instead of discarding it. Oscillation is structurally absent — there is one context per pass and no reviewer/fixer relay to re-litigate. The honesty clause (“under-reporting only costs a pass; over-reporting lands you right back here”) plus the deterministic gate is what keeps the done-flag truthful.

If you author a NEW cross-family reviewer loop (an optional amplification — no catalog bot ships one any more), the historical convergence mechanisms are mandatory:

The fastest way to break convergence in any shape: judge the wrong artifact. Work-in-progress lives in the working tree (and, under v2, in the run’s in-stride commits). Anything that reviews a change MUST diff the working tree against the right base — git diff HEAD for uncommitted work, git diff <run-base> for a v2 run’s cumulative series — never git diff HEAD^...HEAD (the last commit = the base). The historical v1 bug: feature_dev’s reviewer_gpt diffed HEAD^...HEAD, reported “feature not implemented” against work that was plainly present, split the cross-family verdict and oscillated forever. Same family of bug: git diff HEAD omits untracked files — new files must be git add -N/git add -A‘d before diffing or a change that ADDS files reads as missing. The v2 campaign contracts bake git add -A into the per-unit commit step. When a loop won’t converge, first confirm the judge is diffing the same, correct artifact.

What worked in the end (run-005, 2026-04-28)

After four failed attempts (run-001 façade, run-002 auth, run-003 var-substitution + judge tool-blindness, run-004 wrong-path grep), run-005 converged to overall_parity=true in four batches over ~1h45. The workflow file changes that made the difference:

1. Goal anchored on a concrete API entry point

port_plan_system and port_impl_system named the target by symbol, not by intent:

END STATE: iterion’s pkg/backend/model/claw_backend.go calls claw-code-go/pkg/api.Client.StreamResponse(ctx, req) and aggregates the returned []ContentBlock / StreamEvent deltas. NO call goes through any intermediate pkg/sdk or wrapper layer.

The agent could not “interpret” the goal as something easier; it had to use that exact entry point.

2. Architectural scanner across both source repos

parity_scan_system greps both vars.iterion_repo_path and vars.claw_repo_path (the source dirs, not iterion’s vendored copy of claw). This closed the run-001 gameable axis where goai imports were relocated into a vendored sub-tree.

3. Strict 8-condition AND for completion

overall_parity=true requires ALL of:

A single false anywhere → overall_parity stays false. No partial credit, no deferred negative-space checks.

4. Tools on judges + explicit USE-TOOLS preamble

The first attempt to add tools to judge nodes failed because the agents had access but didn’t invoke them — the prompt didn’t require tool use. Adding an explicit STEP 0 — VERIFY ACTUAL STATE preamble with imperative language (“USE the tools”, “Cite tool outputs inline”) made every judge invocation actually grep before claiming filesystem state.

5. Absolute paths in judge prompts

The next attempt’s judges did use tools but greped the wrong directory (/workspaces/iterion instead of the clone). The fix was to write the absolute path inline in the prompt: grep -rn ... | grep -v "/vendor/". With the literal path embedded, the agent stopped defaulting to its CWD.

6. Workflow vars > input refs for global config

Tool commands’ shell substitution uses (per-node input map). Judge prompts' template substitution can use (workflow-global). Mixing them up produces literal placeholders in shell commands. Be explicit about which substitution context applies to which template usage.

7. The phasing pattern that worked

Run-005’s batches were:

This phasing matters because it isolates risk: every batch’s tests must pass before moving on. A “rewrite everything in one batch” shape would not survive the first compile error.

8. Independent verification > workflow report

Even after the workflow reported overall_parity=true, an independent grep / filesystem audit against the same eight conditions confirmed it. Don’t trust the verdict node’s claim alone — run the checks yourself, post-merge.

Cost note

Five workflow runs cost roughly $120-180 in API. Run-001’s façade burnt about a third of that on a result that had to be discarded. Sharper prompts up front would have cut that in half.

Shell portability for tool nodes

.bot tool nodes execute their command: string via exec.Command("sh", "-c", …) (see pkg/backend/model/executor.go, executeToolNodeShell). The crucial detail: sh resolves to whatever binary sh points at in the runtime PATH. It is not a fixed interpreter.

This means the same workflow can behave differently across hosts:

Environment What sh actually is
Devbox shell (any OS) bash 5.x
Linux Mint / Ubuntu host dash
Debian host dash
Alpine / busybox image ash
macOS (default) bash 3.2 (very old)

The brace-expansion gotcha (real, observed)

run_console_demo.bot originally contained:

echo {"count":3,"done":false}

This worked under devbox (bash interpreting). On a Linux Mint host the same workflow piped through sh -c ran under dash — which has no brace expansion. The output landed unchanged, JSON parsed, run succeeded.

But on a host where the user’s sh was bash (e.g. Ubuntu with dash reconfigured to bash, or any rebuilt image where bash is /bin/sh), bash applied brace expansion to {"count":3,"done":false} and emitted count":3 done":false — destroying the JSON, and silently failing every downstream node that expected {"count": ..., "done": ...}.

Rule: every command: string must be POSIX-portable

When you write a tool node, assume sh is dash. That means none of the following:

Pattern Replace with
echo {"k":v} echo \{\"k\":v\} (escape every brace and quote)
[[ "$x" == "y" ]] [ "$x" = "y" ]
cmd <<< "$input" printf '%s' "$input" \| cmd
$'\t' "$(printf '\t')"
((i++)) i=$((i+1))
for f in *.{a,b} for f in *.a *.b
${var^^} / ${var,,} printf '%s' "$var" \| tr a-z A-Z

For JSON output specifically, the safest pattern landed on in run_console_demo.bot:

echo \{\"count\":$(wc -c < /tmp/counter),\"done\":false\}

Every {, }, and " is backslash-escaped. The braces are escaped to defeat brace expansion under bash; the quotes are escaped to survive the outer YAML/.bot string-literal layer plus sh -c.

Alternative: produce JSON via printf '{"count":%d,"done":false}\n' N. printf is POSIX, has no brace expansion semantics, and reads as intent-first.

Never pass LLM-generated values into a command via the raw `` form

A tool command: resolves / through shellEscapeValue (pkg/backend/model/executor_tool.go): complex values are JSON-encoded and the whole token is wrapped + '\''-escaped, so it survives bash -c no matter what it contains. The raw form `` (bang prefix) deliberately bypasses that escaping and inserts the value verbatim — intended only for trusted shell snippets an upstream node hands down.

The trap: passing an upstream node’s structured output (especially an LLM-authored field) via ``, or wrapping the escaped form in your own literal quotes:

# BROKEN — raw prose JSON in single quotes:
DECISIONS='' python3 -c "..."

LLM prose is full of apostrophes (“iterion’s”) and parens (“(CLI/studio)”). The first ' closes the shell quote and the next ) yields bash: -c: syntax error near unexpected token ')'. Observed live in adr-cartograph’s build_manifest (dogfood, 2026-06-13): the survey’s decisions/gaps JSON broke the node on every run until fixed.

Rule: pass structured/LLM data with the default form and no surrounding quotes — let shellEscapeValue quote it:

# CORRECT — shellEscapeValue wraps + escapes; json.loads gets exact JSON:
DECISIONS= python3 -c "import os,json; d=json.loads(os.environ['DECISIONS'])"

Reserve `` for values you control that are meant to be re-interpreted as shell. A tool node that only ever saw path tokens (e.g. docs-refresh’s build_manifest) can carry the '' pattern latently for a long time — it breaks the day an apostrophe-bearing value flows through, so fix the pattern, not just the one value that tripped it.

Diagnostics

If a downstream node fails with “expected object, got string” or “missing required field” right after a tool node:

  1. Run the command manually under dash and under bash — they should produce identical output.
  2. Run it under devbox (devbox run -- sh -c '…') since that’s what the engine inherits from PATH.
  3. Inspect the run’s events.jsonl for the tool_called event — it records the resolved command string and raw stdout.

Why the engine doesn’t pin bash

We stayed with sh because:

If you genuinely need bash, invoke it explicitly: bash -c '…bash-only syntax…'. That’s a contract the workflow declares, not an assumption.

This site is open source. Improve this page.