Nate Swenson

Four ways a stage looks done without being done

Issue Flow · No. 115 · 11 min read ·v0.1.0

Shipped

issueflow takes one open GitHub issue to a pull request through four stages, investigate, design, implement and test. Each stage runs as its own subagent, writes exactly one artifact, and none of them starts until a human has approved the artifact before it. The first release put that ordering in code: a blockers() function that decides what may run, and an accept() that refuses four distinct ways a stage can look finished while being nothing of the sort.

The interesting part is not the pipeline. It is what happens when you try to write the gate as a prompt instruction and discover that a model agreeing to a rule is not the same as the rule holding. This guide builds the code version, small enough to drop into a scratch directory and run.

Why the instruction is not the gate

If you have chained model calls together, you have probably written a line like do not begin implementation until the design has been approved. It works most of the time, which is the problem, because the failures are quiet and arrive as finished-looking work.

Anthropic’s guide to building agent systems draws this distinction explicitly. Writing about prompt chaining, it says you can “add programmatic checks (see “gate” in the diagram below) on any intermediate steps to ensure that the process is still on track”, and separately that agents “can then pause for human feedback at checkpoints or when encountering blockers”. The gate and the checkpoint are ordinary code sitting between the model calls. They are not things the model is asked to respect.

This is an old idea wearing new clothes. Eiffel’s design-by-contract calls a precondition “what must be true before a component is used”, and the point of writing one down is that the caller is checked against it rather than trusted to have read it. A stage in an agent pipeline is a component, its predecessor’s approved artifact is its precondition, and a subagent is the least trustworthy caller you will ever have, because it is fluent, confident, and has no memory of the rule you wrote three calls ago.

Declare the stages in one place

Start with the table everything else reads. A stage that is not here does not exist.

// stages.mjs
export const STAGES = [
  {
    id: 'investigate',
    artifact: 'investigate.md',
    // Sections the artifact must carry as headings before the next stage starts.
    requires: ['Root cause', 'Evidence', 'Unknowns'],
    needsEvidence: false,
  },
  {
    id: 'design',
    artifact: 'design.md',
    requires: ['Approach', 'Rejected', 'Files', 'Proof'],
    needsEvidence: false,
  },
  {
    id: 'implement',
    artifact: 'implement.md',
    requires: ['Changed', 'Deviations'],
    needsEvidence: false,
  },
  {
    id: 'test',
    artifact: 'test.md',
    requires: ['Command', 'Two-sided', 'Result'],
    // The one stage that must hand over a real runner's output, not a summary.
    needsEvidence: true,
  },
];

export const stage = (id) => STAGES.find((s) => s.id === id) ?? null;

// The file the test stage writes its unedited command output to.
export const EVIDENCE_FILE = 'test-output.txt';

requires is the load-bearing field. It is a contract between neighbours: the design stage needs a root cause to design against, so investigate owes it one, and the gate is where that debt gets collected. Pick these by asking what the next stage cannot start without, not by asking what a good document contains.

The gate itself

// gate.mjs
import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';
import { STAGES, stage, EVIDENCE_FILE } from './stages.mjs';

export class GateError extends Error {}

// A fresh run: every stage pending, nothing approved.
export const createRun = () => ({
  steps: STAGES.map((s) => ({ id: s.id, state: 'pending', skipReason: null })),
});

// Returns the steps that must be approved before `id` may run and are not.
// Empty means the gate is open.
//
// "skipped" is deliberately NOT approval: a skipped stage stays a hole all the
// way to ship, which is what stops a run reporting a stage it never did as done.
export function blockers(run, id) {
  const index = run.steps.findIndex((s) => s.id === id);
  if (index === -1) throw new GateError(`no stage "${id}" in this run`);
  return run.steps.slice(0, index).filter((s) => s.state !== 'approved');
}

Note what blockers() returns. Not a boolean, the actual list of offending steps, because the error message you can write from a list (“design is skipped, implement is pending”) is the difference between a gate people keep and a gate people route around.

Four refusals

// gate.mjs, continued
// A touched file is not an artifact.
const hasContent = (path) => existsSync(path) && readFileSync(path, 'utf8').trim().length > 0;

// A required section must be findable, and what makes it findable is a heading.
// Bold-only headers on their own line count too. Prose is what does not.
function hasSection(text, section) {
  const escaped = section.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
  const heading = new RegExp(`^\\s{0,3}#{1,6}\\s+.*${escaped}`, 'im');
  const bold = new RegExp(`^\\s{0,3}\\*\\*.*${escaped}.*\\*\\*:?\\s*$`, 'im');
  return heading.test(text) || bold.test(text);
}

export function accept(dir, run, id) {
  const step = run.steps.find((s) => s.id === id);
  if (!step) throw new GateError(`no stage "${id}" in this run`);
  const declared = stage(id);

  // 1. The gate is shut.
  const blocked = blockers(run, id);
  if (blocked.length > 0) {
    throw new GateError(
      `cannot accept ${id}: ${blocked.map((b) => `${b.id} is ${b.state}`).join(', ')}; ` +
        "no stage runs on anything but its predecessor's approved artifact",
    );
  }

  // 2. Nothing was written.
  const artifact = join(dir, declared.artifact);
  if (!hasContent(artifact)) {
    throw new GateError(`cannot accept ${id}: no artifact at ${artifact}; the stage produced nothing to approve`);
  }

  // 3. Something was written, but not what the next stage needs.
  const text = readFileSync(artifact, 'utf8');
  const missing = declared.requires.filter((section) => !hasSection(text, section));
  if (missing.length > 0) {
    throw new GateError(
      `cannot accept ${id}: the artifact has no ${missing.join(' section, no ')} section; ` +
        `${id} owes the next stage a heading for each of ${declared.requires.join(', ')}`,
    );
  }

  // 4. A test stage with no runner output behind it.
  if (declared.needsEvidence && !hasContent(join(dir, EVIDENCE_FILE))) {
    throw new GateError(
      `cannot accept ${id}: no command output at ${join(dir, EVIDENCE_FILE)}; ` +
        'a suite reported green without its real output is the thing this gate exists to refuse',
    );
  }

  step.state = 'approved';
  return run;
}

Refusal four is the one that pays for the whole file. A subagent asked whether the tests passed will tell you they passed. Requiring the runner’s own bytes on disk moves the claim from something generated to something recorded, and the check costs three lines.

A skip is a hole, not a pass

Every pipeline needs an escape, or people stop using it. The trick is that taking the escape must not look like clearing the bar.

// gate.mjs, continued
export function skip(run, id, reason) {
  if (!reason) throw new GateError('a skip needs a reason; an unexplained hole is indistinguishable from a bug');
  const step = run.steps.find((s) => s.id === id);
  if (!step) throw new GateError(`no stage "${id}" in this run`);
  step.state = 'skipped';
  step.skipReason = reason;
  return run;
}

// Names every unapproved step rather than the first one, so one run tells you
// everything outstanding.
export function ship(run) {
  const outstanding = run.steps.filter((s) => s.state !== 'approved');
  if (outstanding.length > 0) {
    const detail = outstanding
      .map((s) => (s.state === 'skipped' ? `${s.id} (skipped: ${s.skipReason})` : `${s.id} (${s.state})`))
      .join(', ');
    throw new GateError(`refusing to ship: ${detail}`);
  }
  return { shipped: true };
}

export const board = (run) =>
  run.steps.map((s) => ({
    stage: s.id,
    state: s.state,
    gate: blockers(run, s.id).length === 0 ? 'open' : 'blocked',
  }));

skipped is a third state, never a synonym for approved. It opens the gate for the next stage so work can continue, and it keeps failing ship() forever, with the reason attached. Saltzer and Schroeder’s fail-safe defaults principle is the same instinct: base access decisions on permission rather than exclusion, so anything you forgot to think about lands on the safe side. Here the default for “did this stage happen” is no.

Run it

// demo.mjs
import { mkdirSync, rmSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { createRun, accept, skip, ship, board, GateError } from './gate.mjs';

const dir = 'run';
rmSync(dir, { recursive: true, force: true });
mkdirSync(dir);
const run = createRun();

const attempt = (label, fn) => {
  try {
    fn();
    console.log(`OK       ${label}`);
  } catch (error) {
    if (!(error instanceof GateError)) throw error;
    console.log(`REFUSED  ${label}\n         ${error.message}`);
  }
};

attempt('accept design first', () => accept(dir, run, 'design'));
attempt('accept investigate with no artifact', () => accept(dir, run, 'investigate'));

writeFileSync(join(dir, 'investigate.md'), 'I could not determine the root cause.\n');
attempt('accept investigate with prose only', () => accept(dir, run, 'investigate'));

writeFileSync(
  join(dir, 'investigate.md'),
  ['## Root cause', 'Off-by-one in the retry counter.', '', '## Evidence', 'src/retry.js:41', '', '## Unknowns', 'Whether the timeout is related.'].join('\n'),
);
attempt('accept investigate with headings', () => accept(dir, run, 'investigate'));

skip(run, 'design', 'one-line fix, no design needed');
attempt('accept implement after a skipped design', () => accept(dir, run, 'implement'));

console.table(board(run));
attempt('ship', () => ship(run));

Put the three files in a directory and run node demo.mjs. This is the real output:

REFUSED  accept design first
         cannot accept design: investigate is pending; no stage runs on anything but its predecessor's approved artifact
REFUSED  accept investigate with no artifact
         cannot accept investigate: no artifact at run/investigate.md; the stage produced nothing to approve
REFUSED  accept investigate with prose only
         cannot accept investigate: the artifact has no Root cause section, no Evidence section, no Unknowns section; investigate owes the next stage a heading for each of Root cause, Evidence, Unknowns
OK       accept investigate with headings
REFUSED  accept implement after a skipped design
         cannot accept implement: design is skipped; no stage runs on anything but its predecessor's approved artifact
┌─────────┬───────────────┬────────────┬───────────┐
│ (index) │ stage         │ state      │ gate      │
├─────────┼───────────────┼────────────┼───────────┤
│ 0       │ 'investigate' │ 'approved' │ 'open'    │
│ 1       │ 'design'      │ 'skipped'  │ 'open'    │
│ 2       │ 'implement'   │ 'pending'  │ 'blocked' │
│ 3       │ 'test'        │ 'pending'  │ 'blocked' │
└─────────┴───────────────┴────────────┴───────────┘
REFUSED  ship
         refusing to ship: design (skipped: one-line fix, no design needed), implement (pending), test (pending)

Five refusals, and the last one still names the skip a human deliberately took, twenty lines later.

Gotchas

A required section checked with includes() is satisfied by its own negation. The first release checked text.includes('root cause') after lowercasing. An investigation artifact reading I could not determine the root cause contains that string, so the gate opened on a document whose entire content was an admission that the stage had failed. What caught it was grading a real 57-minute run against the skill’s own contract the next day, rather than re-reading the checker; the run’s artifacts were there to be looked at, and the checker had already agreed with itself. The fix in 0.2.0 requires a heading: a #-prefixed line or a bold-only line, both of which a reader can navigate to, neither of which a sentence can accidentally satisfy. The escape is to check for the shape of a section, not for the presence of its words.

A test stage that writes ok clears a naive evidence check. Requiring an evidence file only proves a file exists. In 0.1.0 that was the whole check, and a one-word file passed it. The symptom is a green pipeline with no runner output anywhere in it. The fix that landed in 0.2.0 parses the file for a real runner’s own summary, node’s test reporter, pytest, mocha, jest, go test, or a recorded exit code, and refuses when it finds none. If you build only one thing from this guide past the four refusals, build that parser, and make it read the last result in the file so a deliberately-failing first half of a two-sided proof is not mistaken for a failure.

git walks upward, so a run pointed at a subdirectory creates branches in the wrong repository. The first offline test run of the next release left a stray feature/issue-133 branch in the monorepo that contained the fixture, not in the fixture. If your pipeline creates branches, resolve the repository root explicitly and compare it against the directory you were handed before you write anything.

Do not let the gate be the only thing that orders the stages. In 0.1.0 blockers() is a prefix scan over one flat list, which is correct for a single strictly-sequential run and wrong the moment two independent lines of work exist, because it makes lane two wait for lane one. That cost a real run its second lane, and is the subject of the next release.

Sources

Changelog

  • feat(issueflow): an issue-to-PR pipeline whose gates are code, not guidance (#168) (ecd597f)