Nate Swenson

The test summary that vanished between Node 22 and 25

Issue Flow · No. 133 · 7 min read ·v0.4.0

Shipped

issueflow 0.4.0 rolls up three releases of hardening on a pipeline that turns a GitHub issue into a pull request through gated stages: the evidence gate now reads node --test’s spec reporter, the contracts each stage runs under are written into its rendered brief instead of living in the orchestrator’s head, and a finished run finally has a terminal state with a finish command that cleans up only what GitHub confirms merged. This guide is about the first of those, because the bug is lying in wait for anyone who parses a test runner’s output: the format depends on environment defaults, and defaults move.

A gate that refuses the truth

The gate’s job is honest evidence: a test stage must save its runner’s real output, and the gate parses that file for a pass/fail summary before it accepts the stage. The parser knew the TAP summary form, lines like # pass 25 and # fail 0, anchored tightly:

const pass = [...text.matchAll(/^#\s*pass\s+(\d+)\s*$/gm)].at(-1);

Then two consecutive real runs failed the gate with “nothing in it says a suite ran at all”, over evidence files that said, in plain text, that 31 tests passed. The files were fine. The summary was there. It was spelled ℹ pass 31.

That ℹ is the Node test runner’s spec reporter, and the docs now say plainly: “This is the default reporter.” It was not always. The same page’s changelog pins the move: in v23.0.0, “The default reporter on non-TTY stdout is changed from tap to spec, aligning with TTY stdout.” Before that, piping node --test anywhere got you TAP; the proposal that drove the change called the TTY-detection split out directly, “The test runner currently uses the more human friendly spec reporter when a TTY output is detected. Otherwise, it defaults to generating TAP output”, and argued the inconsistency between local runs and CI was itself the problem.

So the parser was not wrong when it was written. CI ran Node 22, where a piped run emits TAP, and kept working. My laptop ran Node 25, where the same command emits spec no matter where stdout goes, and every capture became invisible. A format contract that lives in an environment default is a contract that expires, and it expires machine by machine.

Build a parser that survives the reporter

Three moves, each earning its place. Save as evidence.js:

const RUNNERS = [
  {
    id: 'node --test',
    parse(text) {
      const pass = [...text.matchAll(/^(?:#|ℹ)\s*pass\s+(\d+)\s*$/gm)].at(-1);
      const fail = [...text.matchAll(/^(?:#|ℹ)\s*fail\s+(\d+)\s*$/gm)].at(-1);
      if (!pass && !fail) return null;
      return { passed: pass ? Number(pass[1]) : null, failed: fail ? Number(fail[1]) : null };
    },
  },
  {
    id: 'exit code',
    parse(text) {
      const line = [...text.matchAll(/^.*\bexit[ _-]?code\b\D{0,4}(\d+)\s*$/gim)].at(-1);
      if (!line) return null;
      const code = Number(line[1]);
      return { passed: null, failed: code === 0 ? 0 : null, exitCode: code };
    },
  },
];

export const RUNNER_IDS = RUNNERS.map((r) => r.id);

export function parseEvidence(raw) {
  const text = String(raw).replace(/\x1b\[[0-9;]*m/g, '');
  for (const runner of RUNNERS) {
    const hit = runner.parse(text);
    if (hit) return { runner: runner.id, ...hit };
  }
  return null;
}

Move one: strip color escapes once, at the top. When a runner detects a terminal, or when output is captured through a pty, the summary line arrives wrapped in SGR sequences; the console_codes man page gives the shape, “The ECMA-48 SGR sequence ESC [ parameters m sets display attributes.” A regex anchored with ^ and $ fails on \x1b[34mℹ pass 31\x1b[39m at both ends. Stripping per-runner means every new regex needs its own guard and most will forget; stripping once fixes the class.

Move two: both spellings in one entry, last match by position. The tempting shape is a second RUNNERS entry for the spec form. It is also wrong, in a way that only shows up on real evidence files. When a stage gets told to re-capture, it appends rather than replaces, so a real file can hold a failing spec-format run followed by a passing TAP re-capture. A parser with form precedence returns whichever format it prefers, which here means reporting the stale red run and refusing a green suite for failing. One entry, one alternation per counter, .at(-1) for last-by-position: the file’s order decides, not the parser’s taste in reporters.

Move three: when you refuse, name what you can read. The original refusal said the file held no runner result at all, over a file that said 31 tests passed; that message sent a debugging session in the wrong direction twice. Derive the capability list from the parser itself, so it cannot drift:

import { parseEvidence, RUNNER_IDS } from './evidence.js';

export function acceptEvidence(raw, path) {
  const result = parseEvidence(raw);
  if (!result) {
    throw new Error(`${path} holds no summary in a format I can parse. I read: ${RUNNER_IDS.join(', ')}.`);
  }
  return result;
}

Verify it with literal strings, never with the runner

node --input-type=module -e "
import { parseEvidence } from './evidence.js';
console.log(parseEvidence('ℹ tests 31\nℹ pass 31\nℹ fail 0'));
console.log(parseEvidence('\x1b[34mℹ pass 31\x1b[39m\n\x1b[34mℹ fail 0\x1b[39m'));
console.log(parseEvidence('ℹ pass 0\nℹ fail 1\n# pass 24\n# fail 0'));
"

{ runner: 'node --test', passed: 31, failed: 0 }
{ runner: 'node --test', passed: 31, failed: 0 }
{ runner: 'node --test', passed: 24, failed: 0 }

The third line is the append-shape case reporting the later green run. Feeding literal strings is itself a rule here, not a convenience: a test that shells out to node --test and asserts on its output format will assert different things on Node 22 and Node 25, which is the same environment-dependent failure this whole fix removes. Pin the bytes you parse; run the runner separately.

Gotchas

The workaround dies quieter than the bug. On Node 22 the escape hatch was “pipe it and you get TAP”, and plenty of scripts leaned on that without writing it down. v23.0.0 removed the TTY split, so the workaround stopped working with no error anywhere; the output just changed shape. When I fixed this gate I deliberately did not write --test-reporter=tap into the stage’s instructions as the remedy, because that pins today’s workaround into a document that outlives it. The instruction says what must be in the file, a pass/fail summary and an exit code; the parser’s job is to read whatever the current runner calls a summary.

Your fallback’s spelling assumptions are format assumptions too. The exit-code fallback above requires the literal word between exit and the number, so a file ending EXIT: 0 misses it. That is the same class of bug as the reporter change, one layer down. Before widening a pattern, check what your tools actually emit; this gate’s real evidence files write exit code: 0, so the tight pattern stands, and a looser one would only add ways for prose to masquerade as a result.

A refusal that misdescribes the file costs more than the bug. Both round-trips in the original incident were spent re-capturing evidence that was already complete, because the error said no suite had run. The file had a summary; the parser had a blind spot; the message blamed the file. Deriving the format list from RUNNER_IDS in the refusal turned the next occurrence of this class into a one-shot fix, and it is three lines.

Sources

Changelog

  • fix(issueflow): parse node --test’s spec reporter in the accept gate (#217) (b7308f0)
  • feat(issueflow): write three contracts down that only lived in the orchestrator’s head (#220) (604a1d1)
  • issueflow: lessons from the #212/#215 runs — no terminal state, weak red-side contract, silent subagent completion, cwd-dependent CLI — the run’s terminal state — runState(), lane.landed/run.finished added… (#221) (dc13a26)