Nate Swenson

Liveness from the filesystem, not a heartbeat

Issue Flow · No. 134 · 12 min read ·v0.5.0

Shipped

issueflow v0.5.0 is about one experience: you dispatch a stage of a gated pipeline to a background worker, and then you stare at a status board that says briefed for minutes at a time. This release adds a live elapsed clock for the stage you are waiting on, a position line at the moment of dispatch, an expected-duration range read from past runs of the same repo, and an optional per-stage progress log. None of it required the worker to phone home. The whole technique is reading liveness off the files the worker already writes, and that is what this guide builds.

The dead-air problem

Any orchestrator that hands work to a slow subprocess has this gap. The worker is an LLM agent in my case, but a test suite, a migration, or a render farm behaves the same way: the orchestrator marks the stage started, the worker goes quiet, and the user is left deciding between “still going” and “wedged” with no evidence either way. Jakob Nielsen’s response-time limits put the ceiling at about ten seconds before people need real feedback, and his guidance for anything longer is a progress indicator that shows the wait is moving. A multi-minute stage with a frozen status word fails that by two orders of magnitude.

The standard fix is a heartbeat: have the worker report in, the way the health check API pattern has a service expose /health for a monitor to poll. That works, but it puts the burden on the worker’s cooperation, and the pattern’s own drawbacks section admits the gap: a check is only as fresh as the last poll, and a worker can die between polls. For a subprocess you do not control, there is a cheaper signal. The worker’s job is to produce files. Files have modification times. The filesystem is already keeping the log you wanted the worker to send.

A stage’s clock is two timestamps

The state file records when each stage was handed to a worker and when its output was accepted. That pair is the whole data model this technique needs. Here is the core, in a file called liveness.mjs:

import { statSync } from 'node:fs';

// Render a millisecond span one way, everywhere: "42s", "4m07s".
export const formatSpan = (ms) => {
  const total = Math.round(ms / 1000);
  return total < 60 ? `${total}s` : `${Math.floor(total / 60)}m${String(total % 60).padStart(2, '0')}s`;
};

// A finished stage's duration: briefed to delivered.
export function durationOf(entry) {
  const { briefed, delivered } = entry.at ?? {};
  if (!briefed || !delivered) return null;
  const ms = Date.parse(delivered) - Date.parse(briefed);
  if (!Number.isFinite(ms) || ms < 0) return null;
  return formatSpan(ms);
}

// A running stage's elapsed time: briefed to now. The "+" marks a
// lower bound, "at least this long", never a finished duration.
export function elapsedOf(entry, now) {
  const { briefed } = entry.at ?? {};
  if (!briefed) return null;
  const ms = Date.parse(now) - Date.parse(briefed);
  if (!Number.isFinite(ms) || ms < 0) return null;
  return `${formatSpan(ms)}+`;
}

durationOf needs delivered to be recorded, and in the old code it was written in exactly one place: the accept step, after a human approved the output. Which means the one stage a user is actively waiting on could never show a duration. The stage had delivered its artifact to disk minutes ago, but the clock would not start answering until someone clicked approve.

The observer fills in what the disk already knows

The fix is a pure function, in the same liveness.mjs, that reads the artifact’s mtime and fills in delivered in memory, without ever writing it:

const mtimeOf = (path) => {
  try {
    return statSync(path).mtime.toISOString();
  } catch {
    return null;
  }
};

// Fill in at.delivered, in memory only, for any stage whose artifact
// has landed on disk but has not been approved yet. The accept step
// stays the only writer; it records the same mtime when the human
// approves, so the observed value and the persisted value agree.
export function observe(run, artifactPathOf) {
  const observed = { ...run, stages: run.stages.map((s) => ({ ...s, at: { ...s.at } })) };
  for (const stage of observed.stages) {
    if (stage.at.delivered) continue;
    const artifact = artifactPathOf(stage);
    if (!artifact) continue;
    const mtime = mtimeOf(artifact);
    if (mtime) stage.at.delivered = mtime;
  }
  return observed;
}

Node’s fs.Stats exposes mtime as a Date (with mtimeMs and mtimeNs if you want more precision than a filesystem may actually store), and that mtime is an honest record of when the worker finished writing. The property that makes this design hold is the one-writer rule: observe() never persists anything, and the accept step derives the permanent delivered from the same mtime. Whether a stage was observed mid-run or not, the recorded duration comes out identical. There is no second clock to disagree with the first.

Every render path calls observe() first, and the board passes now so a stage with no artifact yet shows 4m12s+ instead of a dash. A stage with an artifact awaiting review shows delivered and a real duration. Both of those used to render as nothing at all.

Set the expectation from the runs beside this one

An elapsed clock answers “is it moving”. The question underneath it is “is seven minutes normal”, and the honest answer lives in the durations of past runs. In this layout every run of a repo is a sibling directory, so history is one readdir away. This also goes in liveness.mjs:

import { existsSync, readdirSync, readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';

const median = (sorted) => {
  const mid = Math.floor(sorted.length / 2);
  return sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;
};

// Past stage durations from this repo's other runs: sibling dirs of
// runDir, each holding a run.json. A sibling that cannot be parsed is
// skipped, never thrown; a history scan degrading to "no history"
// must never crash the dispatch it decorates.
export function readTimings(runDir, { schema = 2 } = {}) {
  const parent = dirname(runDir);
  let names;
  try {
    names = existsSync(parent) ? readdirSync(parent) : [];
  } catch {
    return [];
  }
  const byStage = new Map();
  for (const name of names) {
    const dir = join(parent, name);
    if (dir === runDir) continue;
    const statePath = join(dir, 'run.json');
    if (!existsSync(statePath)) continue;
    let run;
    try {
      run = JSON.parse(readFileSync(statePath, 'utf8'));
    } catch {
      continue;
    }
    if (run?.schema !== schema) continue;
    for (const entry of run.stages ?? []) {
      const { briefed, delivered } = entry.at ?? {};
      if (!briefed || !delivered) continue;
      const ms = Date.parse(delivered) - Date.parse(briefed);
      if (!Number.isFinite(ms) || ms < 0) continue;
      if (!byStage.has(entry.id)) byStage.set(entry.id, []);
      byStage.get(entry.id).push(ms);
    }
  }
  return [...byStage.entries()].map(([stage, values]) => {
    const sorted = [...values].sort((a, b) => a - b);
    return {
      stage,
      n: sorted.length,
      min: formatSpan(sorted[0]),
      median: formatSpan(median(sorted)),
      max: formatSpan(sorted[sorted.length - 1]),
    };
  });
}

Two decisions in there matter more than the code. First, the summary is a spread, min, median, max, not a mean. Google’s SRE book makes the case that averages hide the tail: “If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds.” Stage durations are exactly that kind of skewed distribution, and a range with a median tells the user something a mean would lie about. Second, there is no pooling across repos. Stage duration is dominated by codebase size and test-suite runtime, which are properties of the repo; a number pooled from someone else’s repo is confident and wrong. Below two samples, the dispatch line says “no past timings on this repo”, and that honesty is part of the feature.

What the user sees

At dispatch, the brief now prints position and expectation before the wait begins. With a few past runs on the repo, the two lines are shaped like this:

Step 2 of 4 · 1 approved · investigate → [design] → implement → test
design on this repo: 3 past runs, 2m34s–6m01s (median 3m40s). It unblocks implement.

While a stage runs, the status board shows a table with Since (always populated, from the same clock the durations use) and the last line of an optional per-stage progress log the worker may append to, with its age. A worker that never writes the log still shows a real Since and a dash for progress. That degradation is the contract: the mechanical clock is the primary signal, and the log is enrichment on top of it. A worker that ignores the instructions must look quiet, never dead.

To check your own implementation, build a fake run and confirm the two rendering paths disagree the right way: a stage with an artifact on disk renders a finished duration through observe(), and one without renders elapsedOf() with the trailing +:

// check.mjs
import { mkdirSync, writeFileSync } from 'node:fs';
import { durationOf, elapsedOf, observe, readTimings } from './liveness.mjs';

mkdirSync('runs/repo/issue-1', { recursive: true });
writeFileSync('runs/repo/issue-1/design.md', '# the delivered artifact\n');
// A finished sibling gives readTimings one 3-minute design sample.
mkdirSync('runs/repo/issue-0', { recursive: true });
writeFileSync('runs/repo/issue-0/run.json', JSON.stringify({
  schema: 2,
  stages: [{ id: 'design', at: { briefed: '2026-08-13T10:00:00Z', delivered: '2026-08-13T10:03:00Z' } }],
}));

const tenMinAgo = new Date(Date.now() - 10 * 60 * 1000).toISOString();
const run = {
  schema: 2,
  stages: [
    { id: 'design', at: { briefed: tenMinAgo } },      // artifact on disk
    { id: 'implement', at: { briefed: tenMinAgo } },   // nothing yet
  ],
};
const observed = observe(run, (s) => (s.id === 'design' ? 'runs/repo/issue-1/design.md' : null));
const now = new Date().toISOString();
for (const stage of observed.stages) {
  console.log(stage.id, durationOf(stage) ?? elapsedOf(stage, now));
}
console.log(JSON.stringify(readTimings('runs/repo/issue-1')));

Running it just after creating the artifact shows the delivered stage with a real (tiny) duration, the quiet stage as a lower bound, and the sibling’s history:

design 10m00s
implement 10m00s+
[{"stage":"design","n":1,"min":"3m00s","median":"3m00s","max":"3m00s"}]

The first duration reads ten minutes because the fake briefed was backdated and the artifact was written now; the point is the shape, a plain span for delivered work and a + for work still running. If both stages render the same, your observer is not consulting the filesystem.

Gotchas

  • The assertion that passed on both sides of the revert. The first test for the unreadable-run fix asserted that the run’s directory name appeared in the command’s output. It did, before and after the fix, because a different column already contained the path it was derived from. The test proved nothing; reverting the fix kept it green. The escape is to parse the rendered row and assert on the specific cell the fix changed. After that change, reverting the fix alone turns the test red with the old placeholder string as the actual value. If a test guards a fix, run it against the reverted code once and watch it fail.
  • Two writers for one timestamp will eventually disagree. The tempting version of observe() persists the mtime it found, so the state file is “more complete”. Now approval and observation race to write delivered, and the value depends on which path ran first. Keep the observer pure and derive the persisted value from the same mtime at accept time; the duration is then identical no matter who looked first.
  • One corrupt sibling can take down every dispatch. The history scan walks directories this code does not own, and the first truncated run.json or older-schema run it hit would have thrown mid-brief. Every parse in the scan swallows its failure and skips the sibling, and a test pins the case where a legacy-schema run sits beside a good one. The floor test matters just as much: a scan that silently matches nothing renders “no history” forever and looks correct while measuring nothing.
  • Byte-pinned goldens decide where you can add output. Two commands’ outputs are frozen byte-for-byte by baseline tests, so the live clock could not touch them. The board function takes now as an explicit option, and rendering without it is exactly the old output. Time as a parameter instead of a global is what let the feature land without invalidating the goldens, and it is also what makes every timing test deterministic.

Sources

Changelog

  • issueflow: a watched run is minutes of dead air — no progress signal during a dispatched stage, timings only shown after ship, no position at dispatch, and runs renders an old run as ‘(unreadable run)’ — cmdRuns binds its caught RunError and reports the reason and remedy per… (0b49cf0)
  • issueflow: a watched run is minutes of dead air — no progress signal during a dispatched stage, timings only shown after ship, no position at dispatch, and runs renders an old run as ‘(unreadable run)’ — observe() + elapsedOf() in run.mjs, board(run, { now }) with the +… (47d9f7b)
  • issueflow: a watched run is minutes of dead air — no progress signal during a dispatched stage, timings only shown after ship, no position at dispatch, and runs renders an old run as ‘(unreadable run)’ — new timings.mjs reading the run’s sibling directories, positionLine() in… (7c0a9c3)
  • issueflow: a watched run is minutes of dead air — no progress signal during a dispatched stage, timings only shown after ship, no position at dispatch, and runs renders an old run as ‘(unreadable run)’ — progressPath(), the ## While you work section in renderBrief, the… (fa51818)