Nate Swenson

Twelve items, one score, and a line drawn by arrival time

Ship Report · No. 118 · 10 min read ·v0.2.0

Shipped

shipreport ranks a window of shipped work so a short report can carry only what mattered. 0.2.0 came out of replaying the first real run: 5 minutes 18 seconds end to end, a report whose one rule held cleanly, 13 claims and 15 receipts with nothing unresolved. The prose was fine. Underneath it, the ranking was not ranking.

Twelve items had scored exactly 70, every one of them release+50 minor+10 corroborated+10 and nothing else. The cut between the twelfth and the thirteenth was therefore drawn by a timestamp rather than by score. Every session in the window scored exactly 32, because the two signals that scored sessions were edits >= 20 and turns >= 60, and a twenty-edit session and a two-hundred-edit session are the same number to a threshold. Sessions were unordered, and none of them reached the report at all. The skill’s distinguishing claim was failing silently in its first run.

A signal everything clears carries no information

This is measurable rather than a matter of taste, and Shannon put the measurement in the paper that started the field. Listing the properties of his entropy H, he writes that “H = 0 if and only if all the pi but one are zero, this one having the value unity. Thus only when we are certain of the outcome does H vanish.” A signal that awards the same ten points to every item in the corpus has exactly one outcome. Its entropy is zero. It is not a weak signal, it is not a signal.

The mechanism is the one medicine has been arguing about for decades. Altman and Royston, writing on the cost of dichotomising a continuous variable, note that when you do it “much information is lost, so the statistical power to detect a relation between the variable and patient outcome is reduced”, and that “Individuals close to but on opposite sides of the cutpoint are characterised as being very different rather than very similar”. A ranking heuristic built out of >= comparisons is dichotomisation with extra steps, and it inherits both halves of that: items either side of the bar are treated as opposites, and every item well past the bar is treated as identical.

The reason it survives review is that each individual rule reads as sensible. edits >= 20 means “this was a substantial session,” which is true. It is only when you print the column that you see two hundred sessions holding one number.

Thresholds, and what to use instead

// score.mjs
// A threshold: one step, then nothing. Everything past the bar is the same
// number, however far past it went.
export const threshold = (value, at, points) => (value >= at ? points : 0);

// A tier: a step function that keeps climbing. `steps` is [bound, points] in
// ascending order; the highest bound the value clears wins.
export function tier(value, steps) {
  let points = 0;
  for (const [bound, awarded] of steps) if (value >= bound) points = awarded;
  return points;
}

tier is six lines and it is the whole fix. It is still a step function, still coarse, still readable in a table, and it does not stop rewarding a value for growing. You do not need a continuous function here, and a continuous one would be harder to explain in the output; you need one that has not run out of range.

// score.mjs, continued
const BUMP = { major: 20, minor: 10, patch: 4 };

export function scoreFlat(item) {
  return (
    50 +
    (BUMP[item.bump] ?? 0) +
    threshold(item.prs, 1, 10) // "corroborated": one pull request or a hundred
  );
}

export function scoreTiered(item) {
  return (
    50 +
    (BUMP[item.bump] ?? 0) +
    // how much in-window work stands behind this release
    tier(item.prs, [[1, 4], [3, 8], [6, 12], [10, 16]]) +
    // how much the changelog actually says
    tier(item.notesChars, [[200, 3], [800, 6], [2000, 9]]) +
    // a component's first release is news its fourth minor is not
    (item.first ? 12 : 0)
  );
}

Two decisions in scoreTiered are worth stealing along with the shape.

The prs tier tops out at 16, deliberately below a major bump’s 20. A pull request count measures how finely somebody split their work at least as much as how much work there was, so letting it outrank a major version would reward branching habits. When you add a signal, ask what else it correlates with, and cap it where that correlation would start to dominate.

There is also no duration signal, on purpose. It is the most obvious thing to reach for and it measures the wrong quantity: a session’s start and end timestamps bound the wall-clock span of a transcript, not the work inside it, and real sessions span thirty hours because somebody left a terminal open. A signal that is easy to compute and confidently wrong is worse than no signal.

Draw the line, and admit when it means nothing

// score.mjs, continued
// A stable sort means equal scores fall back to input order, so a tie at the
// boundary is decided by whatever order the fetch happened to return.
export function rank(items, score, limit) {
  const scored = items
    .map((item) => ({ ...item, score: score(item) }))
    .sort((a, b) => b.score - a.score);

  const above = scored.slice(0, limit);
  const below = scored.slice(limit);
  const distinct = new Set(scored.map((s) => s.score)).size;

  let tie = null;
  if (above.length && below.length && above.at(-1).score === below[0].score) {
    const at = above.at(-1).score;
    tie = { score: at, shared: scored.filter((s) => s.score === at).length };
  }

  return { ranked: above, distinct, tie };
}

The tie check compares only the last item above the line with the first below it, which is the only comparison that matters. Everything else can tie harmlessly; a tie at the boundary is the one that silently decides what gets published.

Sorting is not the culprit here, and it is worth knowing why. MDN records that “Since version 10 (or ECMAScript 2019), the specification dictates that Array.prototype.sort is stable”, so equal elements keep their original order. That is a guarantee, and it is exactly what makes the failure so quiet: the output is deterministic, reproducible, and completely arbitrary, because the original order is whatever the API returned. Nothing looks broken.

Reporting distinct alongside the ranking is the cheapest health check available. One distinct score across twelve items is not a close race, it is a dead instrument.

Run it

// demo.mjs
import { scoreFlat, scoreTiered, rank } from './score.mjs';

// Twelve minor releases from one week. They differ a lot: one has eleven pull
// requests behind it and a long changelog, one is a component's first release,
// several are near-empty version bumps.
const releases = [
  { name: 'api v1.4.0', bump: 'minor', prs: 11, notesChars: 2400, first: false },
  { name: 'api v1.5.0', bump: 'minor', prs: 7, notesChars: 1400, first: false },
  { name: 'cli v0.1.0', bump: 'minor', prs: 4, notesChars: 900, first: true },
  { name: 'cli v0.2.0', bump: 'minor', prs: 3, notesChars: 850, first: false },
  { name: 'web v2.1.0', bump: 'minor', prs: 2, notesChars: 300, first: false },
  { name: 'web v2.2.0', bump: 'minor', prs: 1, notesChars: 260, first: false },
  { name: 'jobs v0.4.0', bump: 'minor', prs: 1, notesChars: 210, first: false },
  { name: 'jobs v0.5.0', bump: 'minor', prs: 1, notesChars: 190, first: false },
  { name: 'auth v3.1.0', bump: 'minor', prs: 1, notesChars: 150, first: false },
  { name: 'auth v3.2.0', bump: 'minor', prs: 1, notesChars: 120, first: false },
  { name: 'edge v0.9.0', bump: 'minor', prs: 1, notesChars: 90, first: false },
  { name: 'edge v1.0.0', bump: 'minor', prs: 1, notesChars: 60, first: false },
];

const LIMIT = 6;

const show = (label, score) => {
  const { ranked, distinct, tie } = rank(releases, score, LIMIT);
  console.log(`\n${label}`);
  console.log(`  distinct scores across all ${releases.length}: ${distinct}`);
  for (const r of ranked) console.log(`    ${String(r.score).padStart(3)}  ${r.name}`);
  console.log(
    tie
      ? `  CUT: ${tie.shared} items share ${tie.score} at the line; nothing separates them`
      : '  CUT: drawn by score',
  );
};

show('thresholds (every signal tops out)', scoreFlat);
show('tiers (every signal keeps climbing)', scoreTiered);

node demo.mjs:

thresholds (every signal tops out)
  distinct scores across all 12: 1
     70  api v1.4.0
     70  api v1.5.0
     70  cli v0.1.0
     70  cli v0.2.0
     70  web v2.1.0
     70  web v2.2.0
  CUT: 12 items share 70 at the line; nothing separates them

tiers (every signal keeps climbing)
  distinct scores across all 12: 6
     86  cli v0.1.0
     85  api v1.4.0
     78  api v1.5.0
     74  cli v0.2.0
     67  web v2.1.0
     67  web v2.2.0
  CUT: 3 items share 67 at the line; nothing separates them

The top half is the bug, and the released version of it scored twelve real releases at exactly 70 the same way. The bottom half is the fix, and the honest part is its last line: three items still share the boundary score. Tiers reduced the ties, they did not abolish them, and the report says so instead of pretending the sixth item beat the seventh. On the real frozen week the same change took the ranking from 5 distinct scores to 10 and moved the cut out of the tie entirely.

Gotchas

A checker’s false positive teaches the writer to lie more smoothly. The receipts gate here refused correct prose: a \w+/\w+ pattern hunting for raw repository slugs matched the phrase plus/minus, and also CI/CD, read/write and 24/7, while [0-9a-f]{7,40} matched any seven-digit number along with the words effaced and defaced. The run’s response was to reword a true sentence until the gate went quiet, which is the exact inversion of fixing the draft rather than the checker. The escape is to make the pattern know something: a slug is a repository when the corpus already knows that owner, or when the name carries a hyphen, dot, underscore or digit; a hex run is a commit sha when it mixes letters and digits. A false positive costs more than a miss, because a miss is a gap and a false positive is a training signal.

A test can pass for the wrong reason and hide the case it was written for. A smuggled identifier, <<em>em>#412, collapses to em>#412 after tag stripping. The pull-request pattern required whitespace before the #, so it never fired; the case was caught instead by the repo-slug pattern falsely matching 412/em. The test was green, the defect was live, and fixing the false positive is what exposed it. Any non-alphanumeric boundary counts now. If a test passes, check which assertion caught it before you believe it.

Declare your boolean flags. show --brief release:a release:b read only the second receipt: the argument parser could not distinguish a boolean flag from one taking a value like --days 7, so the first receipt was consumed as --brief’s argument. Nothing reported a missing artifact because nothing knew one had been requested. The bug was latent for every boolean flag in the tool and only surfaced when one landed next to a positional argument.

Bound the total, not the item. A --chars cap applied per item multiplies: six receipts at 2400 characters is 14,000 characters into a conversation. Make the budget a total divided across whatever was asked for, so asking for more artifacts buys less of each and a single call is always safe to run bare. The release that added the rule “run these bare, the output is already bounded” is the same release that shipped the one command for which it was false.

Sources

Changelog

  • feat(shipreport): four defects a real run exposed, and the ranking that could not rank (0.2.0) (#188) (ff4f973)