Nate Swenson

Every citation resolved, and the work was not yours

Ship Report · No. 117 · 10 min read ·v0.1.0

Shipped

shipreport turns a window of real commits, pull requests, releases and coding sessions into a short summary someone who was not there can read. Its one rule is that every claim must carry a receipt: an identifier that resolves to a real artifact, checked by a gate that exits non-zero and is re-run at render rather than trusted to have passed earlier.

The first release also learned, by running a three-month window against my own account, that the rule has a blind spot it cannot close. A single documentation pull request I had sent to somebody else’s project pulled eleven of that project’s releases into the window, and the ranking put one of them seventh. Every receipt resolved. The releases were real, the identifiers were correct, and the gate had nothing to complain about, because none of those releases were fabricated. They just were not mine.

This guide is about the filter that catches that, and about the two places it has to run.

Two different questions

It is easy to collapse “can this claim be backed up” and “does this claim belong in this report” into one check, because both feel like validation. They are not the same question, and only the first one has a mechanical answer available from the artifact itself.

The provenance literature keeps them apart deliberately. W3C’s PROV data model defines provenance as “a record that describes the people, institutions, entities, and activities involved in producing, influencing, or delivering a piece of data or a thing”, and gives attribution its own relation: “Attribution is the ascribing of an entity to an agent.” An entity existing and an entity being ascribed to you are separate facts, recorded separately, and a system that stores only the first cannot answer the second no matter how strictly it checks.

In report terms: a receipt gate stops fabrication. It does not stop misattribution, and hardening it will never make it stop misattribution, because the misattributed item is genuinely there. That is why this needed a new filter rather than a stricter receipt.

Normalise the identifier at the edge

Before any of that, a smaller problem that corrupted the data on the way in.

// ownership.mjs
// Two different GitHub search endpoints spell the same value differently, so
// normalise once, at the edge, and refuse to invent a value when neither is
// present. An `undefined` slug flowing onward becomes "undefined/thing" in an
// id, which then silently matches nothing downstream.
export function repoSlug(repository) {
  const slug = repository?.nameWithOwner ?? repository?.fullName ?? null;
  if (!slug) throw new Error(`no repo slug on ${JSON.stringify(repository)}; check the endpoint's field name`);
  return slug;
}

export const ownerOf = (slug) => slug.split('/')[0];

gh search prs and gh search commits both return a repository object, and the GitHub CLI manual lists repository as an available JSON field without documenting its shape. The shapes differ. I checked both while writing this:

$ gh search prs --owner natejswenson --limit 1 --json repository
[{"repository":{"name":"claude-skills","nameWithOwner":"natejswenson/claude-skills"}}]

$ gh search commits --author natejswenson --limit 1 --json repository
[{"repository":{"description":"...","fullName":"natejswenson/SetScore","id":"R_kgDORvCEMA", ...}}]

Pull requests carry nameWithOwner. Commits carry fullName. Same value, two names, no error if you read the wrong one; you get undefined, which becomes a receipt id like commit:undefined@abc1234. In the first run that id then failed to match anything during the squash-merge fold, so every squash-merged pull request was counted twice, and nothing anywhere reported a problem.

Throwing on the unknown shape is the part worth copying. A ?? null that quietly returns null is the same bug one step later.

State the rule once

// ownership.mjs, continued
// Contributing to somebody else's project is your work: pull requests and
// commits are kept wherever they happened. Releasing their project is not, and
// a release is fetched for every repository you touched, so one drive-by pull
// request drags in that project's whole release history.
const OWNERSHIP_SCOPED = new Set(['release']);

export function isMine(item, owners) {
  if (!OWNERSHIP_SCOPED.has(item.kind)) return true;
  return owners.includes(ownerOf(item.repo));
}

The asymmetry is the interesting part, and it took a real window to see. A blanket “drop anything outside your own repos” would have been simpler and wrong: sending a fix to a project you do not own is work worth reporting, and often the most interesting work in the window. What is not yours is the act of releasing that project. So the filter is scoped by kind, not applied to everything, and the set of scoped kinds lives in one named constant so the next kind added has an obvious place to declare itself.

Partition rather than filter

// ownership.mjs, continued
// The dropped half is reportable output, not waste: a filter that removes
// things silently is indistinguishable from a bug.
export function partition(items, owners) {
  const kept = [];
  const dropped = [];
  for (const item of items) (isMine(item, owners) ? kept : dropped).push(item);

  const droppedByRepo = new Map();
  for (const item of dropped) droppedByRepo.set(item.repo, (droppedByRepo.get(item.repo) ?? 0) + 1);

  return { kept, dropped, droppedByRepo };
}

Returning both halves costs one array and buys the only evidence anyone will ever have that the filter is behaving. When the report is short because eleven things were excluded, the reader needs to see the eleven. A silent Array.prototype.filter here would have been a second, quieter version of the bug it was written to fix.

Derive every figure once

// ownership.mjs, continued
// Every figure the report can print, derived ONCE from the kept set.
export function figures(kept) {
  const count = (kind) => kept.filter((i) => i.kind === kind).length;
  return {
    released: count('release'),
    merged: count('pull'),
    commits: count('commit'),
    total: kept.length,
  };
}

This function exists because of the shape of the original bug. The filter went into ranking, ranking got it right, and the summary strip at the top of the sheet was computed separately at render time from the unfiltered list. The rendered sheet said 21 released directly above a ranked list of 19, and both numbers came from code that was individually correct.

Tip 15 of The Pragmatic Programmer is the rule being broken, and it says a piece of knowledge must have “a single, unambiguous, authoritative representation within a system”. The knowledge here is not the filter’s code, which was written once. It is the answer, which was derived twice. Deriving it twice is what makes a fix land in one place and not the other, and it is the usual shape of this failure: when a filter is wrong in one call site, look for the second call site before you look for anything else.

Run it

// demo.mjs
import { repoSlug, partition, figures } from './ownership.mjs';

const OWNERS = ['you'];

// A window: your own work, plus one docs pull request to somebody else's
// project, which is what pulled that project's releases into range.
const raw = [
  { kind: 'pull', repository: { nameWithOwner: 'you/api' }, title: 'add retry budget' },
  { kind: 'pull', repository: { nameWithOwner: 'bigproj/docs' }, title: 'fix a typo in the README' },
  { kind: 'commit', repository: { fullName: 'you/api' }, title: 'tighten the backoff test' },
  { kind: 'release', repository: { nameWithOwner: 'you/api' }, title: 'api v1.4.0' },
  { kind: 'release', repository: { nameWithOwner: 'you/api' }, title: 'api v1.4.1' },
  ...Array.from({ length: 11 }, (_, i) => ({
    kind: 'release',
    repository: { nameWithOwner: 'bigproj/docs' },
    title: `bigproj v3.${i}.0`,
  })),
];

const items = raw.map((r) => ({ kind: r.kind, title: r.title, repo: repoSlug(r.repository) }));
const { kept, dropped, droppedByRepo } = partition(items, OWNERS);

console.log(`owners: ${OWNERS.join(', ')}`);
console.log(`in window: ${items.length}   kept: ${kept.length}   dropped: ${dropped.length}\n`);

console.log('dropped, and why (a release of a project you do not own):');
for (const [repo, n] of droppedByRepo) console.log(`  ${repo.padEnd(16)} ${n} releases`);

console.log('\nkept:');
for (const i of kept) console.log(`  ${i.kind.padEnd(8)} ${i.repo.padEnd(16)} ${i.title}`);

const f = figures(kept);
console.log(`\nfigures (derived once, from kept): ${f.released} released, ${f.merged} merged, ${f.commits} commits`);

// The second call site, doing it again from the wrong list.
const naive = items.filter((i) => i.kind === 'release').length;
console.log(`\nstrip recomputed from the unfiltered list: ${naive} released`);
console.log(`same sheet, two numbers: ${naive} vs ${f.released}`);

try {
  repoSlug({ someOtherShape: true });
} catch (error) {
  console.log(`\nunknown shape -> ${error.message}`);
}

node demo.mjs prints:

owners: you
in window: 16   kept: 5   dropped: 11

dropped, and why (a release of a project you do not own):
  bigproj/docs     11 releases

kept:
  pull     you/api          add retry budget
  pull     bigproj/docs     fix a typo in the README
  commit   you/api          tighten the backoff test
  release  you/api          api v1.4.0
  release  you/api          api v1.4.1

figures (derived once, from kept): 2 released, 2 merged, 1 commits

strip recomputed from the unfiltered list: 13 released
same sheet, two numbers: 13 vs 2

unknown shape -> no repo slug on {"someOtherShape":true}; check the endpoint's field name

The typo pull request survives, which is the whole asymmetry working. Eleven releases go, and they go named. And the last two lines are the two failures reproduced deliberately, so you can watch them happen rather than take my word for it.

Gotchas

Redact on the way into the cache, not on the way out. Anything cached is read by every later run and by every model pass over that corpus, so a token pasted into a changelog body becomes a token written to disk. Doing it at ingest means the raw value exists only in memory for the length of one parse. In the first release this was only true rather than implemented: index promised redaction while merging fetched output into the corpus untouched, which was harmless exactly as long as nothing but a title was stored, and stopped being harmless when bodies were added. If you cache remote text, redact in the function that writes the cache, never in the function that reads it.

A first backfill and an incremental update are different programs wearing one name. The first pass here is a year of contributions and every session transcript on disk; the second reads a watermark and takes only what is newer, filtering by file modification time before anything is opened. On this machine that is 574 files, then one. Write the watermark before you need it, because retrofitting it means re-deriving what “already seen” meant.

Fetching per-repository in a loop is where the time goes. The dominant cost of the first-run backfill was awaiting one API call per repository in sequence. Making those concurrent is a small change with a large effect, and it is worth doing before you start optimising anything you actually wrote.

Do not let the model count. Any figure a summary states in prose is a figure that will eventually contradict a figure computed elsewhere on the same page. Compute them, print them where the writing step can see them, and have the gate refuse a hand-written count. That refusal landed a release later, after a run wrote “eleven components” above a computed strip reading sixteen.

Sources

Changelog

  • feat(shipreport): an executive summary of shipped work, where every claim carries a receipt (0.1.0) (#181) (20a3b91)