Nate Swenson

The plan said yes, but it never said to what

Gmail Triage · No. 120 · 11 min read ·v0.2.0

Shipped

0.2.0 added the other half of triage: a label rule that files an item into one of your own folders and takes it out of the inbox, rather than only moving junk to the bin.

label had been a legal action since the first release and it was a dead end. The authoriser admitted trash entries and nothing else, so a sort rule could be written, validated, planned, shown to you, and then quietly do nothing. Fixing that meant deciding what an approval actually authorises, and the honest answer is not “this item”. It is “this operation, on this item, with this destination”. Same id, same plan, and the difference between the two readings is the difference between filing something and destroying it.

The check has to name the verb

The old authoriser answered “was this item in the plan”. That question has an obvious appeal, because it is the one a set of ids can answer in constant time, and it is not the question that keeps anything safe.

Saltzer and Schroeder’s complete mediation principle is stated as “Every access to every object must be checked for authority”, which they call the primary underpinning of a protection system. The word doing the work is access, not object. Reading and deleting are two accesses. Checking that an object appeared somewhere in an approved list mediates neither of them.

// authorise.mjs
export class AuthError extends Error {}

// What the plan authorised, keyed by the ACTION and its destination as well as
// the object. Same id, different verb, different key.
const key = (action, destination, id) => `${action}:${destination ?? '-'}:${id}`;

export const authorise = (plan) => new Set(plan.map((a) => key(a.action, a.destination, a.id)));

export function assertAuthorised(authorised, action, destination, id) {
  const k = key(action, destination, id);
  if (!authorised.has(k)) {
    throw new AuthError(`refusing ${k}: the plan authorised no such action on this item`);
  }
  return k;
}

// The version this replaced: a set of ids and nothing else. It answers "was
// this item in the plan", which is not the question anyone needed answered.
export const authoriseByIdOnly = (plan) => new Set(plan.map((a) => a.id));

export function assertByIdOnly(authorised, action, destination, id) {
  if (!authorised.has(id)) throw new AuthError(`refusing ${id}: not in the plan`);
  return key(action, destination, id);
}

Including the destination in the key, not just the action, is the part worth arguing about. It means re-approving a plan after changing where a rule files to, which is mild friction. It also means an approval to file into Receipts is not an approval to file into Archive, and once you accept that filing and trashing are different authorities it is hard to defend treating two destinations as one.

A destination that is not a destination

// rules.mjs
export class RuleError extends Error {}

// Reserved destinations. Filing INTO one of these is not filing, it is one of
// the operations the rest of the tool exists to guard, reached through a code
// path that guards nothing.
const RESERVED = new Set(['TRASH', 'SPAM', 'INBOX', 'SENT', 'DRAFT', 'UNREAD', 'STARRED', 'IMPORTANT']);

const KNOWN_KEYS = new Set(['id', 'action', 'note', 'match', 'destination', 'keepInInbox']);

export function validateSortRule(rule) {
  // An unknown key is not an ignorable extra. `keepInbox` reads to a human as
  // "keep it in the inbox" and, being unread, does the exact opposite.
  for (const k of Object.keys(rule)) {
    if (!KNOWN_KEYS.has(k)) {
      const near = [...KNOWN_KEYS].find((known) => known.toLowerCase().startsWith(k.toLowerCase().slice(0, 6)));
      throw new RuleError(`rule "${rule.id}" has unknown key "${k}"${near ? `; did you mean "${near}"?` : ''}`);
    }
  }

  if (rule.action !== 'label') return rule;

  const dest = String(rule.destination ?? '');
  if (!dest) throw new RuleError(`rule "${rule.id}" files into nothing`);

  if (RESERVED.has(dest.toUpperCase()) || dest.toUpperCase().startsWith('CATEGORY_')) {
    throw new RuleError(
      `rule "${rule.id}" files into the reserved label "${dest}"; that is not sorting, and no trash guard would see it`,
    );
  }
  return rule;
}

This is the one that genuinely alarmed me. Gmail’s guide states that “Labels come in two varieties: reserved SYSTEM labels and custom USER labels” and that system names are reserved, TRASH among them. A sort rule filing into TRASH is therefore a destructive operation wearing the costume of a filing operation, and it reaches that outcome through the one code path in the tool that no trash guard inspects, because none of them were written to look at a sort rule. Every safety check I had was on a different branch.

Whenever you add a second action to a system whose guards were written for the first, the question to ask is not “does the new action need guards too”. It is “can the new action reach the old action’s outcome”, and reserved names are the usual way it can.

The unknown-key refusal is the same instinct at a smaller scale. keepInbox where the schema says keepInInbox reads correctly to a human, gets silently dropped by the parser, and produces the opposite behaviour. Suggesting the near-match in the error costs one line and turns a confusing refusal into an obvious one.

Resolve names to ids before anything moves

// labels.mjs
export class LabelError extends Error {}

// Resolve every destination a rule set names against the mailbox's real labels,
// BEFORE anything moves. Without this the run fails partway, having already
// moved some items, with a receipt describing a mailbox that no longer exists.
export function reconcile(destinations, mailboxLabels) {
  const byName = new Map(mailboxLabels.map((l) => [l.name, l.id]));
  const resolved = [];
  const missing = [];

  for (const name of destinations) {
    const id = byName.get(name);
    if (id) resolved.push({ name, id });
    else missing.push(name);
  }
  return { resolved, missing };
}

export function assertResolvable(destinations, mailboxLabels) {
  const { resolved, missing } = reconcile(destinations, mailboxLabels);
  if (missing.length) {
    throw new LabelError(`create these labels first: ${missing.join(', ')}`);
  }
  return resolved;
}

// A move is two operations, because the mailbox has labels rather than folders.
// Filing without removing INBOX leaves the item exactly where it was.
export function toModify(destinationId, { keepInInbox = false } = {}) {
  return {
    addLabelIds: [destinationId],
    removeLabelIds: keepInInbox ? [] : ['INBOX'],
  };
}

// A sort rule must exclude its own destination, or it re-selects everything it
// already filed on every run and volume stops meaning anything.
export const compileQuery = (match, destination) => `${match} -label:${destination}`;

Two API facts shape this file. The modify call takes “A list of IDs of labels to add to this thread” and a matching list to remove, so destinations written as names in a config have to be resolved to ids at some point; doing it up front, for all of them, is what turns “fails on item 27 of 50 with 26 already moved” into “refuses before item 1 and tells you which labels to create”.

The second is that a move is genuinely two operations. There are no folders, so filing is adding a label, and moving is adding one and removing INBOX. A version that only did the first would report every item filed and leave the inbox exactly as full as it was.

Run it

// demo.mjs
import { authorise, assertAuthorised, authoriseByIdOnly, assertByIdOnly, AuthError } from './authorise.mjs';
import { validateSortRule, RuleError } from './rules.mjs';
import { assertResolvable, toModify, compileQuery, LabelError } from './labels.mjs';

// One plan: file t1 into Receipts, file t2 into Receipts. Nothing is trashed.
const plan = [
  { action: 'label', destination: 'Receipts', id: 't1' },
  { action: 'label', destination: 'Receipts', id: 't2' },
];

const attempt = (label, fn) => {
  try {
    console.log(`  ALLOWED  ${label} -> ${fn()}`);
  } catch (error) {
    if (!(error instanceof AuthError || error instanceof RuleError || error instanceof LabelError)) throw error;
    console.log(`  REFUSED  ${label}\n           ${error.message}`);
  }
};

console.log('=== same plan, same thread id, two different verbs ===');
const scoped = authorise(plan);
attempt('file t1 into Receipts', () => assertAuthorised(scoped, 'label', 'Receipts', 't1'));
attempt('trash t1', () => assertAuthorised(scoped, 'trash', null, 't1'));
attempt('file t1 into Archive', () => assertAuthorised(scoped, 'label', 'Archive', 't1'));

console.log('\n=== the same three, checked by id alone ===');
const idOnly = authoriseByIdOnly(plan);
attempt('file t1 into Receipts', () => assertByIdOnly(idOnly, 'label', 'Receipts', 't1'));
attempt('trash t1', () => assertByIdOnly(idOnly, 'trash', null, 't1'));
attempt('file t1 into Archive', () => assertByIdOnly(idOnly, 'label', 'Archive', 't1'));

console.log('\n=== rules refused before they can run ===');
for (const rule of [
  { id: 'r1', action: 'label', note: 'bin it', match: 'from:ads', destination: 'TRASH' },
  { id: 'r2', action: 'label', note: 'tag only', match: 'from:bank', destination: 'Receipts', keepInbox: true },
]) {
  attempt(`rule ${rule.id}`, () => {
    validateSortRule(rule);
    return 'accepted';
  });
}

console.log('\n=== destinations resolve before anything moves ===');
const mailbox = [
  { name: 'Receipts', id: 'Label_19' },
  { name: 'INBOX', id: 'INBOX' },
];
attempt('resolve [Receipts]', () => JSON.stringify(assertResolvable(['Receipts'], mailbox)));
attempt('resolve [Receipts, Travel]', () => JSON.stringify(assertResolvable(['Receipts', 'Travel'], mailbox)));

console.log('\n=== what a move actually is ===');
console.log(`  move:        ${JSON.stringify(toModify('Label_19'))}`);
console.log(`  tag in place:${JSON.stringify(toModify('Label_19', { keepInInbox: true }))}`);
console.log(`  query:       ${compileQuery('from:bank', 'Receipts')}`);

node demo.mjs:

=== same plan, same thread id, two different verbs ===
  ALLOWED  file t1 into Receipts -> label:Receipts:t1
  REFUSED  trash t1
           refusing trash:-:t1: the plan authorised no such action on this item
  REFUSED  file t1 into Archive
           refusing label:Archive:t1: the plan authorised no such action on this item

=== the same three, checked by id alone ===
  ALLOWED  file t1 into Receipts -> label:Receipts:t1
  ALLOWED  trash t1 -> trash:-:t1
  ALLOWED  file t1 into Archive -> label:Archive:t1

=== rules refused before they can run ===
  REFUSED  rule r1
           rule "r1" files into the reserved label "TRASH"; that is not sorting, and no trash guard would see it
  REFUSED  rule r2
           rule "r2" has unknown key "keepInbox"; did you mean "keepInInbox"?

=== destinations resolve before anything moves ===
  ALLOWED  resolve [Receipts] -> [{"name":"Receipts","id":"Label_19"}]
  REFUSED  resolve [Receipts, Travel]
           create these labels first: Travel

=== what a move actually is ===
  move:        {"addLabelIds":["Label_19"],"removeLabelIds":["INBOX"]}
  tag in place:{"addLabelIds":["Label_19"],"removeLabelIds":[]}
  query:       from:bank -label:Receipts

The two blocks at the top are the same plan and the same three requests, checked two ways. The scoped version refuses two of the three. The id-only version allows all three, including trashing an item whose only approval was to be filed, and it allows it without any error, any log line, or anything else to look at afterwards.

Gotchas

Do not assume a list endpoint returns the field you are filtering on. Filtering system labels by a type property looked correct and shipped, and list_labels does not return type at all. Every comparison was therefore against undefined, so INBOX, TRASH, SENT and SPAM were counted and matched as if they were folders somebody had made. This survived unit tests because the fixtures had been written from the same wrong assumption as the code, and it was caught the first time the thing ran against a real mailbox. Print one real response before you write the filter.

A receipt has to record the verb, not only the object. Undo needs to reverse each action with the call that actually reverses it, and “item t1 was in a run” does not tell you whether to restore it from trash or to move it back to the inbox. When you add this to a tool that already has receipts, the old ones carry no action field, so decide explicitly what a missing value means and write it down; here a receipt with no action is read as trash, because that is the only thing the earlier version could do.

A sort rule must exclude its own destination. Without -label:<destination> in the compiled query, every run re-selects everything the rule filed on every previous run. Nothing breaks, which is the problem: the rule reports steadily growing volume, and the one signal worth watching, a rule suddenly taking ten times its usual number of items, stops meaning anything at all.

Check that the maintenance command your docs promise exists. The invariants file had named a one-command refresh for every frozen fixture since the first release, and that file had never been written. It is the kind of thing nobody notices, because the command is only reached on the day someone needs to regenerate a fixture, and on that day they conclude the tooling is broken rather than absent. Run the commands in your own documentation once, from a clean checkout.

Sources

Changelog

  • feat(gmailtriage): sorting — file mail into your own folders, not just the bin (0.2.0) (#190) (48a4ce6)