Nate Swenson

Teach your index to say it does not know

Skill Help · No. 121 · 10 min read ·v0.1.0

Shipped

skillhelp is a knowledge base for the skills in this repo. build extracts one card per skill from the skills’ own files, five fixed sections, every fact carrying the file:line it was read from. check re-extracts and byte-compares against the committed cards. ask returns matching fact lines inline with their sources.

The part that took the longest to get right was the case where there is no answer. Ask a documentation index something it does not cover and the natural behaviour is to return the closest few things, ranked, with total confidence. Everything about the machinery encourages it: there is always a best match, and a list of results looks the same whether the results are good or not.

This guide builds the small version: extraction that grounds facts as it makes them, a retrieval floor, and a refusal that says what it looked through.

Not answering is a feature that has to be built

The problem is old enough to have a benchmark named after it. Know What You Don’t Know: Unanswerable Questions for SQuAD was written because reading-comprehension systems needed to “not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering”, and the paper reports a model scoring 86% F1 on the answerable set dropping to 66% once unanswerable questions are mixed in. Knowing when to stop is a separate, harder skill from answering.

Anthropic’s own guidance on reducing hallucinations puts the same idea in operational terms, listing as its first basic strategy “Allow Claude to say “I don’t know”: Explicitly give Claude permission to admit uncertainty”, and, for citations, that if the model “can’t find a quote, it must retract the claim”.

In a deterministic index you get to enforce both mechanically rather than ask for them. That is the whole reason to build one.

Ground the fact when you make it

// extract.mjs
// The five sections every card carries. A card always has all five, because a
// missing section and an empty one are different facts about a skill.
export const SECTIONS = ['setup', 'commands', 'architecture', 'refusals', 'troubleshooting'];

// Pull facts out of a markdown file, carrying the line each one came from.
//
// Grounding happens HERE, at extraction, not as a check afterwards. A fact that
// cannot say where it came from is never constructed in the first place, so
// there is no later stage at which one can be invented.
export function extract(file, text, section) {
  return text
    .split('\n')
    .map((line, i) => ({ line: line.trim(), no: i + 1 }))
    .filter(({ line }) => line.startsWith('- '))
    .map(({ line, no }) => ({
      section,
      text: line.slice(2),
      source: `${file}:${no}`,
    }));
}

// Build one card. Sections with no source stay empty and are REPORTED, never
// filled in with something plausible.
export function card(skill, sources) {
  const facts = [];
  const empty = [];

  for (const section of SECTIONS) {
    const found = (sources[section] ?? []).flatMap(({ file, text }) => extract(file, text, section));
    if (found.length === 0) empty.push(section);
    facts.push(...found);
  }
  return { skill, facts, empty };
}

Attaching the source at construction is a structural choice rather than a stylistic one. If a fact object cannot exist without a source, then no downstream stage can produce an ungrounded one, and you never need a validator that goes looking for claims without citations. The type makes the bad state unrepresentable, which is cheaper than detecting it.

Reporting empty sections rather than omitting them matters for the same reason. “This skill documents no troubleshooting” is a real, useful answer. Dropping the section leaves the reader unable to distinguish it from “I did not look”.

The floor, and what must not cross it

// retrieve.mjs
const STOP = new Set([
  'how', 'do', 'i', 'the', 'a', 'in', 'is', 'what', 'to', 'my', 'for', 'of',
  'and', 'or', 'it', 'this', 'that', 'with', 'on', 'be', 'are', 'does',
]);

// Terms are matched as SUBSTRINGS below, so that "set" finds "setup" and
// "refuse" finds "refuses". That convenience is also why short words have to
// go: "and" is a substring of "command", and left in, it scores a relevance
// point against every fact that mentions one.
const MIN_TERM = 3;

export const terms = (q) =>
  q.toLowerCase().split(/[^a-z0-9]+/).filter((t) => t.length >= MIN_TERM && !STOP.has(t));

// How much of the question this fact actually answers: the count of query terms
// it contains. Nothing else feeds this number.
export function baseScore(query, fact) {
  const hay = fact.text.toLowerCase();
  return terms(query).filter((t) => hay.includes(t)).length;
}

// Preferences that decide ORDER among facts that already qualified. A section
// match is a hint about relevance, not evidence of it.
export function bonus(query, fact) {
  const q = query.toLowerCase();
  let b = 0;
  if (q.includes('install') || q.includes('set up')) b += fact.section === 'setup' ? 2 : 0;
  if (q.includes('command') || q.includes('flag')) b += fact.section === 'commands' ? 2 : 0;
  if (q.includes('fail') || q.includes('error')) b += fact.section === 'troubleshooting' ? 2 : 0;
  return b;
}

Splitting the score into a base and a bonus, and being strict about which one the floor reads, is the entire trick.

// retrieve.mjs, continued
// The floor is tested against the BASE score alone. A bonus may reorder what
// already cleared the bar; it must never lift something over it.
export function ask(query, cards, { floor = 2 } = {}) {
  const all = cards.flatMap((c) => c.facts.map((f) => ({ ...f, skill: c.skill })));
  const qualified = all
    .map((f) => ({ fact: f, base: baseScore(query, f), extra: bonus(query, f) }))
    .filter((s) => s.base >= floor);

  if (qualified.length === 0) {
    return {
      answered: false,
      searched: { cards: cards.map((c) => c.skill), facts: all.length, terms: terms(query), floor },
    };
  }

  return {
    answered: true,
    hits: qualified.sort((a, b) => b.base + b.extra - (a.base + a.extra)).slice(0, 5),
  };
}

// The version with the floor applied after the bonus. Same data, same query.
export function askUnfloored(query, cards, { floor = 2 } = {}) {
  const all = cards.flatMap((c) => c.facts.map((f) => ({ ...f, skill: c.skill })));
  const qualified = all
    .map((f) => ({ fact: f, base: baseScore(query, f), extra: bonus(query, f) }))
    .filter((s) => s.base + s.extra >= floor);

  if (qualified.length === 0) return { answered: false, searched: null };
  return { answered: true, hits: qualified.sort((a, b) => b.base + b.extra - (a.base + a.extra)).slice(0, 5) };
}

askUnfloored is what you write if you are not thinking about it, and it is one character of difference. The bonus was designed to express a preference among candidates; adding it before the comparison silently promotes it to evidence, and a fact that shares no words at all with the question can clear the bar on a section hint alone.

The refusal carries what was searched: how many facts, across which cards, for which terms, at what floor. A bare “no results” leaves the reader unable to tell a gap in the index from a badly worded question, and those need opposite responses.

Run it

// demo.mjs
import { card } from './extract.mjs';
import { ask, askUnfloored } from './retrieve.mjs';

const gmailtriage = card('gmailtriage', {
  setup: [
    {
      file: 'skills/gmailtriage/SKILL.md',
      text: [
        '- Run `gmailtriage setup` first; it is the only command safe to run cold.',
        '- Connect the Gmail account before writing any rule.',
      ].join('\n'),
    },
  ],
  commands: [
    {
      file: 'skills/gmailtriage/SKILL.md',
      text: [
        '- `plan` enumerates exactly which threads each rule takes and writes nothing.',
        '- `apply` refuses any thread the plan did not name and exits non-zero.',
        '- `labels` reconciles every destination against the real label list.',
      ].join('\n'),
    },
  ],
  architecture: [
    { file: 'skills/gmailtriage/references/gmail.md', text: '- A move is an added label and a removed INBOX label.' },
  ],
  refusals: [
    {
      file: 'skills/gmailtriage/SKILL.md',
      text: [
        '- A label rule naming TRASH or SPAM is refused.',
        '- A trash rule constrained only by age is refused.',
      ].join('\n'),
    },
  ],
  troubleshooting: [],
});

const cards = [gmailtriage];

const render = (label, result) => {
  console.log(`\n${label}`);
  if (result.answered) {
    for (const h of result.hits) console.log(`  [${h.base}+${h.extra}] ${h.fact.text}\n        ${h.fact.source}`);
    return;
  }
  if (!result.searched) return console.log('  (nothing)');
  const s = result.searched;
  console.log('  NOT DOCUMENTED');
  console.log(`    searched ${s.facts} facts across: ${s.cards.join(', ')}`);
  console.log(`    for terms: ${s.terms.join(', ')}   floor: ${s.floor}`);
};

console.log(`empty sections on the gmailtriage card: ${gmailtriage.empty.join(', ') || 'none'}`);

render('ask: "how do I set up gmailtriage"', ask('how do I set up gmailtriage', cards));
render('ask: "what does apply refuse"', ask('what does apply refuse', cards));

const undocumented = 'what is the retry limit and timeout for this command';
render(`ask: "${undocumented}"`, ask(undocumented, cards));
render('same question, same data, floor applied AFTER the bonus', askUnfloored(undocumented, cards));

node demo.mjs:

empty sections on the gmailtriage card: troubleshooting

ask: "how do I set up gmailtriage"
  [2+2] Run `gmailtriage setup` first; it is the only command safe to run cold.
        skills/gmailtriage/SKILL.md:1

ask: "what does apply refuse"
  [2+0] `apply` refuses any thread the plan did not name and exits non-zero.
        skills/gmailtriage/SKILL.md:2

ask: "what is the retry limit and timeout for this command"
  NOT DOCUMENTED
    searched 8 facts across: gmailtriage
    for terms: retry, limit, timeout, command   floor: 2

same question, same data, floor applied AFTER the bonus
  [0+2] `plan` enumerates exactly which threads each rule takes and writes nothing.
        skills/gmailtriage/SKILL.md:1
  [0+2] `apply` refuses any thread the plan did not name and exits non-zero.
        skills/gmailtriage/SKILL.md:2
  [0+2] `labels` reconciles every destination against the real label list.
        skills/gmailtriage/SKILL.md:3

Look at the base scores in the last block. Every one of them is zero: three facts sharing not a single term with the question, returned in rank order, each with a real file:line under it. Nothing here is fabricated, which is what makes it dangerous. Every citation resolves, every line is real, and the answer is still worthless, because relevance was never established and the sources make it look as though it had been.

The floor is what turns that into the block above it.

Gotchas

A gate on a derived artifact must watch the artifact’s inputs. The natural way to wire CI for a tool is to filter its job on its own directory. For an index describing other directories, that is exactly backwards: GitHub’s docs note that with push and pull_request you can “configure a workflow to run based on what file paths are changed”, and filtering on the index’s own path means the staleness check short-circuits to green on precisely the pull request that made it stale. The filter here matches the whole skills tree instead. Any generated artifact has this shape: point the trigger at what it is generated from.

Measure drift on the rendered output, not the source tree. A check that reddens whenever an input file changed will fire on typo fixes, reordered paragraphs and reflowed lines, none of which change a single answer. People then start ignoring it, and shortly after that they delete it. Re-extract and compare the card: if the answers are byte-identical the sources moved but the knowledge did not, and nothing should go red.

A committed index made from markdown is a place secrets go to become permanent. Markdown here is indexed verbatim, so anything pasted into a skill’s docs lands in a card, and cards are committed to a public repository. Source files are treated differently on purpose, mined for identifiers rather than copied. If you build any index that stores text verbatim, run a secret check on the way in, not on the way out, and remember the index outlives the paste: deleting the original line afterwards leaves the copy.

Pin the golden’s input, or the golden becomes a toll booth. A byte-compared build over the live tree would redden on every edit to any of the skills it describes, which means every unrelated pull request in the repo has to regenerate it. The fixture is a frozen snapshot instead, so the golden moves only when the extractor moves, and live coverage is the drift check’s job. When a golden test starts failing for reasons that have nothing to do with the code it tests, the input is too wide.

Sources

Changelog

  • feat(skillhelp): a knowledge base for every skill here, that refuses to guess (0.1.0) (#195) (f237d97)