Measure your checker's precision on runs it never saw
Shipped
This release fixed five defects in a grading skill, all of which made its report assert something false. The measurement that found them is the transferable part: instead of arguing about whether the rules were right, the checker was scored against 47 real session transcripts. It came out at 85.7% precision overall and 14% at high severity, and its worst rule was wrong on all six of its real firings.
The 48-test suite was green the entire time. It was green because the frozen baseline is pinned to a session the tool had been developed against, and that session’s text happens to contain the exact phrases the rules recognise. If you maintain a linter, a policy checker, or an LLM-output grader, that is the failure mode to go looking for, and you can find it in an afternoon.
Why a green suite proves less than it looks
The trap is old and well documented in machine learning. Sebastian Raschka’s write-up on model evaluation and selection puts it directly: reusing a test set introduces bias and produces “overly optimistic estimates of the generalization performance,” because “the test set leaks information.” Every time you tune against it, you fit to it a little more.
A checker’s fixture is exactly that test set, and it leaks the same way. You write a rule, you run it against your sample, you adjust the pattern until the sample is clean. The sample is now a description of your rule rather than an independent test of it.
That matters because rule-based tools do not have a great baseline to begin with. A 2025 benchmark of static analysis tools against ten real C# projects containing 63 embedded vulnerabilities reported F-1 scores of 0.260, 0.386 and 0.546 for the three tools it evaluated. Research on reducing false positives in analytic bug detectors frames the same problem from the user’s side: these tools “tend to report a significant number of false positives, requiring developers to manually verify each warning,” which is precisely what stops people using them.
So assume your checker is worse than its fixture says, and go measure how much.
Build a checker worth grading
A rule-based checker over a log of events. Two rules, two severities, no dependencies.
// checker.mjs
export const RULES = [
{
id: "pr-into-main",
severity: "high",
describe: "a pull request must not target main directly",
// Deliberately naive: it reads the base and ignores the head.
match: (event) =>
event.kind === "command" &&
/gh pr create/.test(event.text) &&
/--base main/.test(event.text),
},
{
id: "secret-in-log",
severity: "medium",
describe: "never print a credential",
match: (event) =>
event.kind === "output" && /(?:token|api[_-]?key)=\S+/i.test(event.text),
},
];
export function check(events) {
const findings = [];
for (const event of events) {
for (const rule of RULES) {
if (rule.match(event)) {
findings.push({ ruleId: rule.id, severity: rule.severity, eventId: event.id });
}
}
}
return findings;
}
Collect real runs, and label them honestly
The input is runs the checker was never tuned on. One JSON object per line: the events, plus your verdict on each finding the checker produces.
Label after you look at the finding, not before. A finding is a true positive only if the thing it names actually happened and the rule genuinely forbids it in that context.
// make-corpus.mjs — stand-in for your real transcripts
import { writeFileSync } from "node:fs";
const runs = [
{
id: "run-1",
events: [
{ id: "e1", kind: "command", text: "gh pr create --base main --head dev" },
],
// The team's own process mandates promoting dev into main.
verdicts: { "pr-into-main:e1": false },
},
{
id: "run-2",
events: [
{ id: "e1", kind: "command", text: "gh pr create --base main --head feature/x" },
],
verdicts: { "pr-into-main:e1": true },
},
{
id: "run-3",
events: [
{ id: "e1", kind: "command", text: "gh pr create --base main --head dev" },
{ id: "e2", kind: "output", text: "GITHUB_TOKEN=ghp_realleakedvalue" },
],
verdicts: { "pr-into-main:e1": false, "secret-in-log:e2": true },
},
{
id: "run-4",
events: [
{ id: "e1", kind: "output", text: "api_key=sk-live-abc123" },
],
verdicts: { "secret-in-log:e1": true },
},
];
writeFileSync("corpus.jsonl", runs.map((r) => JSON.stringify(r)).join("\n") + "\n");
console.log(`wrote ${runs.length} runs`);
Score it, split by severity and by rule
An overall number is where a broken rule hides. Compute precision three ways in one pass.
// score.mjs
import { readFileSync } from "node:fs";
import { check, RULES } from "./checker.mjs";
const runs = readFileSync("corpus.jsonl", "utf8")
.trim().split("\n").map((line) => JSON.parse(line));
const tally = { all: { tp: 0, fp: 0 } };
const bySeverity = {};
const byRule = {};
for (const run of runs) {
for (const finding of check(run.events)) {
const key = `${finding.ruleId}:${finding.eventId}`;
if (!(key in run.verdicts)) {
throw new Error(`${run.id}: unlabelled finding ${key} — label it before scoring`);
}
const correct = run.verdicts[key];
const bucket = correct ? "tp" : "fp";
tally.all[bucket]++;
(bySeverity[finding.severity] ??= { tp: 0, fp: 0 })[bucket]++;
(byRule[finding.ruleId] ??= { tp: 0, fp: 0 })[bucket]++;
}
}
const precision = ({ tp, fp }) => (tp + fp === 0 ? null : tp / (tp + fp));
const pct = (v) => (v === null ? "n/a" : `${(v * 100).toFixed(1)}%`);
console.log(`overall precision: ${pct(precision(tally.all))} ` +
`(${tally.all.tp} true, ${tally.all.fp} false)`);
console.log("\nby severity:");
for (const [severity, counts] of Object.entries(bySeverity)) {
console.log(` ${severity.padEnd(8)} ${pct(precision(counts))} ` +
`(${counts.tp} true, ${counts.fp} false)`);
}
console.log("\nby rule:");
for (const rule of RULES) {
const counts = byRule[rule.id];
if (!counts) { console.log(` ${rule.id.padEnd(14)} never fired`); continue; }
console.log(` ${rule.id.padEnd(14)} ${pct(precision(counts))} ` +
`(${counts.tp} true, ${counts.fp} false)`);
}
Run it
node make-corpus.mjs && node score.mjs
wrote 4 runs
overall precision: 60.0% (3 true, 2 false)
by severity:
high 33.3% (1 true, 2 false)
medium 100.0% (2 true, 0 false)
by rule:
pr-into-main 33.3% (1 true, 2 false)
secret-in-log 100.0% (2 true, 0 false)
Sixty percent overall reads like a tool with a tuning problem. The split says something sharper.
The secret-in-log rule is doing its job. The pr-into-main rule is wrong two times out of three, and every one of those false positives is the sanctioned promotion. On the real corpus that same rule was wrong on six of six firings, all of them the dev into main promotion the repo’s own process document requires.
The fix is not a threshold. The rule read --base and never --head, so it could not distinguish a promotion from a feature branch jumping the queue. A rule that encodes half of a two-part condition is not strict, it is broken, and only real runs show you which half is missing.
The unlabelled-finding exception matters as much as the arithmetic. A scorer that silently skips a finding it has no verdict for will quietly report precision over the subset you happened to label, which is the same optimistic bias one level down.
Gotchas
A rule that fires only on your own fixture will look perfect forever. The 48-test suite stayed green through all five defects because the frozen baseline is a session the tool was developed against, so its text contains the exact phrases the rules match. The escape is to keep the fixture for regression and score precision on a corpus of runs collected after the rules were written. If a rule has never fired on anything except your sample, treat it as unmeasured rather than passing.
A lazy quantifier with a minimum length cannot close on a short span. The clause extractor pulled rules out of markdown with /\*\*([\s\S]{5,400}?)\*\*/g. A bold span shorter than that five-character minimum cannot terminate on its own closing **, so the opener runs on to the next one and the matcher alternates from there, capturing the gap text between rules and discarding the real spans until another short span flips it back. The symptom was a contract that omitted **Never print or commit secrets.** while including the sentence immediately after it at severity high. The escape is to make the inner a run that cannot contain the delimiter, then let a separate length check do the filtering it was always meant to do. Fixing it recovered ten real rules in one contract and shed nine pieces of gap text.
One violating event should produce one finding. The decision ran once per bound clause, so a single command emitted a finding for each clause attached to it, including one citing a rule about pushes for an event that was not a push. Duplicates inflate every count you are about to compute, so deduplicate at the event, and attach each hit to the most specific rule rather than to all of them.
Two copies of the same arithmetic will disagree. The coverage number was computed once for the written report and again for the terminal summary, and the two drifted, so the tool contradicted the file it had just written. Export one function and call it from both. This is the boring one, and it is the one that shipped twice.
Know which fix you are not making. The trace layer was left alone on purpose: its event ids are positional, so a change that alters the count rewires citations to the wrong events with zero rejections, and on compacted sessions that would have silently moved a large share of them. A measurement that improves one number while corrupting the link between a finding and its evidence is not an improvement. Write down what you deliberately did not touch, and why.
Sources
- Model evaluation, model selection, and algorithm selection in machine learning — why reusing a test set leaks information and yields optimistic estimates
- Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability Detection — F-1 scores of 0.260, 0.386 and 0.546 against ten real C# projects with 63 embedded vulnerabilities
- Learning to Reduce False Positives in Analytic Bug Detectors — the false-positive burden on developers, and a 17.5% precision improvement over a static-analysis baseline