The lint rule that flags your own writing is the bug
Shipped
ghostwriter 0.18.0 adds scripts/ai_tells.py, a deterministic AI-fingerprint gate that every draft runs through before a human sees it, plus a fix that opens the draft file on screen before every approval dialog. The interesting part is not the rule list; it is how the rules are kept honest. Each one is tested in both directions, and the whole table runs over the author’s real published posts, with a hard contract: a rule that fires on writing a human actually shipped is a false positive in the rule, never a defect in the writing. That calibration trick is worth building on its own, so that is what this guide walks through.
Why prose rules were not enough
The rules already existed. A voice-notes file listed every ban: no em dashes, no “here’s the thing”, no reflexive closing question. The model read that file and applied it to itself, and that was the whole enforcement story. Nothing on the draft’s actual path executed a single check.
A rule the model applies to itself is a suggestion, and the gap shows up exactly when you cannot afford it: in the drafts that look fine. Generic AI detectors do not close this gap either. OpenAI retired its own AI text classifier for low accuracy; it caught about 26% of AI-written text while flagging human writing 9% of the time. But you do not need to solve detection in general. LLM text has stable, mechanical tells; the excess-vocabulary study of 15 million PubMed abstracts found signs of LLM processing in at least 13.5% of 2024 abstracts by tracking a handful of overused words. Your own list of tells, tuned to your own corpus, is a much easier target: a small rule table you can run deterministically, for free, on every draft.
Build the rule table
One file, no dependencies. Each rule is an id, a severity, and a pattern. Two severities is the whole vocabulary: FAIL is a hard ban, WARN is a smell your real writing sometimes carries, so it prints but never blocks.
# ai_tells.py
import re
import sys
from pathlib import Path
FAIL, WARN = "FAIL", "WARN"
RULES = [
("em_dash", FAIL, re.compile("—")),
("staccato_no_list", FAIL, re.compile(r"\bNo \w+\. No \w+\. No \w+\.")),
("heres_the_thing", FAIL, re.compile(r"\bhere.s the thing\b", re.IGNORECASE)),
("slop_words", FAIL, re.compile(r"\b(delve|tapestry|testament to|game.changer)\b", re.IGNORECASE)),
("hedge_words", WARN, re.compile(r"\b(honestly|arguably|to be fair)\b", re.IGNORECASE)),
]
# The last line gets its own rules: endings are where generated text
# reaches for a reflexive question or a tidy reversal.
REFLEXIVE_CTA = re.compile(r"^(thoughts\?|what('| i)s your .{1,40}\?)$", re.IGNORECASE)
def check(text):
findings = []
for lineno, line in enumerate(text.splitlines(), start=1):
for rule, severity, pattern in RULES:
match = pattern.search(line)
if match:
findings.append({"rule": rule, "severity": severity,
"line": lineno, "excerpt": match.group(0)})
stripped = [ln.strip() for ln in text.splitlines() if ln.strip()]
if stripped and REFLEXIVE_CTA.match(stripped[-1]):
findings.append({"rule": "reflexive_cta", "severity": FAIL,
"line": len(text.splitlines()), "excerpt": stripped[-1]})
return findings
def exit_code(findings):
if any(f["severity"] == FAIL for f in findings):
return 2
if findings:
return 1
return 0
if __name__ == "__main__":
draft = Path(sys.argv[1]).read_text()
results = check(draft)
for f in results:
print(f'{f["severity"]} {f["rule"]:<18} line {f["line"]}: "{f["excerpt"]}"')
sys.exit(exit_code(results))
The three exit tiers matter more than they look. Clean is 0, WARN-only is 1, any FAIL is 2. That lets the publish step block on 2 while still surfacing 1, so a smell gets seen without a human having to override the gate to ship real work.
Test every rule in both directions
A one-sided test suite is how a linter rots. If you only assert that bad input fires, someone can later widen a pattern until it fires on everything, and the suite stays green. ESLint bakes this into its tooling: RuleTester runs every rule over paired valid and invalid case arrays. The same convention is one parametrized test in pytest, and the near-miss column is where the real design work happens:
# test_ai_tells.py
import pytest
from ai_tells import FAIL, check
TWO_SIDED = [
("em_dash", "the fix — which held", "the fix, which held"),
("staccato_no_list", "No backend. No database. No CMS.",
"There is no backend here, and no database either."),
("heres_the_thing", "Here's the thing about caching.",
"The thing about caching is eviction."),
("slop_words", "Let me delve into the details.",
"Let me get into the details."),
("reflexive_cta", "Shipped the gate today.\n\nThoughts?",
"Shipped the gate today.\n\nThe rollout notes are next."),
]
@pytest.mark.parametrize("rule,bad,near_miss", TWO_SIDED)
def test_rule_two_sided(rule, bad, near_miss):
assert any(f["rule"] == rule for f in check(bad)), f"{rule} missed its target"
assert not any(f["rule"] == rule for f in check(near_miss)), f"{rule} fired on a near miss"
Each near miss is a sentence that legitimately gets close: the same words in an honest order, the banned phrase minus its tell. Writing it forces you to decide where the rule’s edge actually is, before a false positive decides for you.
Calibrate against writing a human actually shipped
Here is the piece that makes the gate trustworthy. Collect your published writing, the posts that went out and that you would defend, into a corpus directory, and assert the entire rule table stays silent on all of it:
# test_corpus.py
from pathlib import Path
from ai_tells import FAIL, check
CORPUS = Path(__file__).parent / "published"
def test_published_writing_passes_the_gate():
posts = sorted(CORPUS.glob("*.md"))
assert len(posts) >= 8, "corpus glob matched too little to mean anything"
for post in posts:
fails = [f for f in check(post.read_text()) if f["severity"] == FAIL]
assert not fails, f"{post.name}: {fails}"
This test encodes a policy: the corpus outranks the rule. In ghostwriter it runs over ten published drafts, and it reshaped the table twice during this release. A paragraph-length rule initially matched the style guide’s 40-word target, but real published posts carried paragraphs up to 57 words, so 40 became the WARN line and 60 the FAIL line, an essay block no real post had ever produced. And an antithesis rule fired on a mid-post line the author had published on purpose, which is how the ban narrowed to closing lines only, with mid-body matches demoted to WARN.
Both changes go the same direction. When rule and corpus disagree, the rule narrows. You are not building a general AI detector; you are drawing the boundary of one person’s writing, and the shipped posts are the only ground truth you have.
The >= 8 floor is not decoration. A glob that matches nothing iterates zero times and passes, which turns the most important test in the suite into a green light wired to nothing. Assert the corpus is big enough to mean something before asserting anything about it.
Put the gate on the path the draft actually takes
A gate only counts if the artifact cannot route around it. In ghostwriter that is two hooks: the skill runs the checker before a draft is ever shown (and again after every edit), and the publish script refuses any FAIL on the exact text being posted. The shell version of that contract:
python3 ai_tells.py draft.md
code=$?
if [ "$code" -ge 2 ]; then
echo "gate refused the draft (exit $code)"
exit 1
fi
Try it end to end. Seed a corpus (your own posts in real life; placeholders prove the wiring), then run the suite and the gate. pytest is the only dependency, and it lives in a venv:
python3 -m venv .venv && ./.venv/bin/pip -q install pytest
mkdir -p published
for i in 1 2 3 4 5 6 7 8; do
printf 'A short post about a real thing that shipped.\n' > "published/post-$i.md"
done
./.venv/bin/python -m pytest test_ai_tells.py test_corpus.py -q
printf "Let me delve into the details.\n\nThoughts?\n" > draft.md
python3 ai_tells.py draft.md; echo "exit: $?"
The output below is from a real run of exactly these files:
...... [100%]
6 passed in 0.01s
FAIL slop_words line 1: "delve"
FAIL reflexive_cta line 3: "Thoughts?"
exit: 2
One deliberate escape hatch: the real publish script accepts a bypass flag, documented as human-only. The model is instructed never to pass it, and because the gate is deterministic, a bypassed FAIL is visible in the transcript instead of silently absorbed.
ghostwriter layers one more check on top: an LLM judge (--judge) that scores AI-likeness 0 to 10 against the author’s real voice files, using claude -p for a non-interactive call with JSON output, a pre-call spend cap (default $0.10), and a floor score of 7. It catches the thing regexes cannot see: tone. But it arrives second for a reason: the deterministic layer is free, instant, and never changes its mind, so it takes the load, and the judge only rules on drafts that already pass it. If the judge CLI is missing, the gate says so and runs without it rather than failing the draft.
Gotchas
- The judge read an empty directory and scored confidently anyway. The judge builds its prompt from the author’s voice files, and it resolved them from the repo’s
voice/directory, which is gitignored precisely because the real files are personal. Symptom: plausible scores with no grounding at all, indistinguishable from working. The fix resolves the user-level config directory (~/.claude/ghostwriter/voice/) first. If your checker loads context from a path, prove the load happened; a missing directory should print a line, not silently shrink the prompt. - The example voice file carried the banned em dash. The template users copy to seed their own voice rules had an em dash in a heading, so every fresh install shipped a spec that violated itself, and the judge’s context taught the tell it was supposed to catch. Escape: your gate’s own reference files belong in its test corpus.
- The docs claimed a 31-draft corpus; the committed corpus was ten. Nothing failed, because nothing counted the files. That is the same failure mode the
min_corpusfloor exists for, one level up: any number in prose that code can check should be checked by code, or it drifts. - A rule fired on a line the author liked. The antithesis pattern flagged a mid-post mechanistic contrast that had shipped in a real post and worked. It would have been easy to call that an acceptable loss; the corpus contract says otherwise, and the rule got position awareness instead. Narrowing one rule is cheap compared to a linter people learn to ignore.
Sources
- OpenAI: New AI classifier for indicating AI-written text — the classifier’s retirement for low accuracy, and its 26% true-positive / 9% false-positive rates
- Kobak et al., Delving into LLM-assisted writing in biomedical publications through excess vocabulary — measurable lexical tells in at least 13.5% of 2024 PubMed abstracts
- ESLint: RuleTester — the paired valid/invalid convention for testing lint rules in both directions
- Claude Code CLI reference — the
-pprint mode and--output-format jsonused for the scripted judge call