Nate Swenson

Your model keeps saying the thing you told it not to say

Local Fitness · No. 067 · 8 min read ·v0.28.1

Shipped

My fitness agent grades a workout and writes a short coaching read above the grades. The read is told, in capitals, never to name a letter grade, because the letters are already printed in the table right below it. It named one anyway, in about three percent of paragraphs. This release drove that to zero.

The fix is a prompt-design lesson that applies far past this repo: a negative instruction cannot beat a token you have placed in the model’s own context. The prompt was handing the model every letter grade and then forbidding it to repeat them. This is how to find that pattern and remove it.

First, measure the thing you are about to fix

Before changing anything, I wrote a detector and got a number. My first detector was wrong: it counted the article “A” as a grade, so “A blown interval session” scored as a leak. A measurement instrument needs its own test before its output is allowed to drive a decision, so the two lists below, real leaks and lookalikes, are the actual specification of the detector.

# grades.py
import re

# A letter grade named in prose. Deliberately NARROW: the false positive is the
# expensive error, because a bare "A" is almost always the article ("A blown
# session..."), and treating it as a grade throws away good output and pays for
# a regeneration. A signed letter is unambiguous; a bare letter only counts
# when it is punctuated like a sentence, preceded by an article, or followed by
# grade-talk.
_GRADE = re.compile(
    r"(?:^|(?<=[\s(]))"
    r"(?:"
    r"[A-DF][+-]"                                   # signed: unambiguous
    r"|[A-DF](?=[.,;:!?]\"?\s)"                     # "F. Target..."
    r"|(?:an?\s+)[A-DF](?=[\s.,;:])"                # "an F", "a B"
    r"|[A-DF](?=\s+(?:is|on paper|effort|grade))"   # "F is F", "B on paper"
    r"|[A-DF](?=-grade\b)"                          # "F-grade pace"
    r")"
)

def find_grade(text):
    """The first letter grade named in `text`, or None if it is clean."""
    m = _GRADE.search(text or "")
    return m.group(0).strip() if m else None

Why the rule was losing

Language models are not good at “not”. Truong and colleagues, in Language models are not naysayers (2023), found LLMs show “insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.” The classic demonstration is older: Kassner and Schütze’s Negated and Misprimed Probes for Pretrained Language Models (ACL 2020) showed models “do not distinguish between negated and non-negated” prompts, filling both “Birds can [MASK]” and “Birds cannot [MASK]” the same way.

Anthropic’s own prompt engineering guidance draws the practical conclusion first, under “Control the format of responses”: “Tell Claude what to do instead of what not to do.” Instead of “Do not use markdown,” say “Your response should be composed of smoothly flowing prose paragraphs.”

My prompt was doing the opposite, and worse than the usual case. It was not just saying “do not name a grade”; it was printing every grade the model was asked to write about, one line above the ban:

Overall grade D (1.05 GPA).

Metric grades (already computed — phrase, don't re-derive):
  Distance: D- — actual 5.95 mi vs target 5.00 mi.
  Pace: F — actual 9:25/mi best mile vs target 6:58/mi.

The read was not disobeying. It was completing. And a retry produced the same letter, because the retry saw the same prompt.

Carry the meaning without the token

The move is to keep the judgment and drop the token. Map each grade to a severity phrase, and build the context out of phrases:

# grades.py, continued
_SEVERITY = {"A": "on target", "B": "slightly off target", "C": "off target",
             "D": "well off target", "F": "missed badly"}

def severity(grade):
    """Severity word for a grade, keyed on the base letter so 'D-' and 'D+'
    read the same. Unknown or ungraded -> 'n/a'."""
    if not grade or grade == "n/a":
        return "n/a"
    return _SEVERITY.get(grade[0], "n/a")

def build_context(metrics):
    """The model's input. `metrics` maps a name to its computed letter grade;
    the context names the SEVERITY, never the letter, so the token the model is
    told not to repeat is not sitting in front of it to complete."""
    lines = [f"{name}: {severity(grade)}" for name, grade in metrics.items()]
    return "Verdicts (phrase these, never name a letter grade):\n" + "\n".join(lines)

Severity, not silence. Dropping the judgment entirely would be worse than the leak: an overshoot is “well off target” for a reason that the raw numbers do not carry on their own, and a read that praised it would contradict the grade printed beside it. What must not survive into the context is the letter.

Keep a backstop, and make it cheap

The prompt change does the real work, but a prompt is a request, not a guarantee. Wrap the generation in a check that regenerates once on a hit, and only keeps the retry if it is actually clean:

# grades.py, continued
def generate_clean(context, generate, max_attempts=2):
    """Call `generate(context)`, and if the result names a grade, try once
    more. Keep a retry only if it is actually clean; the first result always
    stands as the floor, so a pathological input never costs more than
    `max_attempts` calls.

    `generate` is your model call. Swap in whatever client you use.
    """
    best = generate(context)
    for _ in range(max_attempts - 1):
        if find_grade(best) is None:
            return best
        retry = generate(context)
        if find_grade(retry) is None:
            return retry
    return best

One retry, never a loop. Sampling is not deterministic, so a second draw is genuinely different and clears most leaks; but a card that leaks no matter what must not spend unbounded calls, and the first result is no worse than any later one.

Use it, then verify it

build_context proves the fix by construction, and you can see it:

# run_grades.py
from grades import build_context, find_grade, generate_clean

metrics = {"distance": "D-", "pace": "F", "hr": "B+", "load": "D+"}

leaky = "\n".join(f"{k}: {v}" for k, v in metrics.items())
print("leaky context detector says:", find_grade(leaky))
print("clean context detector says:", find_grade(build_context(metrics)))

python run_grades.py prints:

leaky context detector says: D-
clean context detector says: None

The letters are gone from the context the model sees. Now pin the detector and the enforcement loop with tests. The false-positive list is the half that matters, so it is the longer one:

# test_grades.py
import pytest
from grades import build_context, find_grade, generate_clean, severity

REAL_LEAKS = [
    "F is F. Your best mile was 9:25 against a 6:58 target.",
    "F. Target 6:58/mi, your best mile was 9:25.",
    "B+ on paper, but that is the tell.",
    "You got a B for that.",
    "That is a D- effort and you know it.",
    "Combine that with F-grade pace and it was a write-off.",
    "2:27 slow, an F, no rounding it up.",
]

LOOKALIKES = [
    "A blown interval session that was also light on load.",
    "81 against a 105 target means you left work on the table.",
    "136 bpm sits under the 145 floor.",
    "A full minute per mile too hot for a recovery day.",
    "An easy day is supposed to be easy.",
    "Your 5K pace is not your 10K pace.",
]

@pytest.mark.parametrize("text", REAL_LEAKS)
def test_a_real_leak_is_caught(text):
    assert find_grade(text) is not None

@pytest.mark.parametrize("text", LOOKALIKES)
def test_a_lookalike_does_not_fire(text):
    assert find_grade(text) is None

def test_context_names_severity_not_letters():
    ctx = build_context({"pace": "F", "distance": "D-", "hr": "B+"})
    assert "missed badly" in ctx
    assert find_grade(ctx) is None  # no letter reached the model

def test_generate_clean_retries_a_leak_once():
    calls = []
    def fake(_ctx):
        calls.append(1)
        return "an F, no rounding it" if len(calls) == 1 else "2:27 off the ask"
    assert generate_clean("ctx", fake) == "2:27 off the ask"
    assert len(calls) == 2

def test_generate_clean_keeps_the_first_when_both_leak():
    assert generate_clean("ctx", lambda _c: "an F, first") == "an F, first"

python -m pytest test_grades.py -q reports 16 passed on the cases shown here, which are the load-bearing ones. On my real cards, across three cards and eight generations each, the leak rate went from 3 of 96 paragraphs to 0 of 96, with median latency unchanged, because the retry almost never has to fire once the letters leave the context.

Gotchas

A retry against the same prompt reproduces the same leak. I built the detector-and-regenerate loop first and expected it to mostly work on its own. It did not: the log showed the read named “D-”, regenerated, and named “D-” again. The symptom is a retry rate that barely helps. The escape is to realize the retry is not the fix; the prompt is. Remove the token from the context and the retry becomes a rare backstop instead of a coin flip.

A greedy detector eats your good output. My first pattern matched a bare “A”, which is almost always the article, so it would have discarded clean reads and paid to regenerate them. The symptom is a “leak rate” that looks alarming and a retry budget you keep raising. The escape is to treat the false-positive list as the real specification: write the lookalikes down first, make them all pass, and only then trust the number the detector reports.

Deleting the judgment is worse than the leak. The tempting shortcut is to strip the grade from the context and say nothing in its place. Then the read has no idea a mild-looking number is actually a miss, and it will cheerfully praise something the table grades poorly. The escape is severity words: carry the verdict, drop only the token.

Sources

Changelog

  • release: 0.28.1 — stop showing the read the letter grades it was told never to say (dev → main) (cee795f)