Nate Swenson

The minus sign my checker could not see

Local Fitness · No. 125 · 6 min read ·v0.60.2

Shipped

A one-commit fix to my fitness agent’s grounding checker, the code that scans an LLM-written brief for numbers that do not exist in the data it was given. A live audit caught the checker reporting an invention rate of 1.000, every number flagged as hallucinated, on a brief that was actually clean. Two false-positive classes were responsible: the typographic minus sign, and calendar dates. Both fixes generalize to anything that verifies model output by matching strings against source data.

A checker that cries wolf is worse than no checker

The grounding check works like this: collect every number the model was allowed to cite into a pool, tokenize every number out of the prose, and flag prose numbers with no plausible source. It is the last line of defense against the model inventing a heart rate, and its output feeds an eval gate that can fail a release.

Which is why a 1.000 invention rate matters twice over. It buries any real invention in noise, and it means the recorded eval baselines partially encode parser artifacts rather than model behavior. When your verifier is wrong, everything calibrated against it is wrong with it. So the debugging session was not about the model at all; it was about reading the flagged sentences one by one and asking what the tokenizer actually saw.

Class one: the minus sign that is not a hyphen

The brief’s markdown rendered negative numbers the typographically correct way: “−7.5”, using U+2212, the dedicated minus sign from Unicode’s Mathematical Operators block. Unicode deliberately separates it from the keyboard’s hyphen-minus, U+002D, and mathematical typesetting prefers U+2212 because its metrics match the plus sign. Models write it constantly, because well-edited text in their training data writes it.

My tokenizer’s regex knew only the ASCII hyphen:

import re

_NUM_RE = re.compile(r"[-+]?\d[\d,]*(?:\.\d+)?\s*[kK%]?")

_NUM_RE.findall("TSB is −7.5 today")   # ['7.5 ']  ← the sign is gone

So “−7.5” tokenized as a bare positive 7.5, and the checker flagged it as a sign inversion against the true value of -7.5. The model had written the number correctly; the checker could not read its own generator’s output. This is the same character-identity trap that UTS #39 catalogs for security purposes: visually identical strings that are different code points, detectable only if you map both to a canonical form before comparing.

The fix is a one-character normalization at the top of every tokenizing entry point:

_MINUS_SIGN = "−"

def _normalize(text: str) -> str:
    return text.replace(_MINUS_SIGN, "-")

One deliberate property: U+2212 and “-” are both one character, so the replacement is offset-safe. Every positional check the tokenizer does later (lookbacks and lookaheads keyed on match positions in the same string) stays valid. Normalize with anything that changes string length and every m.start() downstream is quietly wrong.

Class two: a number after a month name is a date

The second class: “the Aug 7 long run” sign-flagged against a metric worth -7.5, three times in one audit, and “Sept 18” matched a workout’s training load. Calendar dates are numbers in prose that will never be in a metric pool, and briefs about training schedules are full of them.

The checker already had a veto for the mirror case, a number followed by a window word (“14 days” is a window, not a claim). Dates get the mirrored rule, a veto for a number directly preceded by a month word:

_MONTH_BEFORE = re.compile(
    r"\b(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?\s+$",
    re.IGNORECASE,
)

def _after_month_name(text: str, start: int) -> bool:
    return bool(_MONTH_BEFORE.search(text, max(0, start - 12), start))

The prefix-plus-tail pattern (aug, then any letters, then an optional dot) covers “Aug”, “Aug.”, and “August” in one branch, and the 12-character lookback window comfortably holds "September " plus a word boundary. Wired into the tokenizer:

def numeric_tokens(text: str) -> list[str]:
    text = _normalize(text)
    tokens = []
    for m in _NUM_RE.finditer(text):
        if _after_month_name(text, m.start()):
            continue          # "Aug 7" is a date, not a metric claim
        tokens.append(m.group().strip())
    return tokens

Pin the sensitivity, not just the fix

Every veto you add to a checker is a hole you have punched in it. The test suite has to hold both directions, the false positive gone and the true positive still caught:

def test_typographic_minus_keeps_its_sign():
    assert numeric_tokens("TSB is −7.5 today") == ["-7.5"]

def test_month_prefixed_number_is_vetoed():
    assert numeric_tokens("the Aug 7 long run went well") == []

def test_bare_number_still_flags():
    # Sensitivity is positional: 7 NOT after a month must still tokenize.
    assert numeric_tokens("your score of 7 was invented") == ["7"]

And the sentences from the live audit are frozen as a regression fixture, so the exact prose that fooled the checker is what future versions are graded against. A checker’s best test corpus is its own past mistakes.

Verify it

Drop the three functions and the tests above into a file and run pytest against it:

$ pytest test_grounding_tokens.py -q
...                                                                      [100%]
3 passed in 0.00s

Then run your own generator’s real output through numeric_tokens and read the token list next to the prose. Every number you see in the text should appear in the list with the sign you see on screen, and no date fragments should appear at all.

Gotchas

  • Do not map the en dash while you are at it. U+2013 looks like one more dash to normalize, but it is a range mark: “3-4mi” already tokenizes its second half as a negative number in a naive regex, and mapping more dash lookalikes to “-” manufactures new sign-inversion false positives. My checker maps exactly one character, U+2212, because that is the one the generator’s markdown actually emits as a minus.
  • Fix every tokenizing entry point, or a sibling caller keeps the bug. The checker had two ways in, the flagging pass and a public numeric_tokens re-export that another feature’s grounding uses. The first audit fixed one; the second audit found the same class again through the other. If tokenizing logic is duplicated even slightly, route both through one function before tuning either.
  • The second audit came the same day as the first. The unit-binding fix from the morning audit looked complete; the 13:48 regeneration immediately measured 1.000 again from classes the first fix did not cover. Re-run the checker on live output after every fix; one clean fixture proves the fixture, not the checker.
  • Some residue is designed tolerance. A bare unitless number near a metric (“streak alive by 45” against a resting heart rate of 50) is indistinguishable from a mis-stated metric without semantic understanding. Chasing it with more string rules costs sensitivity elsewhere. Write the accepted residue down in the changelog so the next audit does not re-litigate it.

Sources

  • U+2212 MINUS SIGN — the character’s identity: Math Symbol category, Mathematical Operators block, distinct from the hyphen
  • Hyphen-minus — U+002D’s overloaded roles, and parsers generally recognizing only it as an operator
  • UTS #39: Unicode Security Mechanisms — confusable characters, and testing visual confusability by comparing canonical skeletons

Changelog

  • fix: grounding normalizes the typographic minus and vetoes month-name dates (0.60.2) (#221) (#222) (da31349)