Nate Swenson

Sanitizing a public repo when the leak isn't a credential

Local Budget · No. 082 · 10 min read ·v0.2.0

Shipped

v0.2.0 of a local-first budgeting tool shipped four things: splitting one bank charge across categories, a connector that pulls item-level detail behind a merchant charge, dev-to-main promotion automation, and a redesigned monthly PDF.

Four other commits in the same range did something less fun. The repo was already public, and a pass over it found real personal values sitting in code comments, design docs, .env.example, and test fixtures. None of it was a credential. That is exactly why nothing had flagged it.

This is the how-to for that pass: finding the values a scanner will never flag, replacing them without corrupting the file, and verifying the bytes you committed rather than the ones on your disk.

What a secret scanner will not find for you

GitHub’s push protection blocks pushes containing recognizable secrets, and it works well: it matches known token shapes from a long list of providers, and organizations can add custom patterns for their own. Every one of those is a pattern someone can write down in advance.

Your own data has no shape. A dollar amount looks like every other dollar amount. A first name in a golden file looks like test data, because it is test data; it is just test data that happens to be a real person.

NIST SP 800-122 is useful here because it defines personally identifiable information to include anything “linked or linkable to an individual,” and calls out financial information by name. That is a wider net than “credentials,” and it is the net you want when reading your own repo. The categories worth grepping for:

  • Amounts from a real account, in comments, fixtures, and docs written against live data.
  • Labels that encode a place or an institution. A category name for a savings plan can carry the US state it was opened in. That is linkable information doing its job.
  • Names in golden files and fixtures, including family members.
  • Payee strings copied out of a real statement for a parser test.

Start with an inventory rather than an edit. Write down each real value and what it should become before touching a file:

# Candidate leaks: money-shaped strings outside of formatting code.
grep -rnE '\$[0-9][0-9,]*\.[0-9]{2}' src/ tests/ docs/ \
  | grep -vE 'format|f"|:,\.2f'

# Candidate leaks: capitalized words in fixtures, which is where names hide.
grep -rnE '"[A-Z][a-z]{2,}"' tests/fixtures/ tests/golden/

One amount has several spellings

This is the part that turns a five-minute job into a broken test suite.

A single amount appears in a repo in more than one form. The fixture stores it as integer cents, the assertion beside it stores the rendered string with a currency symbol and a thousands separator, and a doc quotes it plainly. Replace one spelling and you have not scrubbed the file; you have made it internally inconsistent.

So generate the spellings from the value instead of hunting them by eye:

# scrub.py
"""Replace real values with synthetic ones across a working tree."""
from __future__ import annotations

import sys
from pathlib import Path

# Real value -> synthetic replacement. Keep the replacement the same SHAPE as
# the original (same magnitude, same length where a test measures width) so
# the code paths under test still behave the same way.
AMOUNTS = {
    "4821.37": "900.00",
    "21.37": "12.00",
}
NAMES = {
    "Springfield529": "State529",
    "Dana": "Alex",
}


def money_variants(raw: str) -> list[str]:
    """Every spelling one amount has in a repo."""
    n = float(raw)
    return [
        f"${n:,.2f}",         # $4,821.37
        f"{n:,.2f}",          # 4,821.37
        f"${n:.2f}",          # $4821.37
        f"{n:.2f}",           # 4821.37
        str(round(n * 100)),  # 482137, the integer-cents column
    ]

Pick replacements that keep the shape of the original. If a test asserts a column is 12 characters wide, a synthetic value of a different length turns a privacy fix into a layout failure. If a chart test exists to prove a savings row an order of magnitude above the spending rows does not flatten the scale, the synthetic savings value has to stay an order of magnitude above them. The substitution should exercise the same code path, not merely occupy the same slot.

Apply the longest match first

The second trap is ordering. 21.37 is the tail of 4821.37. Replace the short one first and the long one becomes 4812.00, which is a number that never existed in any account and now lives in your tests.

Sorting the rules by length, descending, fixes it:

def build_rules() -> list[tuple[str, str]]:
    """(find, replace) pairs, longest find first."""
    rules: list[tuple[str, str]] = []
    for old, new in AMOUNTS.items():
        for old_v, new_v in zip(money_variants(old), money_variants(new)):
            rules.append((old_v, new_v))
    rules.extend(NAMES.items())
    # Longest first: without this, "21.37" rewrites the tail of "4821.37".
    return sorted(set(rules), key=lambda r: len(r[0]), reverse=True)


def scrub(paths: list[Path]) -> int:
    rules = build_rules()
    changed = 0
    for path in paths:
        text = original = path.read_text(encoding="utf-8")
        for find, replace in rules:
            text = text.replace(find, replace)
        if text != original:
            path.write_text(text, encoding="utf-8")
            changed += 1
            print(f"scrubbed {path}")
    return changed


if __name__ == "__main__":
    n = scrub([Path(p) for p in sys.argv[1:]])
    print(f"{n} file(s) changed")

Run it against a fixture and the test that reads it:

python scrub.py fixtures.py test_report.py

scrubbed fixtures.py
scrubbed test_report.py
2 file(s) changed

Then run your tests. If the suite goes red, that is the tool telling you a spelling was missed, and it is worth reading the failure rather than reaching for the fixture:

    def test_label_matches_the_amount():
>       assert render(SAVINGS_CENTS) == SAVINGS_LABEL
E       AssertionError: assert '$4,821.37' == '$900.00'

The label was replaced and the integer-cents constant was not. That failure is the one that argues for generating variants from the value: the missing spelling was the storage format, not a display format, and no amount of squinting at the file finds it reliably.

Verify the bytes you committed, not the ones on your disk

Here is the step people skip, and it is the reason this pass took an extra commit.

Your working copy is not what you pushed. If you stage some of the scrubbed files and not others, your local suite still passes, because your disk holds a fully consistent set. The commit holds a mixed one.

So check out the commit into a clean directory and run the suite there:

# verify-committed.sh
set -euo pipefail
VERIFY_DIR="$(mktemp -d)"
git archive HEAD | tar -x -C "$VERIFY_DIR"
cd "$VERIFY_DIR"
python -m pytest -q

git archive HEAD writes exactly the tracked bytes at HEAD, with no working-tree state and no untracked files. Running the suite there answers a different question from the one your local run answers.

Staging only the fixture and committing gives you two different answers from the same repo:

=== working copy ===
..                                                                       [100%]
2 passed in 0.00s

=== committed tree ===
E         + Alex / State529
FAILED test_report.py::test_report_header - AssertionError: assert 'Alex / St...
1 failed, 1 passed in 0.01s

Two passed on disk. One failed in the commit. Add that script to CI, or run it before you push a sanitization pass, and the class of bug disappears.

Scrubbing the tree does not scrub the history

Everything above fixes what a visitor reads when they open your repo. It does nothing about what git log -p shows them, because the parent commit still holds the original blob and is still reachable.

That is worth stating plainly, because it is easy to finish a sanitization pass feeling done. Confirm it for yourself:

git merge-base --is-ancestor <scrub-commit>^ origin/main && echo "pre-scrub blob still reachable"

Removing values from history is a separate job with real costs. GitHub’s own guidance points at git filter-repo or the BFG Repo-Cleaner, and is direct about the limits: rewriting changes the SHAs of every dependent commit, which affects open pull requests, and the old commits may still be reachable in clones, in forks, and through cached views. For replacing a string rather than deleting a file, git-filter-repo takes a rules file in the same old==>new format the scrub map already is, so the substitution table is reusable.

Decide deliberately which of the two jobs you are doing, and say which one you did. A tree that reads clean while the history does not is a defensible state; believing the history is clean when it is not is not.

Gotchas

A number can be the tail of another number. Replace 21.37 before 4821.37 and you silently produce 4812.00. Symptom: an amount in your tests that never existed, and arithmetic assertions that fail for reasons unrelated to the value you were scrubbing. Escape: sort substitution rules by find-length descending, and diff the result rather than trusting the run.

The integer-cents column is a spelling you will forget. Money stored as 482137 does not match a search for 4821.37, so a scrub can rewrite every visible amount and leave the stored one behind. Symptom: a test asserting a rendered label against a stored value fails immediately after the pass. Escape: generate variants from the value, including the storage format, instead of listing them by hand.

Your working copy lies about what you committed. Partial staging leaves your disk consistent and the commit inconsistent, so the suite passes locally and fails for everyone else. This cost an extra commit here, whose message says it plainly: the verification was against the wrong bytes. Escape: git archive HEAD into a temp directory and run the suite there before pushing.

Golden files derive from fixtures and do not update themselves. Scrubbing a fixture leaves any committed HTML or snapshot output holding the old value, which is both an unscrubbed leak and a failing test. Escape: regenerate goldens as part of the pass, and grep the regenerated files for the old values afterward.

Not everything that looks personal should go. The author field in your package metadata and your name in a brand attribution are your name on your own repository, which is the point of publishing it. Deciding what stays is part of the pass; scrubbing reflexively just makes the repo worse.

Sources

Changelog

  • ci: adopt shipflow dev-main-promotion automation (242b3f6)
  • feat(splits): allocate one charge across categories (closes #1) (9ae77ca)
  • chore(privacy): replace personal category names and fixture identity (6032864)
  • chore(privacy): sanitize real figures out of the design docs (11dd6f1)
  • chore(privacy): remove real ledger figures from a public repo (f1a228d)
  • test(amazon): cover the two sync failure paths before promotion (31c5b9c)
  • fix(security): read-deny amazon_sync_runs.error_message to the agent (b8967f7)
  • docs(visualizer): re-truth the skill after the PRESS + Amazon merges (8f0ba5b)
  • feat(amazon): make “break down my Amazon purchases” a real answer (423480e)
  • feat(amazon): refresh item data before a report, never at its expense (2130dcd)
  • fix(amazon): a restored session must be marked authenticated (59f494b)
  • feat(amazon): capture a session instead of storing a password (a633a13)
  • test(amazon): pin the contract against the real parser, not our fakes (d7f7402)
  • feat(amazon): connector for item-level detail behind Amazon charges (f9d4a4e)
  • feat(report): full-bleed paper, an opening line, and accent that measures (3438b07)
  • fix(report): floor rows no longer set the spend-vs-budget scale (8c458aa)
  • feat(report): put the monthly PDF on the shared PRESS brand (d30bb94)