A covered line and a tested line are not the same thing
Shipped
Local-fitness is the personal app that turns my Garmin data into a daily brief. Version 0.13.0 added a chart tool the agent can call to render a metric as a terminal graph, and alongside it ran a hardening pass: about 400 lines of confirmed-unused code deleted (three throwaway scripts, a handful of dead functions, an orphaned frontend component), and test coverage raised from 65% to 89%, with the CI coverage gate raised from 43% to 85% to lock the gain in. Raising a coverage number is the easy part. A later pass over the new tests found ones that raised the number without actually checking anything, which is the part worth writing down.
The number can be lying to you
A coverage report answers one question: did this line run during the test suite. It does not answer the question you actually care about, which is whether a test would notice if the code were wrong. Martin Fowler’s summary of this is blunt: test coverage is a poor target but a useful tool, good for finding code that no test touches at all, bad as a score to chase. The gap between “ran” and “verified” is exactly where a bad test hides.
Two ways that gap shows up in a real codebase:
- A test that calls a function and asserts something trivially true, so the line executes but nothing is actually checked.
- A function whose only caller, anywhere in the codebase, is its own unit test. The coverage tool marks it green. The function is still dead code; the “usage” is a test file talking to itself.
Both count as covered. Neither is tested. Mutation testing exists as a discipline specifically to catch the first kind: deliberately break the implementation in a small way and see if any test fails. If nothing fails, the mutation “survives,” and you’ve found a test that was decoration. You don’t need a mutation-testing tool installed to use the idea; you can run the check by hand on code you’re suspicious of.
Confirm it’s dead before you delete it
Dead code isn’t neutral while it waits. Fowler’s Yagni argument is that code you don’t need adds complexity that makes the software around it harder to modify and debug, and that cost keeps running until the code is removed. But coverage percentage alone won’t tell you a function is safe to delete, because of that second failure mode above. Before removing anything, grep for real callers, not just for test coverage.
You can follow the whole thing in a scratch directory; the only setup is pip install pytest pytest-cov and these two files:
# pricing.py
def apply_discount(price: float, pct: float) -> float:
"""Apply a percentage discount, floored at 0."""
return max(0.0, price * (1 - pct))
def legacy_discount(price: float) -> float:
"""Old flat $5-off discount. Superseded by apply_discount."""
return max(0.0, price - 5)
# test_pricing.py
from pricing import apply_discount, legacy_discount
def test_apply_discount():
result = apply_discount(100, 0.2)
assert result or True # always true, whatever apply_discount returns
def test_legacy_discount():
assert legacy_discount(10) == 5
Run coverage and both functions show fully covered:
pytest --cov=pricing --cov-report=term-missing
Name Stmts Miss Cover Missing
------------------------------------------
pricing.py 4 0 100%
------------------------------------------
TOTAL 4 0 100%
2 passed in 0.03s
100% and green. Now grep for who actually calls legacy_discount, outside its own test:
grep -rn "legacy_discount" --include="*.py" . | grep -v test_pricing.py
pricing.py:6:def legacy_discount(price: float) -> float:
Only the definition. No production caller, anywhere. That’s this release’s exact pattern: a batch of config accessors that turned out to have no callers in the app itself, only in their own test file, so their tests came out with the functions.
Delete it, then fix the test that let it hide
With the caller check done, delete the dead function and its self-referential test in the same change:
# pricing.py
def apply_discount(price: float, pct: float) -> float:
"""Apply a percentage discount, floored at 0."""
return max(0.0, price * (1 - pct))
# test_pricing.py
from pricing import apply_discount
def test_apply_discount():
assert apply_discount(100, 0.2) == 80.0
Note the second change: test_apply_discount no longer just asserts result or True, it checks the actual number. Rerun the suite:
pytest --cov=pricing --cov-report=term-missing
Name Stmts Miss Cover Missing
------------------------------------------
pricing.py 2 0 100%
------------------------------------------
TOTAL 2 0 100%
1 passed in 0.02s
Still green, still 100%, half the code. If the deletion had broken something real, this is the step that would have gone red.
Verify the surviving test would actually catch a bug
The coverage report can’t tell you whether test_apply_discount is checking the real behavior or just running the line. Prove it by hand: break the implementation on purpose and confirm the test fails.
sed -i.bak 's/1 - pct/1 + pct/' pricing.py
pytest -q
F [100%]
_____________________________ test_apply_discount ______________________________
def test_apply_discount():
> assert apply_discount(100, 0.2) == 80.0
E assert 120.0 == 80.0
E + where 120.0 = apply_discount(100, 0.2)
1 failed in 0.02s
The test catches the mutation; it’s a real assertion, not a green light with nothing behind it. Restore the file and you’re back to a passing suite that you now have direct evidence you can trust:
mv pricing.py.bak pricing.py
pytest -q
. [100%]
1 passed in 0.01s
Gotchas
assert x or Truealways passes. It looks like an assertion and shows up green in CI, but it can never fail regardless of whatxis. A quality-gate pass over this release’s new tests found exactly this pattern in a retry-cancellation test; the fix was asserting the actual cancellation state instead, confirmed by checking that removing the real.cancel()call made the test fail.- A misnamed test can assert behavior the code doesn’t implement. The same review found a test whose name claimed one contract while the code actually implemented a different one; the fix was renaming the test to match the real behavior and adding a separate test for what the name had originally promised.
- An unexercised branch can sit inside a “covered” file. A skip-on-error branch in one module was never actually hit by any test until the test data was deliberately made invalid enough to trigger it. File-level coverage percentage doesn’t tell you a specific branch was ever taken.
- Type errors don’t fail your test suite if you never run the type checker. This release also added a frontend build-and-typecheck step to CI, because until then a broken TypeScript build could pass every existing check and still ship broken.
Sources
- TestCoverage — Martin Fowler — coverage is a poor target but a useful tool, including for finding code no test touches at all.
- Yagni — Martin Fowler — code with no real caller still carries an ongoing cost in comprehension and maintenance until it’s removed.
- What is mutation testing? — CircleCI — 100% line coverage can still miss bugs, because coverage measures which lines ran, not whether the assertions caught a defect.