Nate Swenson

The label said running; the pace said otherwise

Local Fitness · No. 065 · 12 min read ·v0.26.0

Shipped

0.26.0 fixed four grading bugs in my fitness agent’s workout report card. The one worth writing about: the card was comparing runs against walks. It picks a reference cohort of similar recent activities, takes the median, and grades the workout against that. Cohort membership was decided by the activity_type column, and my walking-desk sessions are recorded by the watch as treadmill_running. So a real interval session was being measured against a 15:50/mi stroll, and it scored A+ on heart rate and training load, which is 40% of the composite grade, for clearing a bar set by walking.

If you have any feature that scores a record against similar records, you have this decision to make. This is how to build a cohort that partitions on a measurement rather than on a label, and how to make it say what it excluded.

Why a category label can’t define a cohort

The label on a row is usually a classifier output, and classifier outputs have error rates. A validation study in JMIR mHealth and uHealth put four consumer trackers through supervised activity sessions and found the automatic activity type was right 93 to 100% of the time for a clean treadmill run, and right only 36 to 62% of the time for a walk-run-walk session. The device is not broken. It is answering a harder question than the column name implies.

Human-applied labels are no better. Northcutt, Athalye and Mueller audited ten of the most-used ML benchmark datasets and estimated at least 3.3% label errors on average, at least 6% in the ImageNet validation set, and showed that correcting them was enough to flip which model wins: ResNet-18 beats ResNet-50 once the prevalence of originally mislabeled test examples rises by 6%.

That is the part that bites. A contaminated cohort does not throw an error, it returns a number. And when the contamination splits a population into two groups, aggregating over the mixture gives you a statistic that describes neither one. That is Simpson’s paradox in its everyday form: an association can emerge, disappear or reverse when the population is divided into subpopulations. Google’s team made the same argument for production pipelines in Data Validation for Machine Learning: errors in input data quietly nullify whatever accuracy you gained elsewhere, which is why they built validation into the pipeline rather than trusting the upstream schema.

Measure the thing the label is standing in for

Start by naming the physical fact you actually care about. I don’t care what the watch called the activity, I care whether the ground was covered at running speed. That is one number and one threshold.

# cohort.py
"""Build a reference cohort by measurement, not by label."""

from statistics import median

# The run/walk boundary, in seconds per mile. A brisk walker reaches a
# 13:00 mile; sustained running essentially never falls below it.
RUN_PACE_CEILING_SEC_PER_MI = 13 * 60

# Fewer comparable rows than this and there is no cohort worth grading against.
MIN_COHORT = 5


def is_running_effort(pace_sec_per_mi):
    """True if run, False if walked, None if the mode is unknowable.

    None is a real answer, not a failure: a row with no usable pace belongs
    to neither cohort, and `x is mode` keeps it out of both without a
    special case.
    """
    if not pace_sec_per_mi or pace_sec_per_mi <= 0:
        return None
    return pace_sec_per_mi <= RUN_PACE_CEILING_SEC_PER_MI

The three-state return is the load-bearing bit. A boolean has to guess for the rows with no pace at all, and whichever way it guesses, those rows silently join a cohort they don’t belong to. Databases solved this a long time ago: PostgreSQL documents that ordinary comparison operators yield null, signifying unknown, when either input is null, so 7 = NULL is neither true nor false. Python’s is gives you the same property for free, since None is True and None is False are both false, and an unknown-mode row drops out of both cohorts without a branch.

Pick the threshold from the data, not from intuition. In my 60-day window the paces were cleanly bimodal: 16 real runs between 8:40 and 11:46 per mile, and 30 walking-pad sessions between 14:08 and 84:20. Nothing at all in between, so 13:00 sits in the empty band with about two minutes of margin on either side.

Partition first, then widen

Most cohort builders have a fallback. Mine prefers an exact type match and widens to all on-foot activity when the exact pool is too thin. The ordering of those two steps is the whole bug: widening is precisely the step that reaches for the contaminated corpus.

# cohort.py, continued

def exact_type(label):
    target = (label or "").lower()
    return lambda row: (row.get("type") or "").lower() == target


def on_foot(row):
    """The wider net: anything you cover ground with on your own two feet."""
    t = (row.get("type") or "").lower()
    return "running" in t or "walking" in t or "hiking" in t


def reference_cohort(subject, history):
    """Median pace and HR over rows comparable to `subject`.

    Order matters: partition on measured locomotion FIRST, then filter by
    label, then widen. Widening is what would otherwise drag the whole
    walking corpus into a thin running pool.
    """
    mode = is_running_effort(subject.get("pace_sec_per_mi"))
    rows, excluded = list(history), 0

    if mode is not None:
        in_mode = [r for r in rows
                   if is_running_effort(r.get("pace_sec_per_mi")) is mode]
        # Count only rows this cohort could plausibly have drawn from, so the
        # disclosure line is true of every one of them. A row honestly typed
        # `walking` was never a candidate for a running cohort.
        same_label = exact_type(subject.get("type"))
        excluded = sum(1 for r in rows if r not in in_mode and same_label(r))
        rows = in_mode

    pool, widened = (subject.get("type") or "comparable").lower(), False
    matched = [r for r in rows if exact_type(subject.get("type"))(r)]
    if len(matched) < MIN_COHORT:
        matched, pool, widened = [r for r in rows if on_foot(r)], "on-foot", True

    if len(matched) < MIN_COHORT:
        return {"ok": False, "n": len(matched), "pool": pool,
                "excluded": excluded, "widened": widened}

    return {
        "ok": True,
        "n": len(matched),
        "pool": pool,
        "widened": widened,
        "excluded": excluded,
        "mode": {True: "running", False: "walking"}[mode] if mode is not None else None,
        "median_pace_sec_per_mi": median(r["pace_sec_per_mi"] for r in matched),
        "median_hr": median(r["hr"] for r in matched),
    }

Note the MIN_COHORT floor returning ok: False rather than a median over two rows. Refusing to grade is a legitimate output, and it beats a confident number built on noise.

Say what you filtered

A cohort filter is invisible in its own result. The consumer sees a median and has no way to reconstruct which rows produced it, so if they cross-check against the source app they will get a different number and no explanation. State the yardstick alongside the answer.

# cohort.py, continued

def mmss(seconds):
    return f"{int(seconds) // 60}:{int(seconds) % 60:02d}"


def describe(cohort):
    """The yardstick, in words. A filter you never state is a filter nobody
    can check."""
    if not cohort["ok"]:
        return (f"Not enough comparable activities to grade against "
                f"({cohort['n']} in the {cohort['pool']} pool).")
    line = (f"Compared against {cohort['n']} {cohort['pool']} activities: "
            f"median {mmss(cohort['median_pace_sec_per_mi'])}/mi "
            f"at {cohort['median_hr']:.0f} bpm.")
    if cohort["widened"]:
        line += " Pool widened to all on-foot activities."
    if cohort["excluded"]:
        other = "walking" if cohort["mode"] == "running" else "running"
        n = cohort["excluded"]
        line += (f" {n} same-label {other}-effort "
                 f"{'activity' if n == 1 else 'activities'} excluded; the label "
                 f"says otherwise but the pace does not.")
    return line

Run it

You need a history to compare against. This fixture is fabricated, shaped to match the bimodal split I found in my own data, with everything filed under one label.

# fixture.py
"""A deliberately bimodal history: real runs and walking-pad sessions, all
filed by the device under the same label."""

RUNS = [
    {"id": i, "type": "treadmill_running", "pace_sec_per_mi": p, "hr": h}
    for i, (p, h) in enumerate(
        [(520, 141), (545, 138), (566, 152), (580, 149), (601, 145),
         (612, 158), (628, 143), (640, 172), (655, 136), (668, 147),
         (679, 140), (690, 155), (701, 133), (694, 150), (706, 114),
         (612, 161)], start=1)
]

WALKS = [
    {"id": 100 + i, "type": "treadmill_running", "pace_sec_per_mi": p, "hr": h}
    for i, (p, h) in enumerate(
        [(848, 120), (861, 118), (874, 117), (883, 116), (897, 116),
         (905, 115), (918, 114), (926, 113), (939, 112), (947, 112),
         (955, 111), (963, 110), (972, 109), (980, 108), (994, 107),
         (1008, 106), (1021, 104), (1043, 103), (1067, 101), (1090, 99),
         (1122, 97), (1168, 95), (1210, 92), (1284, 90), (1355, 88),
         (1490, 85), (1666, 83), (2140, 80), (3025, 78), (5060, 76)],
        start=1)
]

HISTORY = RUNS + WALKS

Now grade one hard interval session both ways.

# run_demo.py
from statistics import median

from cohort import describe, exact_type, reference_cohort, mmss
from fixture import HISTORY

# The interval session being graded: 7:12/mi average, 164 bpm.
subject = {"id": 999, "type": "treadmill_running",
           "pace_sec_per_mi": 432, "hr": 164}

naive = [r for r in HISTORY if exact_type(subject["type"])(r)]
print("label-only cohort:  "
      f"n={len(naive)}  median {mmss(median(r['pace_sec_per_mi'] for r in naive))}/mi "
      f"at {median(r['hr'] for r in naive):.0f} bpm")

gated = reference_cohort(subject, HISTORY)
print("measured cohort:    "
      f"n={gated['n']}  median {mmss(gated['median_pace_sec_per_mi'])}/mi "
      f"at {gated['median_hr']:.0f} bpm")
print()
print(describe(gated))

python3 run_demo.py prints:

label-only cohort:  n=46  median 15:22/mi at 114 bpm
measured cohort:    n=16  median 10:34/mi at 146 bpm

Compared against 16 treadmill_running activities: median 10:34/mi at 146 bpm. 30 same-label walking-effort activities excluded; the label says otherwise but the pace does not.

Both cohorts are drawn from the same 46 rows and both return a plausible median. That is the failure mode in one line: a 32 bpm error in the yardstick, with nothing in the output to flag it.

Then pin the behavior. These tests fail if you delete the mode filter, if you reorder the partition and the widening step, or if the tri-state collapses to a boolean.

# test_cohort.py
import pytest

from cohort import is_running_effort, reference_cohort, describe
from fixture import HISTORY, RUNS, WALKS

RUN = {"type": "treadmill_running", "pace_sec_per_mi": 432, "hr": 164}
WALK = {"type": "treadmill_running", "pace_sec_per_mi": 1020, "hr": 100}


@pytest.mark.parametrize("pace,expected", [(432, True), (779, True),
                                           (780, True), (781, False),
                                           (1020, False)])
def test_split_is_on_pace_not_label(pace, expected):
    assert is_running_effort(pace) is expected


@pytest.mark.parametrize("pace", [None, 0, -1])
def test_unusable_pace_is_neither_cohort(pace):
    assert is_running_effort(pace) is None


def test_a_run_is_graded_against_runs_only():
    c = reference_cohort(RUN, HISTORY)
    assert c["n"] == len(RUNS)
    assert c["median_pace_sec_per_mi"] <= 780
    assert c["excluded"] == len(WALKS)


def test_a_walk_is_graded_against_walks_only():
    c = reference_cohort(WALK, HISTORY)
    assert c["n"] == len(WALKS)
    assert c["median_pace_sec_per_mi"] > 780
    assert c["excluded"] == len(RUNS)


def test_the_filter_runs_before_widening():
    # Only three same-label runs, so the cohort widens to on-foot. Widening
    # must not readmit the walks it just excluded.
    thin = RUNS[:3] + [dict(r, type="running") for r in RUNS[3:]] + WALKS
    c = reference_cohort(RUN, thin)
    assert c["widened"] is True
    assert c["median_pace_sec_per_mi"] <= 780


def test_a_paceless_row_joins_neither_cohort():
    ghost = {"id": 500, "type": "treadmill_running",
             "pace_sec_per_mi": None, "hr": 130}
    assert reference_cohort(RUN, HISTORY + [ghost])["n"] == len(RUNS)
    assert reference_cohort(WALK, HISTORY + [ghost])["n"] == len(WALKS)


def test_exclusions_are_disclosed():
    assert "30 same-label walking-effort activities excluded" in describe(
        reference_cohort(RUN, HISTORY))


def test_nothing_is_claimed_when_nothing_was_excluded():
    assert "excluded" not in describe(reference_cohort(RUN, RUNS))

With pytest installed, python3 -m pytest test_cohort.py -q reports 14 passed.

Gotchas

Widening happens after partitioning, or the fix does nothing. The trap is that a thin exact-match cohort falls back to a broader class, and the broader class is where the mislabeled rows live. The symptom is encouraging: your cohort size jumps from 3 to 40 and the median moves the wrong way. The escape is the ordering in reference_cohort, and test_the_filter_runs_before_widening is there because that ordering is invisible to a reader skimming the function.

A boolean predicate picks a side for your missing data. If is_running_effort returned False on a null pace, every row with no pace would join the walking cohort and drag its median down, and nothing would look wrong. Return the third state and filter with is.

Only count exclusions the cohort could have drawn from. The trap is reporting every row the mode filter dropped, which includes activities honestly typed walking that were never candidates for a running cohort in the first place. The symptom is a disclosure line that overstates its own case: it claims those rows share a label with the subject, and for some of them that is simply false. Intersect the exclusion count with the label filter that would otherwise have applied.

Cleaning the cohort moves every constant that was calibrated against the dirty one. My heart-rate grading bands had been tuned the day before, against the contaminated distribution, so I had to re-derive it before trusting them. Excluding the walking-pad sessions moved my treadmill median heart rate from 116 to 145, which lines up with the outdoor median of 144.5 that was never contaminated in the first place. The bands survived unchanged, but only because I checked rather than assumed. Any threshold downstream of a cohort is a threshold you re-verify after fixing the cohort.

Pick the boundary from the gap, not from a round number. Sort your distribution and look for the empty band between the two modes. Mine ran from 11:46 to 14:08 per mile, so 13:00 has real margin on both sides and no row sits close enough to flip on measurement noise. If there is no gap, the two populations are not separable on that variable and you need a different one.

Sources

Changelog

  • release: 0.26.0 — report-card rubric fixes, model right-sizing, and a per-tool MCP reference (dev → main) (#135) (d60fa18)