Nate Swenson

When the perf gate measures the hardware instead of the code

Local Fitness · No. 070 · 6 min read ·v0.38.1

Shipped

This release promoted a four-batch improvement pass over my fitness agent: accuracy fixes, a speed pass, tool-surface UX, and maintainability work, versions 0.35.0 through 0.38.1. Midway through, the CI perf gate failed a PR whose hot path I had measured locally at parity with the old code. Chasing that failure produced the most reusable lesson of the whole pass: a benchmark baseline captured on shared CI runners has a shelf life, and you need an evidence protocol for telling code regressions from hardware drift. This guide builds the gate, then the protocol.

The gate: pytest-benchmark with a committed baseline

The pieces: benchmarks that are skipped in ordinary test runs, a CI step that opts back in and compares against a baseline JSON committed to the repo, and a threshold that fails the build.

# pyproject.toml
[tool.pytest.ini_options]
addopts = "--benchmark-skip"

# tests/test_perf_benchmarks.py
from myapp import assemble_context  # the hot path you care about

def test_bench_assemble_context(benchmark, perf_db):
    result = benchmark(assemble_context, perf_db)
    assert result is not None  # keep a real assertion; a benchmark is still a test

perf_db is your fixture returning a seeded database handle; build it from fabricated data, never from anything real. The CI step opts in and compares:

- name: Perf-benchmark regression gate
  run: |
    uv run pytest tests/test_perf_benchmarks.py --benchmark-only \
      --benchmark-storage=file://./.benchmarks --benchmark-autosave \
      --benchmark-compare=0001 --benchmark-compare-fail=min:15% --no-cov

Per the pytest-benchmark comparing docs, --benchmark-compare=0001 selects the saved run by its numeric prefix, and --benchmark-compare-fail=min:15% fails the job when a test’s minimum time degrades more than 15% against it. I gate on min, not mean: the minimum is the least noisy statistic on a shared machine, since noise only ever adds time.

One structural detail decides whether this works at all. pytest-benchmark nests saves under a machine id like .benchmarks/Linux-CPython-3.12-64bit/0001_<hash>_<timestamp>.json. The committed baseline must live under the id CI will compute, which means it must be captured on CI hardware. The gotchas cover the two ways I got this wrong before writing it down.

Capture the baseline where the gate runs

Make baseline capture a manually triggered workflow on the same runner image, and pull the result out as an artifact:

# .github/workflows/capture-perf-baseline.yml
on:
  workflow_dispatch: {}
permissions:
  contents: read
jobs:
  capture:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v7
      - run: uv python install 3.12 && uv sync --dev
      - name: Capture benchmark baseline
        run: |
          uv run pytest tests/test_perf_benchmarks.py --benchmark-only \
            --benchmark-storage=file://./.benchmarks --benchmark-autosave --no-cov
      - uses: actions/upload-artifact@v7
        with:
          name: perf-baseline
          path: .benchmarks/

Download the artifact, rename the newest save to the 0001_ prefix your gate compares against, commit it. Dispatching on your default branch also captures pre-change code, which matters when you rebaseline mid-investigation: the honest floor is the code before your PR, on today’s hardware.

The drift, and the evidence test

During the speed batch, the gate failed one benchmark at +18% against a baseline captured seventeen days earlier. My local before/after measurement of the same path showed +1.9%. I optimized that real 1.9% away (the fix was pushing a filter into SQL); the gate then failed at +15.6%. At parity locally, still failing in CI.

The control experiment settled it: the last passing run of the same gate on completely unchanged code was already reading +13.7% against that baseline. GitHub hosts these runners as virtual machines in Microsoft Azure, and the fleet under a fixed machine-id string is not one machine; my failing run and my baseline were backed by different CPU models with different cache geometry. The Criterion FAQ says this plainly for its own ecosystem: cloud CI virtualization “introduces a great deal of noise into the benchmarking process”, enough to show large apparent changes when the code hasn’t changed.

So the protocol, before you either “fix” phantom slowness or quietly raise the threshold. Call drift only when both hold:

  1. A local before/after A/B of the exact flagged path is at or near parity. Run the benchmark on the base commit, then on your branch, same machine, same session.
  2. The last passing CI run on unchanged code already shows a double-digit margin against the baseline. Pull its report from the job logs; pytest-benchmark prints the comparison table.

Both true: recapture the baseline from pre-change code on current runners and document why. Either false: believe the gate and go find your regression. Mine, at +1.9% real, was worth fixing anyway.

Verify your gate end to end

After committing a baseline, prove the loop closes. Push a branch with a deliberate slowdown in one benchmarked path:

def assemble_context(db):
    import time
    time.sleep(0.002)  # temporary: ~3x this path's budget
    ...

The gate job should fail with a line like:

test_bench_assemble_context (0001_ab12cd3) - Field 'min' has failed
PercentageRegressionCheck: 214.3 > 15.0

Revert the sleep, and the job should go green on the same runner class. If instead you see the comparison silently not happen, check the machine-id directory name in .benchmarks/ against what the CI job computes; a mismatch means the gate is comparing against nothing.

Gotchas

A locally captured baseline turns the gate into a permanent failure. pytest-benchmark keys saves by platform: my Mac writes Darwin-CPython-3.12-64bit, CI looks in Linux-CPython-3.12-64bit. Capture locally and the comparison target never exists on CI. The escape is the dispatch workflow above; the rule in my repo docs is capture on CI, never on the laptop.

An emulated container matches the machine-id string and still poisons the baseline. Before the workflow existed, I tried capturing via docker run --platform=linux/amd64 python:3.12-slim on Apple Silicon. The id string matched perfectly. The emulated CPU was far slower than real CI metal, so the committed floor was garbage and the 15% gate blew on pure hardware noise. String equality is necessary, not sufficient; the baseline has to come from the hardware class the gate runs on.

Chasing the percentage across runner draws wastes a day. Removing my real +1.9% moved the CI reading only from +18% to +15.6%; the other 2.4 points of gap between what I fixed and what CI reported were the fleet, since each run lands on whatever VM it hands out. Any single run’s percentage is one sample from a noisy distribution. That’s why the protocol above wants the unchanged-code control, not another re-run of your branch.

workflow_dispatch can’t bootstrap itself. A dispatch-only workflow becomes triggerable via the API only once the workflow file exists on the repo’s default branch. If your default branch only receives deliberate promotions, the first baseline can’t come from the workflow at all; mine came from a real CI run’s autosaved artifact, hand-promoted to the 0001 slot. Plan the first capture separately from the steady-state one.

Sources

Changelog

  • release: 0.38.1 — four-batch audit pass: accuracy, speed, UX, maintainability (dev → main) (#159) (31b13f0)