Field notes · No. 138
Senior DevOps Engineer building in public. Shipped features across Local Fitness, Ghostwriter, and more — the tradeoffs behind them, and what broke.
A read-back loop that polls until a field changes has a blind spot: a command whose target state was already true can never change anything, so the loop sits out its whole ceiling for nothing. Here is the two-line fix and the verdict it forces.
Network commands to a TV fail silently all the time. Here is how to build a controller that never reports a command as done until the device's own state agrees, and what to do when the device has no state to read.
Every post my LinkedIn skill published stalled under 300 impressions, and the only way to prove it was a log a human fills in by hand. How to build a publish log, an outcome CLI with numeric fields, and the listing flag that keeps the data complete.
An agent skill spent a minute of every run hand-transcribing API responses into its own JSON, and that transcription was the main way data got mangled. The fix is a contract: raw tool output hits disk verbatim, and one deterministic ingest step does all the reshaping.
A dispatched background stage ran for minutes with no clock, no position, and no expectation. The fix was not a heartbeat protocol; every signal the user needed was already on disk, in the mtimes of files the worker writes anyway.
A gate that parses test-runner output worked in CI and refused every real run on my laptop, because Node changed its default reporter out from under it. The fix is a parser that accepts every summary the runner actually emits.
An allowlist entry that can never match anything is worse than no entry: it reports success and changes nothing. Here is how a text-prefix IPv6 rule becomes exactly that, and the two fixes that closed it.
package-lock.json carries your package's own version number in two places, and nothing in the standard toolchain fails when they fall behind. In one monorepo, five packages drifted before anyone noticed.
You can see every outbound connection your machine holds, per process, with no sudo and no packet capture. The trick is deciding up front that your tool reports facts and refuses verdicts.
A discovery pipeline that relies on web search alone will report walled accounts as nonexistent. Probe handles by URL, give every probe three possible outcomes instead of two, and put the checklist in code so you can prove it ran.
A generator that proposes additions to a user-owned rule set has to subtract what those rules already claim, and the dangerous overlap is not the duplicate. Building the claimed-set filter, and reporting what it excludes.
A publish approval is only consent if the full artifact is visible at the moment of the decision. The one-line fix, and the CI invariant that keeps the rule from being edited away later.
A voice system built from accumulated bans produces drafts that violate no rule and still read like incident reports. Why negative style specs bottom out at neutral, and how to write the positive register contract that gets past it.
A groundedness checker that verifies numbers by string matching has to read text the way it renders. Two false-positive classes from a live audit, the typographic minus and the calendar date, and the offset-safe fixes for both.
A metric that gets revised through the day is not the same as a metric that accumulates, and a verdict computed on an unrevised value is wrong even though the sync succeeded. How to give settling metrics their own contract.
Every tool your MCP server exposes is prompt the model pays for in every session, and every verbose payload is context it pays for on every call. How to find the expensive tools by recording real traffic, and the cuts that followed on a live server.
An upsert loop can add and correct, but it has no way to say a thing should no longer exist. Here is how to build a one-way sync that deletes safely, converges to zero writes, and never overrules the person on the other end.
A lookup that always returns its best match will answer questions it has no material for. The floor that prevents it only works if nothing is allowed to lift a result over it.
Python's SMTP client encrypts by default and authenticates nothing. Here is how to build a mailer that actually verifies the server, and a test that fails when the verification is off.
An authorisation check that keys on the object alone answers a question nobody asked. Same item, same approved plan, and the verb decides whether the outcome is filing or destruction.
A guard on an irreversible bulk action should over-match on purpose, because a false positive and a false negative are not two versions of the same mistake.
A scoring signal built from a threshold stops carrying information the moment everything clears it. The ranking still returns a list; it just is not ranked any more.
A generated card read its version, date and install steps off the published release and got all of them right. The one figure a person copied from a design doc was stale by half.
Swapping a five-bucket grade for a continuous score usually means recalibrating from scratch. It doesn't have to, if the new curve's knots are the old rubric's boundaries.
A gate that checks whether a claim can be backed up will happily pass a claim about somebody else's work. Provenance is a second question, and it needs its own filter.
Ordering steps by their position in a list is not the same as ordering them by what they need. One measured run spent an hour at a gate enforcing a dependency that did not exist.
A dependency fixed a bug by refusing to act. Nothing in its response changed, so the only way to know you have the fix is to check the version, and to check it yourself.
A scaffolder kept emitting a CI trigger the project had banned, because the ban was enforced on files people write and generated files skip review entirely.
A release tool read three version facts, reported one state, and its fast path acted on that state alone. The fix was to stop collapsing the facts and to decide the irreversible target exactly once.
A rule-based checker with a green test suite told me its own repo was violating a rule that repo mandates. Here is how to score a checker against real runs, per severity and per rule, so the number that matters cannot hide inside an average.
A path-based auth gate is only as trustworthy as the string you hand it. Here is how to build one on the ASGI scope, and how to prove a crafted Host header cannot move it.
A scoring system can pass every unit test it has and still be wrong, because unit tests check the arithmetic and never the answer. Regrading real history and gating on the shape of the letter distribution catches the failure the tests structurally cannot.
A staged agent pipeline where the rule that stops a stage running early is a function, not a sentence in a prompt. Here is how to build the gate, and the four refusals that make it worth having.
Update commands exit 0 whether or not anything moved, so a successful update and a silent no-op are indistinguishable. The fix is to read the version back off disk after every write and report what did not move as loudly as what failed.
Every release flow rests on one question: what has landed since the last tag. In CI that question is usually asked of a clone that cannot answer it, and git answers anyway. Here is how to make the answer either right or refused.
If your tool writes GitHub Actions YAML from a config file, you have built a code generator whose output runs with a privileged token. Here is how to validate every substituted value against the grammar it actually lands in, and prove it from both sides.
Rename a generator and every artifact it ever emitted keeps signing itself with the old identity, in repositories you do not control. Putting a version and a content hash in the marker is what turns that from silent rot into a list you can hand someone.
A golden file is only worth what its refresh command is worth. If running the refresh on an unchanged tree produces a different file, the golden is pinned to nothing and every diff reads as drift.
Three tools shipped under metaphor names and all three had to be renamed. This turns 'the name must say what it does' into a check that runs before the name is spent, using a word-break over the description the author already wrote.
Generating a banner into a README looks like string concatenation until markdown reinterprets it. One missing blank line turns the whole thing into a heading, and only a rendered page would ever show you.
When generated output contains the version of the generator that produced it, a byte comparison can no longer tell you whether anything really changed. Every release looks like a content change, and the label that was supposed to mean something stops meaning it.
A generator that splices a block into someone else's file has one hard requirement: the file must mean exactly what it meant before. Here is how to emit a comment banner into a YAML workflow and prove, without a YAML parser, that it changed nothing.
A fan-out bot that opens a pull request per release, and never closes the last one, does not merely leave clutter. The older pull requests fail their own checks by construction, and a bot with permanently red pull requests teaches everyone to ignore it.
A tool that writes generated blocks into other repositories has to decide which of its targets apply to the checkout it is standing in. File presence answers that badly the moment a target's path is a common filename, and the fix is to ask the checkout what repository it is.
An update bot that only acts when the generated values change leaves every consumer carrying several disagreeing version numbers. Open a pull request every release instead, and put the difference between a real change and routine adoption in the title.
Automation that clones other people's repositories has to pick a branch, and both obvious choices are wrong. A shallow clone cannot see the branch the work is on, and the default branch is not the branch teams integrate into.
A workflow triggered by a tag that another workflow created never runs, and nothing anywhere reports an error. Here is how to spot a dead trigger and how to chain the two workflows so the second one actually fires.
A release that adds something consumers must have will fail every consumer the moment they upgrade, unless the same change that raises their version also supplies the thing. Ship the requirement and its fulfillment together, and require consent before creating state.
A fan-out that pushes to several repositories has two ways to lie: it can fail to authenticate while every other step still works, and it can count those failures and exit zero anyway. Both shipped here.
Byte comparison is the cleanest way to prove a refactor changed nothing, until the build embeds something that differs every run. Canonicalize the volatile field instead of deleting it, and keep a control that proves the check can still fail.
A pinned dependency check proves a consumer is intact. It can never prove the consumer is current, because it passes forever against the version it was pinned to. Push freshness from the source instead of waiting for each repo to notice.
A font-family list that resolves perfectly in Chromium can silently pick a different face in a PDF renderer, because one engine never reaches the fallbacks and the other walks them for real. Predict which face wins, then measure the file.
Before moving a product onto a shared token set, read what it already ships and diff it against what the system can actually express. The values that fail that diff are the ones a migration quietly loses.
A shared value copied into eight files is eight values that happen to agree today. Here's how to generate a marked region inside each consumer instead, and gate it in CI so drift fails the build.
Long imports get interrupted. Instead of writing a checkpoint file that can disagree with your data, derive the resume point from the run rows you already persist, and let a killed job pick up exactly where it stopped.
Secret scanners catch tokens and keys. They do not catch the amounts, names, and place-encoded labels that end up in your fixtures. Here is how to find those, replace them safely, and verify the bytes you actually committed.
If two axes of a weighted score are computed from the same underlying input, one of them will punish whatever the other rewards. Here is how to detect that with a sweep, and why the fix is removing the axis rather than lowering its weight.
A generated PDF can look flawless and still be unreadable to the software that reads it first. Here is how to render one, read the text back out, find the exact CSS value where your own stack breaks, and turn that into a test that can actually fail.
A default in an agent's instruction file is not what one rule says; it is what every mention of the behavior adds up to. Here is how to find all of them and keep the count from growing back.
Your write succeeded, your build succeeded, and the page still 404s. Here is how to check that the content you published actually became a page, and that the page actually serves, before you call it published.
Some APIs answer a malformed question with a plausible number instead of an error. Here's how to pin every query as a tested record so a wrong one can't ship.
A presigned upload signed with an empty Content-Type rejects both the obvious header and no header at all. The fix is sending the empty string on purpose.
Every generated image passed review on its own and the set still looked like one template. Here is how to measure a property that no single artifact can violate, and gate on it.
npm always ships your README, right up until your repo layout puts it one level above the package directory. Here is how to stage docs at pack time, and the part of this that I could not fix.
A regression gate built from work you already shipped is only worth having if you prove it can fail. Here is how to build one, and how to check it isn't quietly asserting nothing.
A grounding check that flags almost everything is the same as one that flags nothing. Here's how to build four synthetic arms with known answers and prove your detector can actually tell them apart.
X's 280-character limit isn't Python's len(). Some characters cost 2, a URL costs a flat 23 no matter how long it is, and getting the overlap logic wrong will silently mis-count a real post.
A CI benchmark gate compares every PR against a committed baseline. That works until the runner fleet under it drifts. Here's the gate, the drift, and the evidence test that tells a real regression from a hardware mirage.
Capping an agent's memory means archiving entries out of the prompt. This release added the other half: full-text recall over the archive with SQLite FTS5, no search service required.
My fitness coach now remembers streaks, repeat excuses, and its own past reads. The hard part wasn't storing memories; it was injecting them into prompts that are cache-keyed on their own bytes.
A rule in the system prompt cannot stop a model from repeating a token that is sitting in its context. Take the token out of the context, carry the meaning in a form the model is free to use, and keep a validated detector as a backstop.
A generated PDF that overflows to a second page is a layout you never measured. Render it, count the pages, and step down a density ladder until it fits, then drop content only as a last resort and say what you dropped.
Any feature that scores something against 'similar' records has to decide what similar means, and a category label is the tempting wrong answer. Here is how to build a reference cohort that partitions on a measurement instead, and discloses what it threw out.
A performance audit of this site found the largest image on every page was lazy-loaded, and my first WebP 'optimization' produced files bigger than the PNGs they replaced. Here is the build-time image pipeline that fixed both, with the measurements that decided it.
Any system that grades a user against their own history hits the same wall: the naive rubric punishes the behavior you asked for. Four design decisions fix it, and a real graded report card shows what each one is worth.
If your app generates both PDFs and chart images, the colors live in two places and drift apart. Put them in one token dict, let a local JSON file override any subset of it, and make a broken override impossible to notice.
A streaming model call that stops delivering messages but never closes will hang your job with no error at all. Here is how to bound it on the gap between messages, retry it, and log the failure so you diagnose the right thing.
Three releases of my LinkedIn ghostwriter went to interaction cost: one dialog instead of a chain of questions, silent setup instead of narrated bookkeeping, and generated visuals grounded in real command output after an invented one cost nine review rounds.
Version 0.11.0 of this skill came entirely out of one exercise: asking the model to review the skill's own previous six runs. Here is the four-lens audit loop that produced it, and what the round cost in tokens.
Moving generated content into a subfolder of the site repo turns every publish push into a deploy. The catch: your publisher's already-published check is full of path assumptions, and missing one silently re-plans your entire archive.
Every card my LinkedIn ghostwriter generated had the same skeleton, because a template plus an LLM equals the same output forever. The fix was a component vocabulary, a card-history file, and a rule that each new card must differ from the last three on at least two axes.
Style-guide prose can ask an agent to draw inside a box; it can't make it happen. How to measure the rendered DOM in headless Chromium and refuse to rasterize any composition that breaks the layout contract.
How to measure a rendered HTML page for overflow and truncation before a human ever sees it, and the CSS trick that lets one layout adapt itself to however much content lands in it.
A pattern for agentic pipelines that generate documents: make every fact either cited against something the code fetched, or explicitly marked unavailable, so the model physically cannot ship an invented number.
devlog v0.8.1 fixed bland, repetitive cover images, but the fix that stuck wasn't in the render code. It was a line added to the skill's own instructions and a machine-checked invariant, which is what let one Claude Code session regenerate all 49 backfilled covers in parallel and get them right the second time.
My dev log grew past the point where scrolling found anything, so I wired the topic tags I was already generating for every post into a live, keyboard-accessible tag search — and hit two classic 'the whole card is a link' accessibility traps along the way.
A subscription can vanish from a recurring-charges report simply because the bank changed how it labels the charge. The fix wasn't new logic, it was reading a canonical key that already existed.
My analysis tools were handing the LLM raw numbers and a legend, then trusting it to classify. Here is how I pulled every judgment into a pure, tested Python module so the model only has to write the sentence.
I rewrote 38 blog posts against a writing contract and watched one rule get skipped over and over: every source in the bibliography must also be cited in the body. The rule was written down; nothing enforced it. Here's how to promote a style-guide sentence into a deterministic check your build can fail.
My dev-log feed sorted newest-first by date, which worked until the day I cut three releases before dinner and the newest one rendered at the bottom. The fix is a two-key comparator: date first, version as the tie-break, with the version segments compared as numbers. Here's how to build one that survives v0.10.0.
Two sibling packages in the same repo both got a patch release from the exact same commit. Here's the git-tag-plus-path-filter technique that tells you which packages actually need a release when one commit touches all of them.
A local MCP server turns your bank statements into tools an AI agent can call. Here is how to build one where the model gets answers, never raw account numbers.
Ghostwriter's voice profile and credentials lived inside its own repo, so a dev checkout and a marketplace-installed copy each needed their own setup. Moving personal config to the home directory collapses that back to one source of truth.
Ghostwriter moved from a manual symlink install to a real Claude Code plugin, which meant learning the marketplace.json/plugin.json schema and one directory-nesting rule that Claude Desktop enforces and the CLI quietly didn't.
A rendered card and its stylesheet are two files that only agree by convention, nothing in HTML or CSS enforces it. Here's how to write a fast, deterministic test that catches the drift before a render silently ships unstyled.
Every npx invocation of my CLI exited 0 with no output, while running the file directly worked fine. The bug was one strict-equality check that can't survive npm's bin symlink. Here's how to detect 'am I the main module' correctly in an ESM CLI.
I rebuilt this dev log's skill so the LLM stops running twenty hand-rolled bash steps and starts consuming one JSON plan from a tested CLI. Here's the pattern: put everything deterministic in code you can test, and put everything non-deterministic behind a lint gate and a budget-capped judge.
devlog's voice-profile fallback pointed at a path its sibling tool never actually used, so the fallback could never fire and nothing ever errored. A small patch, and a useful lesson about tools that share files by convention.
A plugin marketplace's skill-discovery rule lives entirely in a directory shape, not a manifest field, so it has to be built, lint-checked, and released against that same shape. Here's how to do all three without a CI job that lies to you.
The resume skill's 1.0.0 deleted its own subprocess LLM call and let the invoking agent do the tailoring directly. The part worth stealing is what replaced the trust that used to live in that subprocess: a deterministic checkpoint plus a budget-capped judge, and the shape you can build for any skill that mixes judgment with facts.
One of my fitness agent's MCP tools was opening nine SQLite connections to answer a single question. The fix was routine; this walkthrough builds the part that outlives it, a pytest-benchmark harness that asserts connection counts, compares against a committed baseline, and fails any PR that regresses either.
My fitness agent's daily PDF gained a Claude-written coaching line, which put a network call in the middle of a synchronous render. Here's how to pair that call with a deterministic fallback, and how to prove the fallback works before the day you need it.
Adding an opt-in escape hatch to a domain blocklist looks like a one-line change until you ask what happens the day someone else adds a second domain to that same list.
My fitness agent could read the database but never freshen it. Here is how to add a side-effectful sync tool to an MCP server, keep it gap-aware, and keep it out of the loops that should stay read-only.
The same sync code got rate-limited on the host and ran clean in the container, and the difference was an env-var fallback buried in a client library. How to make session caching explicit and pin it with a test so it can't silently regress.
Ghostwriter's post images moved from a dark look to a light one by changing a single CSS class. Here's how to build a token-driven theme switch, render it to a deterministic PNG, and catch the one missing token that makes text disappear.
An agent that writes a confident claim will just as confidently skip checking it. Version 0.7.0 adds a code gate that blocks publishing until every outside-world claim traces to real, live sources, and it has to survive garbage input and a harness that lies about testing it.
devlog 0.4.0 rewrote the instructions behind /devlog so every release post has to become a researched, cited implementation guide. The instructions alone don't guarantee it happens, so here's how to turn a soft citation ask into a structured contract and a checker that catches it when the model doesn't comply.
When one prompt is doing eleven jobs, most of them have a right answer and don't need a model. Here is how to split an LLM feature into a tested planner, one toolless model call, and an advisory grounding check, then prove the cutover is safe before you flip it.
Before making a repo public, wire in the two security workflows GitHub gives you free: CodeQL for the code you wrote, dependency review for the code you pull in. Setup, verification with real output, and the traps that make both silently useless.
A squash-merged dev-to-main promotion leaves the two branches diverged, so the next promotion PR shows commits that already shipped. Here's a script and workflow that resets the branch automatically, including the branch-protection dance it has to do safely.
This release deleted a pile of confirmed-dead code and raised test coverage on the parts that were genuinely thin. The useful part wasn't the percentage, it was catching the tests that were only pretending to check something.
How to wire a GitHub release off a green CI run instead of a push, guard it so an already-shipped version can't double-release, and the default-branch trap that catches anyone wiring workflow_run for the first time.
Claude Code's own docs say its session transcripts are an internal format that can change on any release, yet discovering what you shipped recently has no sanctioned API. Here is how to read the logs anyway, defensively enough to survive the warning.
Ghostwriter's new matrix card type shipped as one new file, with no registry code touched. Here's how a naming convention can stand in for a plugin registry, and the test that keeps it from shipping half-wired or quietly duplicated.
Five coaching preferences were hardcoded into a public repo as if they were universal truths. Turning them into user settings meant building a small resolver: one precedence order across three config layers, where every bad input falls back to the value that already worked instead of raising.
A training-plan feature graded today's finished workout as still pending, because the code asked the calendar instead of the outcome. Here's how to build a status grader that reads outcomes first, holds judgment only where the day is genuinely unsettled, and hands the UI one verdict it never has to second-guess.
An MCP server can hand its client a system-prompt hint at initialize. This build wires that field to a SQLite setting, resolves it fresh on every connect, and fails open when the read misses, so a config change lands without a restart and a fresh clone still starts.
My fitness coach's brief had one hardcoded voice. Making it four selectable tones was the easy part; the real work was making sure a numeric 'harshness' dial did something a test could catch, not just something the model might notice.
My fitness agent's plan tool needed to tell the difference between a workout that failed and one my Garmin sync simply hasn't caught up to yet. Here's how to build a single agent tool that grades real-world state against a data freshness boundary instead of the calendar.
Ghostwriter's image cards got pulled onto one shared base and grew a portrait carousel with a progress bar. The mechanism behind it, CSS custom properties feeding a headless-browser render pipeline, is a pattern you can reuse for any HTML-to-image or HTML-to-PDF generator.
The resume skill's v0.2.0 benchmark scores its output with two LLM judges that run through the Claude CLI instead of the metered API, for zero marginal cost. Making that safe took a timeout that actually kills the child process, a wrapper that fails open instead of crashing the build, and a hard line between what the judge gets to block and what it only gets to report.
A Claude-generated daily fitness brief went from about four minutes to under 90 seconds, but not from the parallel rewrite I designed. Here's how to gate a fix behind a kill criterion, find out which agent settings are real versus silently inert, and confirm the trade with a blind judge instead of a hunch.
devlog's v0.3.1 test suite needed bin/devlog.js's logic importable without also running its command-line dispatch. Here's the guard pattern that split the two, and the symlink trap that broke it again a few weeks later.
devlog switched from a daily commit log to one entry per release tag. Building that meant treating every tag name as untrusted input, picking the right previous release even when prereleases are mixed in, and making the whole run safe to repeat.
Once an MCP server can write back as well as read, the model has no reason to run inside it anymore. Here's how to build that write tool, and the exact way a 'read-only' guard fails if you enforce it with a keyword check instead of at the connection itself.
I exposed local-fitness's in-process Claude agent tools to real MCP clients over HTTP and stdio without writing a second copy of them, then closed the three gaps a network-reachable localhost server actually has that a subprocess-launched one doesn't.
I gave an AI agent its first real database write path by making the dangerous action unreachable instead of trusting it to behave: the tool schema never accepts a status field, and a partial unique index backstops the single-active-row rule even if the application code has a bug. Here's how to build the same draft-only boundary for any agent that writes to your own data.
Porting a tool that reads live data means your old and new versions can both be correct and still print different numbers. A parity harness that compares them needs a tolerance and a declared list of fields allowed to differ.
ghostwriter v0.3.0 added carousel support: a headless-browser render step that turns HTML slides into a single PDF, and a publisher path that uploads it through a document API LinkedIn renders as a swipeable carousel. Here's the render-and-ship pipeline, built generic enough for any chunked document upload.
Ghostwriter's 0.1.0 release added one new card type: three ascending bars for an accelerating progression, with the bar heights fixed for effect rather than computed from the numbers next to them. Building it is a clean example of a small, reusable system: a CSS brand guide, a headless-browser render step, and a verification step that checks the honesty claim is actually true.
v0.2.0 let one devlog config serve several projects out of a shared monorepo. Getting there safely is a short lesson in two habits: scoping git history correctly, and treating any value headed for a shell command as input you don't trust yet.
A résumé-tailoring skill that shells out to the claude CLI was burning 11-15k thinking tokens on an 800-token JSON answer. The fix was one environment variable, a validation step that moved out of the prompt and into code, and a sweep that proved the fix instead of assuming it.
I turned a personal Garmin-coaching script into real software: a pytest suite behind a coverage gate, CI wiring, and a scorer that cross-validates the agent's prompt against the typed contract its own output has to satisfy. Here's how to build that check for any prompt-driven agent.
The ghostwriter skill's first versioned release made a markdown instruction file gradeable by CI and pushed its automation scripts to 100% line coverage. The two techniques worth stealing: a pass/fail scorer over frontmatter and body text, and monkeypatching a dependency your code doesn't even import until the function runs.
Ghostwriter can now attach a locally rendered diagram or card to a post, and the two techniques worth taking are waiting on a real completion signal instead of network idleness, and keeping personal styling in a gitignored file a stranger's clone never has to see.
The drop-in React component that renders a dev-log feed toggled each entry with a plain onClick, so every site that copied it inherited a feed keyboard and screen-reader users couldn't operate. Here's how to rebuild that toggle as a real ARIA disclosure control, and the test that proves the keyboard path works.
Making a working personal tool installable by a stranger means declaring a real executable, prompting for defaults instead of demanding a config edit, and verifying the tarball before you trust a publish. Here's what that took, including the two ways npm quietly rejected the first attempt.
