Move the deterministic half of your LLM prompt into code

Shipped

local-fitness v0.15.1 cut its daily-brief composer over to V2. V1 was a single agent run with an 18,000-character prompt doing roughly eleven jobs, tools attached, up to 20 turns. V2 is a tested planner in code that assembles a typed context, one single-turn toolless model call that writes the prose, and an advisory grounding signal that watches for invented numbers. The release also gave the agent a scoped tool to re-prescribe single days on an active training plan, but the part worth a walkthrough is the split itself, because it applies to any LLM feature whose prompt has grown into a monolith.

Sort the prompt’s jobs before touching anything

Audit what your big prompt actually asks the model to do, job by job. In my case the audit found that 40 to 50 percent of the prompt was deterministic work dressed up as instructions: evaluate thresholds, rank what matters, pick which context to fetch, assign a chart to a metric. Every one of those has a right answer, which means every one of them can be a tested function instead of a paragraph the model hopefully follows.

This is the same conclusion Anthropic reached in Building effective agents: they recommend “finding the simplest solution possible”, and they distinguish workflows, where LLMs run through predefined code paths, from agents that direct their own process. A daily summary over structured data is a workflow. The model earns its keep on exactly two jobs, selecting which of the candidate stories to tell and writing prose a human wants to read. Everything else moves to code.

The payoff at the end of this release: the system prompt shrank by about 57 percent, the run went from up to 20 tool-calling turns to one toolless turn, and the logic that used to live in prose is now ordinary code with ordinary tests.

Give the two halves a typed contract

The planner and the generator need a boundary, and the boundary is a schema. Pydantic models work well here because they parse untrusted data and guarantee the result conforms to the declared types, which you want on both sides: the planner’s output is validated before it reaches the prompt, and the model’s JSON is validated on the way back.

Two design points matter more than the field names. First, every number the generator may cite is carried in the context, paired with the exact string the prose should render, so grounding has a closed set to check against. Second, the tone is a suggestion, not a command; the model may override it when the prose demands.

# context.py
from typing import Literal
from pydantic import BaseModel, Field

Tone = Literal["positive", "caution", "critical", "neutral"]

class GroundedValue(BaseModel):
    name: str          # "p95_ms"
    value: float
    display: str       # "480 ms": exactly how the prose should render it

class Candidate(BaseModel):
    category: str                 # what this card is about
    fired_triggers: list[str]     # which rules put it here, by name
    metrics: list[GroundedValue]  # the only numbers the model may cite
    suggested_tone: Tone          # advisory prior, not a command
    evidence: str                 # one line: why this candidate fired

class BriefContext(BaseModel):
    date: str
    candidates: list[Candidate] = Field(default_factory=list)

Write the triggers as boring functions

The planner is the deterministic half: it reads your data, evaluates named thresholds, and over-generates ranked candidates for the model to choose from. The example below uses service metrics; swap in whatever your domain measures. The properties to preserve are that it takes no clock and does no hidden I/O, so the same input always produces the same context, and that thresholds live in one named block instead of scattered through prompt prose.

# planner.py
from context import BriefContext, Candidate, GroundedValue

# Thresholds live HERE, named, in one block. Tune them in code review,
# not by rewording a paragraph of prompt prose.
ERROR_RATE_CAUTION = 1.0     # percent
P95_OVER_BASELINE = 1.2      # 20% over baseline is a problem

def _gv(name: str, value: float, display: str) -> GroundedValue:
    return GroundedValue(name=name, value=value, display=display)

def assemble_context(date: str, m: dict) -> BriefContext:
    """Deterministic: same metrics in, same context out. No clock, no I/O."""
    candidates = []
    if m["error_rate"] > ERROR_RATE_CAUTION:
        candidates.append(Candidate(
            category="errors",
            fired_triggers=["error_rate_above_1pct"],
            metrics=[_gv("error_rate", m["error_rate"], f"{m['error_rate']}%"),
                     _gv("error_baseline", m["error_baseline"], f"{m['error_baseline']}%")],
            suggested_tone="caution",
            evidence=f"error rate {m['error_rate']}% vs {m['error_baseline']}% baseline"))
    if m["p95_ms"] > P95_OVER_BASELINE * m["p95_baseline_ms"]:
        candidates.append(Candidate(
            category="latency",
            fired_triggers=["p95_20pct_over_baseline"],
            metrics=[_gv("p95_ms", m["p95_ms"], f"{m['p95_ms']} ms"),
                     _gv("p95_baseline_ms", m["p95_baseline_ms"], f"{m['p95_baseline_ms']} ms")],
            suggested_tone="critical",
            evidence="p95 latency is 20%+ over its baseline"))
    return BriefContext(date=date, candidates=candidates)

In the real release this file absorbed every trigger predicate the old prompt described in words, plus a fixed priority ranking and the tone heuristics, and each predicate got a test asserting it fires exactly on its documented condition. That is the trade you are making: prompt paragraphs you can only eyeball become functions you can cover.

One toolless call, measured on the way out

With the planner carrying the full data payload, the generator no longer needs tools. It gets the context, selects candidates, and writes, all in a single turn with no tool round trips.

# generator.py
from context import BriefContext

def build_prompt(ctx: BriefContext) -> str:
    return (
        "Write today's brief as 1-3 short takeaways, one per candidate below.\n"
        "Cite ONLY numbers present in the context; never derive or estimate "
        "new ones. If a candidate has few numbers, say the data is thin "
        "instead of inventing a figure.\n\n"
        f"{ctx.model_dump_json(indent=2)}"
    )

# Your one model call goes here: single turn, no tools.
# prose = llm(build_prompt(ctx))

Toolless buys you something subtle: because the context is the model’s only data source, you can measure whether the prose invented numbers. Build the pool of citable numbers from the context, then match each number in the prose against its nearest known value by relative distance. Exact match is a faithful citation. Near a known metric but unequal is a likely corrupted value, so flag it. Far from everything is an unrelated quantity, like a request count or a prescribed duration, so ignore it.

# grounding.py
import re
from context import BriefContext

_NUM = re.compile(r"[-+]?\d[\d,]*(?:\.\d+)?")
_WINDOW = re.compile(r"[\s-]*(?:day|week|month|year)s?\b", re.IGNORECASE)

EXACT = 0.03    # within 3% of a known number: a faithful citation
NEARBY = 0.12   # within 12% but not exact: looks corrupted, flag it
FLOOR = 0.5     # tiny absolute diffs on small values are not contradictions

def _pool(ctx: BriefContext) -> list[tuple[float, str]]:
    """Every number the prose may legitimately cite, from the displays."""
    out = []
    for c in ctx.candidates:
        for gv in c.metrics:
            for tok in _NUM.findall(gv.display):
                out.append((abs(float(tok.replace(",", ""))), gv.name))
    return out

def flagged_numbers(prose: str, ctx: BriefContext) -> list[dict]:
    """Advisory. Never raises, never blocks: it returns a list you log."""
    pool = _pool(ctx)
    if not pool:
        return []
    flags = []
    for m in _NUM.finditer(prose):
        if _WINDOW.match(prose, m.end()):
            continue                      # "14 days" is a window, not a metric
        x = abs(float(m.group().replace(",", "")))
        known, name = min(pool, key=lambda p: abs(p[0] - x))
        rel = abs(x - known) / max(x, known, 1.0)
        if rel <= EXACT or abs(x - known) <= FLOOR:
            continue                      # faithful citation
        if rel <= NEARBY:                 # near a real metric but off
            flags.append({"token": m.group(), "nearest": name, "known": known})
        # beyond NEARBY: an unrelated quantity (a count, a goal) -> ignored
    return flags

Log the flag rate on every run and watch it over time. Do not gate on it; the gotchas below cover why.

Prove parity, then flip the flag

Before the new pipeline serves anyone, run it in the shadow. Shadow testing means running the new version on the same inputs as the current one and comparing outputs without users ever seeing the candidate’s responses. For an LLM feature the comparison cannot be string equality, since the prose differs every run. Compare a structural fingerprint instead: which categories showed up, how many takeaways, whether required elements are present. In this release that meant six committed fixture scenarios, a baseline of twelve live generations frozen from V1, and a parity check of V2’s fingerprints against it. The cutover shipped behind an environment flag that started default-off, exactly the release-toggle pattern Martin Fowler describes for shipping latent code paths safely, and flipped to default-on only after parity held on all six scenarios. V1 stayed reachable as the rollback value.

You can verify the deterministic pieces on your machine right now:

# verify.py
from planner import assemble_context
from grounding import flagged_numbers

metrics = {"error_rate": 1.4, "error_baseline": 0.6,
           "p95_ms": 480, "p95_baseline_ms": 350}

a = assemble_context("2026-06-27", metrics)
b = assemble_context("2026-06-27", metrics)
assert a == b, "planner must be deterministic"
print("fingerprint:", sorted(c.category for c in a.candidates))

faithful = ("Error rate hit 1.4% against a 0.6% baseline over the last "
            "14 days; p95 sits at 480 ms.")
corrupted = ("Error rate hit 1.4% against a 0.6% baseline; p95 sits at "
             "512 ms after 2,300 requests.")
print("faithful prose flags:", flagged_numbers(faithful, a))
print("corrupted prose flags:", flagged_numbers(corrupted, a))

Running python3 verify.py with these files in one directory prints:

fingerprint: ['errors', 'latency']
faithful prose flags: []
corrupted prose flags: [{'token': '512', 'nearest': 'p95_ms', 'known': 480.0}]

The faithful prose passes clean, including the “14 days” window and the unrelated request count. The corrupted 512, sitting near the real 480 but not equal to it, gets flagged. That is the whole advisory signal working on your own machine before any model is involved.

Gotchas

Every one of these comes from this release’s actual history.

• Grounding as a hard gate fails on correct prose. The first live shadow run failed the invention gate on three of six scenarios, and on inspection every flagged token was right: a baseline the model had derived correctly, a “14 days” time window colliding with a metric magnitude. The escape was threefold: make the signal advisory-only, skip number-plus-window-word tokens, and expose derived values like baselines as citable context entries so the model quotes the real number instead of computing one grounding cannot trace.
• Summarizing the context starves the prose. The first planner collapsed 14 days of workouts into counts, and the briefs went noticeably generic; one never mentioned the previous day’s 6-mile run because the generator literally did not have it. If the model should cite a thing, put the thing itself in the context, not an aggregate of it. Summaries are for triggers; the generator needs the rows.
• An eval harness that writes production files will eventually do it. The A/B harness called the composer with its save path enabled, so every eval run silently overwrote the live saved brief, and a single unparseable model output crashed the whole run. Eval code paths should never persist by default, and each generation should be wrapped so a failure is recorded as a flake rate instead of aborting the batch.
• Wall-clock time hiding in the planner breaks determinism. The trend cutoff was computed from today’s date inside assembly, so fixture tests passed only when run on the date the fixtures assumed. Thread the date in as a parameter everywhere; a planner that consults the clock is not deterministic, whatever its tests say on the day you wrote them.

Sources

• Building effective agents — Anthropic — prefer the simplest solution; workflows route LLMs through predefined code paths, and complexity must demonstrably earn its place.
• Models — Pydantic — models parse untrusted data and guarantee the result conforms to the declared field types.
• Shadow Testing — Microsoft Engineering Playbook — run the candidate version on the same traffic as the current one and compare outputs without affecting users.
• Feature Toggles — Martin Fowler — release toggles ship latent code paths to production and keep the old path available while the new one proves out.

Changelog

• release: 0.15.1 — V2 brief composer cutover + agent-owned plan edits (dev → main) (#73) (9570248)