Nate Swenson

Add memory to an LLM agent without breaking its prompt caches

Local Fitness · No. 068 · 9 min read ·v0.32.0

Shipped

v0.32.0 rolls five internal releases into one public tag: the coach agent gained a durable two-layer memory on every surface that speaks in its voice, report cards now persist as dated snapshots the agent can query later, a tunable personality spec, and fifteen audit fixes. The piece worth teaching is the memory. Storage was easy. The real problem was that several of my prompt surfaces cache expensive LLM calls on a hash of the assembled prompt, and memory is by definition the part of the prompt that changes. This guide builds the same design: a ledger your code computes, a journal your model writes, and injection rules that keep those caches warm.

Why two layers

The research on agent memory keeps landing on the same split. The Generative Agents paper (Park et al., 2023) gives agents a chronological memory stream plus a reflection step that distills raw observations into higher-level insights. MemGPT frames it as an operating system problem: tiers of memory, with the model managing what moves into the limited context window.

For a single-user agent you can collapse all that machinery into two layers with one rule between them:

  • The ledger is computed. Streak counts, repeat patterns, notable results; anything that is a query over data you already store. Code derives every number the agent may quote.
  • The journal is written. Short dated lines the model authors about the relationship (“blamed the heat again, second time this month”); the color a query can’t produce.

The rule: the model phrases judgments, it never derives them. If a count can come from SQL, it must come from SQL, or your agent will confidently remember things that never happened.

The journal: a capped table the model writes

Two contracts make a model-written table safe to inject into every prompt: a hard cap so the token cost is bounded forever, and structural idempotency so the same event can never be remembered twice. SQLite gives you the second one with a partial unique index, which enforces uniqueness only over rows matching its WHERE clause; chat-authored rows with a NULL key stay exempt.

import sqlite3

SCHEMA = """
CREATE TABLE IF NOT EXISTS agent_journal (
    entry_id   INTEGER PRIMARY KEY AUTOINCREMENT,
    created_at TEXT NOT NULL,
    entry_date TEXT NOT NULL,
    source     TEXT NOT NULL,      -- 'daily_report' | 'review' | 'chat'
    source_key TEXT,               -- the artifact id; NULL for chat
    seq        INTEGER NOT NULL DEFAULT 1,
    text       TEXT NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_journal_event
    ON agent_journal(source, source_key, seq)
    WHERE source_key IS NOT NULL;
"""

def connect(path="agent_memory.db"):
    conn = sqlite3.connect(path)
    conn.executescript(SCHEMA)
    return conn

Writes prune past the cap in the same transaction, so the table can never grow unbounded no matter who writes:

from datetime import datetime, date

JOURNAL_CAP = 60
ENTRY_MAX_CHARS = 240

def save_entry(conn, text, *, source, source_key=None, seq=1, entry_date=None):
    text = text.strip()[:ENTRY_MAX_CHARS]
    conn.execute(
        "INSERT INTO agent_journal (created_at, entry_date, source, source_key, seq, text) "
        "VALUES (?, ?, ?, ?, ?, ?)",
        (datetime.now().isoformat(timespec="seconds"),
         entry_date or date.today().isoformat(), source, source_key, seq, text),
    )
    conn.execute(
        "DELETE FROM agent_journal WHERE entry_id NOT IN "
        "(SELECT entry_id FROM agent_journal ORDER BY entry_id DESC LIMIT ?)",
        (JOURNAL_CAP,),
    )
    conn.commit()

def has_event(conn, source, source_key):
    row = conn.execute(
        "SELECT 1 FROM agent_journal WHERE source = ? AND source_key = ? LIMIT 1",
        (source, source_key),
    ).fetchone()
    return row is not None

In my repo the cap is 60 entries and lines are 240 characters; a memory is a line, not an essay.

The ledger: facts computed as of yesterday

Here is where the cache constraint starts shaping the design. My PDF coach calls cache a ~10 second LLM generation on a hash of the assembled prompt, and Anthropic’s server-side prompt caching has the same property in stricter form: matching is exact-prefix, so any changed byte at or before the cache breakpoint is a miss. Either way, every byte you inject into a prompt is part of a cache key.

A streak computed “through today” breaks that. Today’s step count is partial all day, so the fact flips as the day progresses and every render is a cache miss. Compute it through yesterday instead and the block changes exactly once per day, at midnight:

from datetime import date, timedelta

def streak_as_of_yesterday(daily_counts, goal, today):
    """daily_counts maps iso-date -> count. Today's count is partial all
    day, so the streak is measured through yesterday and holds until
    midnight."""
    yesterday = date.fromisoformat(today) - timedelta(days=1)
    streak, d = 0, yesterday
    while daily_counts.get(d.isoformat(), 0) >= goal:
        streak += 1
        d -= timedelta(days=1)
    return {"streak_days": streak, "through": yesterday.isoformat()}

The same shape works for any ledger fact: missed-workout streaks from your grader’s verdicts, repeat patterns from logged observations, personal bests. Each is a pure function over rows you already have, returning stable strings.

Injecting: one block, a grounding contract, an exclusion

Memory enters the prompt as a single rendered block, and the block’s header is a grounding contract: the agent may cite only what is listed, and an empty section means no callbacks. Without that header the model pads the gaps with plausible history.

HEADER = (
    "MEMORY (code-derived facts and journal lines):\n"
    "Cite ONLY facts listed below, with their counts and dates.\n"
    "If this section is empty, make no callbacks to past events.\n"
)

def render_memory_block(conn, ledger_facts, *, exclude_source_key=None, max_chars=None):
    lines = [f"- {fact}" for fact in ledger_facts]
    rows = conn.execute(
        "SELECT entry_date, source, source_key, text FROM agent_journal "
        "ORDER BY entry_id DESC LIMIT 20"
    ).fetchall()
    for entry_date, source, source_key, text in rows:
        if exclude_source_key and (source, source_key) == exclude_source_key:
            continue
        lines.append(f"- {entry_date}: {text}")
    if not lines:
        return ""
    block = HEADER + "\n".join(lines)
    if max_chars and len(block) > max_chars:
        block = block[:max_chars].rsplit("\n", 1)[0]
    return block

Resolve this once at the call site and pass the string into your prompt builders. Keep the builders pure. exclude_source_key matters in the next step. The max_chars cap exists because one of my surfaces runs a deliberately compact prompt (600 characters of memory, no more); a memory feature must not quietly bloat the prompt you spent weeks shrinking.

Reflection: the agent writes one line per event, once

Reflection runs after an artifact is saved, never before. The model reads the finished report and emits at most two MEMORY: lines, or NONE. Idempotency is layered: has_event is the cheap pre-check, and if two renders race past it, the partial unique index makes the second insert fail loudly instead of double-writing.

def reflect_on_artifact(conn, artifact_text, *, source, source_key, generate):
    """Run AFTER the artifact is saved. generate(prompt) -> str is your
    LLM call, e.g. a small fast model with tools disabled."""
    if has_event(conn, source, source_key):
        return
    prompt = (
        "You are the agent's memory. Read the report below. If something is "
        "worth remembering about this relationship, output at most two lines, "
        "each starting with 'MEMORY: ' and under 240 characters. "
        "Otherwise output NONE.\n\n" + artifact_text
    )
    lines = [l for l in generate(prompt).splitlines() if l.startswith("MEMORY: ")]
    try:
        for seq, line in enumerate(lines[:2], start=1):
            save_entry(conn, line.removeprefix("MEMORY: "),
                       source=source, source_key=source_key, seq=seq)
    except sqlite3.IntegrityError:
        pass  # a concurrent render already reflected this event

Make the whole step fail-silent in production: a reflection that dies must never take the artifact with it. In my pipeline it adds about ten seconds to a scheduled job that already runs unattended.

Verify the cache actually stays warm

Wire the pieces together in a scratch file and check the three properties that make this design work: same-day renders are byte-identical, reflection changes the block exactly once, and an artifact never sees its own memory.

import hashlib

conn = connect(":memory:")
counts = {"2026-07-23": 10500, "2026-07-24": 11200, "2026-07-25": 3100}
facts = ["step goal streak: {streak_days} days (through {through})".format(
    **streak_as_of_yesterday(counts, 10000, "2026-07-25"))]

a = render_memory_block(conn, facts)
b = render_memory_block(conn, facts)
print("stable:", hashlib.sha256(a.encode()).hexdigest() ==
                 hashlib.sha256(b.encode()).hexdigest())

reflect_on_artifact(conn, "Missed the interval session again.",
                    source="review", source_key="run-421",
                    generate=lambda p: "MEMORY: Second missed interval day this month.")
print("--- prompt for run-421 (its own line excluded):")
print(render_memory_block(conn, facts, exclude_source_key=("review", "run-421")))
print("--- prompt for every other surface:")
print(render_memory_block(conn, facts))

Running that prints:

stable: True
--- prompt for run-421 (its own line excluded):
MEMORY (code-derived facts and journal lines):
Cite ONLY facts listed below, with their counts and dates.
If this section is empty, make no callbacks to past events.
- step goal streak: 2 days (through 2026-07-24)
--- prompt for every other surface:
MEMORY (code-derived facts and journal lines):
Cite ONLY facts listed below, with their counts and dates.
If this section is empty, make no callbacks to past events.
- step goal streak: 2 days (through 2026-07-24)
- 2026-07-25: Second missed interval day this month.

Note the streak says 2, not 3: the partial 3,100 steps on the 25th are ignored until the day is over. Also add a kill switch (mine is an env var) that disables injection and reflection while leaving the data untouched; you want memory to be removable the day it misbehaves.

Gotchas

These all bit me in this release’s history, not in theory.

  • Resolving memory inside the prompt builders. My prompts module assembles a system prompt at import time, and the PDF coaches key disk caches on the assembled prompt’s hash. A builder that read the DB itself would open the database on import and hide a cache-key input inside the builder. Symptom: mysterious cache misses and an import with side effects. Escape: resolve memory once at the call site and pass the rendered string in as an argument.
  • The reflection self-cascade. Reflecting on a report card writes a memory about it; that line lands in the same card’s next prompt, which changes the hash, which busts its cache, which regenerates and re-reflects, forever. I hit this twice. Escape is layered: has_event short-circuits re-renders, and exclude_source_key drops an artifact’s own journal lines from its own prompt.
  • Facts that move during the day. This one is designed against rather than survived: a streak that counts today would produce a different memory block on every render as steps accumulate, silently turning the prompt caches into decoration. The symptom, if you ship it, is caches that miss all day and hit only at night. Escape: compute ledger facts as of yesterday, so the block changes once per day, at midnight.
  • args.get("limit") or 50 eats an explicit zero. In the chat tool handlers, or-defaulting silently turned an invalid but explicit limit=0 into the default instead of rejecting it. A validation test caught it. Escape: test is None, then validate the value on its own.

Sources

Changelog

  • release: 0.32.0 — coach memory, tunable personality, and durable report cards (dev → main) (#150) (ad86eaa)
  • chore(deps): Bump mcp from 1.27.0 to 1.28.1 (#110) (5e60d6f)
  • chore(deps): Bump garminconnect from 0.3.3 to 0.3.5 (#109) (cb08ce3)
  • chore(deps): Bump cryptography from 47.0.0 to 48.0.1 (#100) (f9a2925)