Nate Swenson

How to make an LLM agent cite its sources, enforced by code

Personal · No. 054 · 9 min read ·v0.1.0

Shipped

v0.1.0 of the market-research skill generates real-estate market report PDFs. An agent researches a location on the web, drafts a structured report-data.json, and a renderer turns it into a branded PDF. The interesting part isn’t the PDF, it’s the rule the JSON has to satisfy first: every stat, listing, and comp is either backed by a URL and a verbatim snippet the code can find in something it fetched, or explicitly marked unavailable. Nothing in the data shape allows a number to exist without landing in one of those two states.

That’s the pattern this post walks through: treating “cited or unavailable” as a contract enforced by code, not an instruction hoped for in a prompt.

Why asking nicely doesn’t hold up

Tell a model “only report real numbers, never make one up” and it will still hand you a confident median home price for a town it has no data on. It’s not being dishonest by its own lights; it’s doing what language models do, producing the most plausible continuation of the prompt. A plausible-sounding price is exactly the kind of continuation a market-report prompt invites. Anthropic’s own Citations feature exists because of this gap: rather than trust a model to describe where a claim came from, the API extracts the cited passage directly from the source document you gave it, which is a structural guarantee instead of a hope (Claude Platform docs). A recent citation-grounding study on legal LLM output found the same failure mode in a different domain: citations that look correctly formatted but point at case law that doesn’t say what the model claims it says (Citation Grounding, arXiv:2606.00898). The fix in both cases is the same shape: stop trusting the model’s self-report and check the claim against the actual source.

For a small agentic pipeline you don’t need Anthropic’s citations API to get this property. You need three things: a data shape that has no “trust me” state, a check that verifies a citation against what was genuinely fetched, and for any source that’s already structured data (a CSV, an API), a fetcher that never lets the model touch the raw numbers at all.

Step 1: make “cited or unavailable” the shape of the data

Start with a schema where the two acceptable states are structural, not a convention you’re hoping the model follows. Zod’s superRefine lets you attach a cross-field check that runs after the individual fields validate, which is exactly what “citation XOR unavailable” needs:

// facts-schema.mjs
import { z } from 'zod';

function citedOrUnavailable(shape) {
  return z.object({
    ...shape,
    source: z.string().url().optional(),
    snippet: z.string().min(1).optional(),
    accessedAt: z.string().optional(),
    unavailable: z.literal(true).optional(),
  }).superRefine((val, ctx) => {
    const hasCited = val.source !== undefined && val.snippet !== undefined;
    const hasUnavailable = val.unavailable === true;
    if (hasCited === hasUnavailable) {
      ctx.addIssue({
        code: z.ZodIssueCode.custom,
        message: 'a fact must be either fully cited (source + snippet) or unavailable:true -- never both, never neither',
      });
    }
  });
}

export const factSchema = citedOrUnavailable({ label: z.string() });

export const reportSchema = z.object({
  facts: z.array(factSchema),
});

source and snippet alone won’t do it, because a schema built only from .optional() fields lets an object satisfy validation with neither set. The superRefine is the part that closes the gap: a fact with no citation and no unavailable flag fails just as hard as one with both.

Step 2: verify the citation against what was fetched

A schema only proves shape. It has no idea whether source is a real URL the pipeline visited or a plausible one the model typed. Closing that gap takes two more pieces: a domain allowlist, so the model can’t cite a host it invented, and a check that the snippet genuinely appears in text the pipeline captured from that exact URL, not just somewhere in whatever else got fetched during the run. An allowlist for outbound fetches is standard SSRF-prevention practice for any code path that turns model- or user-supplied input into a network request; OWASP’s cheat sheet recommends exactly this, resolving and checking the destination against a known-good list rather than trying to blocklist the bad ones (OWASP SSRF Prevention Cheat Sheet):

// verify-citations.mjs
const DOMAIN_ALLOWLIST = ['example-data-source.gov'];

function hostAllowed(urlString) {
  let host;
  try {
    host = new URL(urlString).hostname.toLowerCase();
  } catch {
    return false;
  }
  return DOMAIN_ALLOWLIST.some((d) => host === d || host.endsWith(`.${d}`));
}

// captures: [{ file: '1.txt', sourceUrl: 'https://...', text: '...' }]
export function verifyCitations(report, captures) {
  const violations = [];
  for (const [i, fact] of report.facts.entries()) {
    if (fact.unavailable) continue;

    if (!hostAllowed(fact.source)) {
      violations.push(`facts[${i}]: source host not on the allow-list: ${fact.source}`);
      continue;
    }

    const ownCapture = captures.find((c) => c.sourceUrl === fact.source);
    const pool = ownCapture ? [ownCapture] : captures;
    const found = pool.some((c) => c.text.includes(fact.snippet));
    if (!found) {
      violations.push(`facts[${i}]: snippet does not appear verbatim in the capture for ${fact.source}`);
    }
  }
  return violations;
}

The captures array is written by a small wrapper around your fetch calls, not by the model: every time the pipeline fetches a page, it writes the raw text to disk alongside the URL it came from. That’s what makes verifyCitations a real check instead of a formality. It isn’t asking “does this snippet appear somewhere in the general vicinity of research done today,” it’s asking “does this snippet appear in the specific document this fact claims to be quoting.”

Step 3: for structured sources, skip the model entirely

Some sources aren’t prose to cite, they’re already numbers: a government CSV, a stats API. For those, there’s no reason to route the value through the model at all. Fetch it with code, parse it with code, and the “is this number real” question disappears because nothing capable of inventing a number ever touched it:

// fetch-series.mjs -- no model call in this file at all.
const SERIES_URL = 'https://example-data-source.gov/series.csv';

export function parseSeriesCsv(csvText, maxPoints = 6) {
  const rows = csvText
    .trim()
    .split('\n')
    .slice(1) // header
    .map((line) => {
      const [date, value] = line.split(',');
      return { date, value: Number(value) };
    })
    .filter((r) => Number.isFinite(r.value));
  return rows.slice(-maxPoints);
}

export async function fetchSeries(maxPoints = 6) {
  const response = await fetch(SERIES_URL);
  if (!response.ok) throw new Error(`fetch failed: HTTP ${response.status}`);
  const text = await response.text();
  return { points: parseSeriesCsv(text, maxPoints), sourceUrl: SERIES_URL };
}

One detail worth keeping: government time series routinely use a bare . for a missing observation instead of leaving the field blank. Number('.') is NaN, and the Number.isFinite filter drops it, which is why that filter is there rather than trusting every row the source hands back.

Use it, then verify it

Put the three pieces together and run a report that cheats against one that doesn’t:

import { reportSchema } from './facts-schema.mjs';
import { verifyCitations } from './verify-citations.mjs';

const captures = [
  { file: '1.txt', sourceUrl: 'https://example-data-source.gov/median-price', text: 'The median sale price in the region was $310,000 as of June 2026.' },
];

const fabricated = {
  facts: [
    { label: 'medianSalePrice', source: 'https://example-data-source.gov/median-price', snippet: 'median sale price was $412,000', accessedAt: '2026-07-16' },
  ],
};

const clean = {
  facts: [
    { label: 'medianSalePrice', source: 'https://example-data-source.gov/median-price', snippet: 'The median sale price in the region was $310,000', accessedAt: '2026-07-16' },
    { label: 'inventoryCount', unavailable: true },
  ],
};

for (const [name, report] of [['fabricated', fabricated], ['clean', clean]]) {
  const parsed = reportSchema.safeParse(report);
  if (!parsed.success) {
    console.log(`${name}: SCHEMA REJECTED -- ${parsed.error.issues.map((i) => i.message).join('; ')}`);
    continue;
  }
  const violations = verifyCitations(parsed.data, captures);
  console.log(`${name}:`, violations.length === 0 ? 'PASSED' : `REJECTED -- ${violations.join('; ')}`);
}

Running it produces:

fabricated: REJECTED -- facts[0]: snippet does not appear verbatim in the capture for https://example-data-source.gov/median-price
clean: PASSED

The fabricated report has a real, allowlisted URL, it just doesn’t quote what that URL says, and the check catches it. Leave out both source/snippet and unavailable on a fact entirely and you don’t even get that far; reportSchema.safeParse rejects it before verifyCitations runs, with a fact must be either fully cited (source + snippet) or unavailable:true -- never both, never neither.

Gotchas

The negation check that punctuation quietly broke. The real skill also excludes distressed-sale comps (foreclosures, short sales) from valuations unless the source text explicitly negates the term (“not a short sale”). That check tokenizes the snippet and looks for the trigger phrase within a window of a negation word. The first version compared tokens as-is, and a trigger phrase sitting at the end of a sentence, like “…this is a short sale.”, tokenized to sale. with the period still attached. sale. never equals sale, so the window comparison silently failed and a genuinely distressed comp sailed through unflagged. The fix was to strip leading and trailing punctuation from every token up front, not just when checking for a negation word next to it.

A citation that verified against the wrong capture. The first version of the snippet check asked “does this snippet appear in any file the pipeline fetched during this run,” not “does it appear in the file fetched from this fact’s own URL.” That’s too loose: a real snippet captured for one fact could accidentally satisfy a fabricated source URL on a different fact, as long as the two snippets happened to overlap. The fix is the ownCapture line in verifyCitations above, scoping the check to the capture whose recorded source URL matches the fact being checked, and only falling back to the wider pool when there’s no capture recorded for that URL at all.

Treating “a manifest exists” as all-or-nothing. Once captures were tied to their source URLs, it was tempting to gate the scoped check on whether a manifest file existed at all: if one’s there, only check against it; if not, fall back to the loose search. That breaks the moment a run mixes manifest-tracked captures with older or manually-added ones that never recorded a URL. The manifest coverage decision has to be made per fact, not once for the run, or facts with real citations start failing for having the misfortune of running alongside facts that don’t have manifest coverage.

A code-fetched number isn’t automatically the right number. Fetching structured data with code instead of a model removes the fabrication risk, but not the “did I fetch the right thing” risk. One government price series used for a regional reference chart turned out to be a house-price index, a base-year-100 number like “579.92”, not a dollar figure. It’s real, it’s code-fetched, and rendering it with a currency prefix would have shown a home value off by two orders of magnitude. Nothing in a schema catches this, because 579.92 is a perfectly valid number. The only fix was reading what the source series represents before wiring it in, and picking a genuinely dollar-denominated series instead.

Sources