nate.swenson_
Resume

Let the agent judge, let the script verify: a pattern for self-checking skills

· 10 min read ·v1.0.0

Shipped

v1.0.0 of the resume skill deletes its entire subprocess pipeline. It used to shell out to its own claude -p call just to rewrite résumé bullets, which meant a second API surface, its own timeout handling, and its own cost on every run. Now the agent you’re already talking to reads the résumé, reads the job posting, and rewrites the bullets in-conversation, with no subprocess LLM call anywhere in the loop. What’s left as actual code is only the part that has to be code: a schema plus a deterministic content-truth checker (scripts/validate.mjs), and PDF rendering. This is the second attempt at this exact architecture; the first shipped and got reverted the same day because nobody ran it end to end before calling it done. This time it shipped behind a real evaluation harness, and that harness is a pattern you can build for any skill that hands an agent judgment calls but still needs to catch it when the judgment goes wrong.

What you need

Node with zod installed (npm install zod) is enough to build and run every block below. Nothing here needs a paid API key to follow along; the one piece that does (the LLM-judge call) is written so a missing key degrades gracefully instead of blocking anything.

Draw the line between judgment and verification

The first design decision isn’t architecture, it’s a question: which parts of this task have more than one right answer, and which parts have exactly one? Rewriting a bullet to foreground the right experience is a judgment call; there’s no single correct rewrite, and an agent with the résumé and the job posting in its own context is better positioned to make that call than a second model that has to be told both from scratch. Checking whether that rewrite invented a number the source never stated is not a judgment call. It has one correct answer, checked the same way every time. Anthropic’s own framing of Agent Skills draws this same line at the tool level: “large language models excel at many tasks, but certain operations are better suited for traditional code execution” (Anthropic: Equipping agents for the real world with Agent Skills), and its skill-authoring guidance frames the choice as a dial rather than a switch: high freedom (prose) for tasks where “multiple approaches are valid” and “decisions depend on context,” low freedom (a fixed script, few or no parameters) where “operations are fragile and error-prone” and “consistency is critical” (Anthropic: Skill authoring best practices). Tailoring sits at the high-freedom end. Fact-checking the output sits at the low-freedom end. Everything below is just building the low-freedom half properly.

Write the deterministic checkpoint

The same best-practices guidance names this pattern directly: “the ‘plan-validate-execute’ pattern catches errors early by having Claude first create a plan in a structured format, then validate that plan with a script before executing it.” The intermediate artifact here is a small JSON object holding the generated claims; the agent writes it before anything downstream consumes it, and a script checks it against the source text it’s supposed to be grounded in.

// validate.mjs: the deterministic half: a schema gate, then a content-truth gate
import { readFileSync } from "node:fs";
import { z } from "zod";

const ClaimsJSON = z.object({
  bullets: z.array(z.string()),
});

function norm(s) {
  return s.toLowerCase().replace(/\s+/g, " ").trim();
}

// Matches "5 years", "40%", "12 yrs": a claimed number tied to a unit.
// Note the \b sits *before* the unit alternation, not after it: a trailing
// \b never fires after punctuation like "%", since \b only exists between a
// word character and a non-word character, and "%" followed by a space is
// non-word-to-non-word. Putting \b only where a real word char follows
// (years, yrs, percent) and leaving "%" to end on its own is what makes
// this catch "40%" instead of silently skipping it.
const NUMERIC_CLAIM = /\b(\d+)\+?\s*(?:%|percent\b|years?\b|yrs?\b)/gi;

function findUnsupportedNumericClaims(bullet, sourceText) {
  const claimed = [...bullet.matchAll(NUMERIC_CLAIM)].map((m) => m[1]);
  const sourceNumbers = new Set(
    [...sourceText.matchAll(NUMERIC_CLAIM)].map((m) => m[1]),
  );
  return claimed.filter((n) => !sourceNumbers.has(n));
}

export function validate(claimsPath, sourcePath) {
  const parsed = ClaimsJSON.safeParse(JSON.parse(readFileSync(claimsPath, "utf8")));
  if (!parsed.success) return { ok: false, violations: [`schema: ${parsed.error.message}`] };

  const sourceText = norm(readFileSync(sourcePath, "utf8"));
  const violations = [];
  for (const bullet of parsed.data.bullets) {
    for (const n of findUnsupportedNumericClaims(norm(bullet), sourceText)) {
      violations.push(`bullet "${bullet}" claims "${n}" not found in source`);
    }
  }
  return { ok: violations.length === 0, violations };
}

// validate-cli.mjs: thin CLI wrapper: exit 0 and print "clean", or list
// violations and exit 1. This exit code is the signal the agent's retry
// loop reads.
import { validate } from "./validate.mjs";

const [, , claimsPath, sourcePath] = process.argv;
const result = validate(claimsPath, sourcePath);
if (result.ok) {
  console.log("clean");
  process.exit(0);
} else {
  console.log("violations:");
  for (const v of result.violations) console.log(` - ${v}`);
  process.exit(1);
}

The schema check runs first, deliberately, before the content check. A malformed JSON that skipped it would fail later with a confusing error somewhere downstream instead of a clear one here.

Cap the one gate that isn’t free

The deterministic check above only catches what it was explicitly written to catch: an invented number, in this example. Whether a rewrite is well-targeted at the job, not just factually clean, is an open-ended quality question, and that’s exactly the class of check a second model is suited for: an LLM judge scoring output against a rubric on a dimension a fixed assertion can’t express. It’s also the one part of this pipeline that costs real money on every call, so it needs its own guardrail, not just a try/catch. Two rules make that guardrail worth trusting: the spend check happens before the call, not after, and a failure in the judge (budget, network, a parse error) can never throw or block the deterministic gate above it.

// budget.mjs — pre-call budget enforcement plus a fail-open wrapper for the
// optional judge call
export class BudgetExceededError extends Error {}

export class BudgetGate {
  constructor({ capUsd, safetyFactor = 1.3 }) {
    this.capUsd = capUsd;
    this.safetyFactor = safetyFactor; // headroom for estimate error
    this.spentUsd = 0;
  }
  assertBudget(estimatedUsd) {
    const projected = this.spentUsd + estimatedUsd * this.safetyFactor;
    if (projected > this.capUsd) {
      throw new BudgetExceededError(`would spend $${projected.toFixed(2)}, cap is $${this.capUsd}`);
    }
  }
  record(actualUsd) {
    this.spentUsd += actualUsd;
  }
}

// callJudge: () => Promise<{ verdict, actualUsd }> — your actual API call,
// stubbed out here since it needs a real key and a real endpoint.
export async function runJudgeSafely(callJudge, estimatedUsd, gate) {
  try {
    gate.assertBudget(estimatedUsd);
    const { verdict, actualUsd } = await callJudge();
    gate.record(actualUsd);
    return verdict;
  } catch (err) {
    return { incomplete: true, reason: String(err) };
  }
}

Everything the judge produces stays informational. The build’s single pass/fail verdict comes from the deterministic gate, never from the judge’s score, because a judge call failing for infrastructure reasons that have nothing to do with output quality shouldn’t be able to fail a build that’s otherwise clean. A guide to LLM-as-a-judge makes the same case for combining the two rather than picking one: “you could start with rule-based systems to filter out obvious issues, and then use LLM judges or human reviews for more complex, nuanced tasks,” partly because “running evaluations with powerful LLMs can get expensive” on their own (Evidently AI: LLM-as-a-judge, a complete guide).

Run it and watch it catch a bad claim

This is cheap enough to run for real. Give the checker a source paragraph and two claim sets, one with an invented number and one without.

$ cat source.txt
Led a team that migrated the billing service to a new queue over 8 months, cutting
incident volume during that window. Worked closely with the SRE group on rollout.

$ cat bad-claims.json
{
  "bullets": [
    "Led a 5 year migration of the billing service to a new queue",
    "Cut incidents by 40% during the rollout"
  ]
}

$ node validate-cli.mjs bad-claims.json source.txt
violations:
 - bullet "Led a 5 year migration of the billing service to a new queue" claims "5" not found in source
 - bullet "Cut incidents by 40% during the rollout" claims "40" not found in source

Exit code 1, two real violations, neither number present in the source. Fix the claims to only state what the source actually supports, and the same command turns clean:

$ cat good-claims.json
{
  "bullets": [
    "Led an 8 month migration of the billing service to a new queue",
    "Cut incidents during the rollout window, working closely with SRE"
  ]
}

$ node validate-cli.mjs good-claims.json source.txt
clean

That’s the retry loop in miniature: an agent writes the claims JSON, runs the checker, and if it fails, fixes the JSON directly, since it wrote it and knows exactly why each rule matters. In the resume skill, that loop is capped at three attempts before it stops and shows the user the remaining violations instead of retrying silently forever.

Gotchas

Trap: a shared global regex with the g flag looks safe inside matchAll, but \b after punctuation is not. Symptom: a numeric-claim check that catches “5 years” but silently lets “40%” through, because \b only exists at a transition between a word character and a non-word character, and % followed by a space is non-word-to-non-word: no boundary, no match. Escape: anchor \b only where a real word character follows (years\b, percent\b), and let punctuation-ending alternatives like % close on their own.

Trap: structured output silently no-ops on the claude -p CLI. If you drive a judge or extraction call through claude -p --json-schema, and you build that schema with a library like zod’s .toJSONSchema(), the schema carries a top-level $schema key by default. That key alone is enough to make the CLI’s structured-output tool registration silently fail: the model answers in plain prose, structured_output never populates, and nothing errors to tell you why. Escape: strip the $schema key before passing the schema to the CLI.

Trap: an uncapped retry loop on a validation failure. An agent that wrote bad JSON and got told why will keep trying to fix it, which is usually good, except when the underlying rule and the source material genuinely can’t be reconciled, in which case it retries forever instead of surfacing the conflict. Escape: cap the loop at a small fixed number of attempts (three, in this build) and hand the remaining violations to a human at that point instead of continuing silently.

Trap: treating “the architecture is right” as the same claim as “it works.” The first version of this exact agent-native design shipped and was reverted the same day, not because the architecture was wrong, but because it was never run end to end against a real input before being called finished. The eval harness above proves the output is good across a fixture set. It doesn’t prove anyone ran the real thing. That check still has to happen by hand, once, against a real input, before anything gets called done.

Sources

Changelog

  • [claude-skills] feat(resume): agent-native rewrite, take 2 — with eval harness (1.0.0) (#28) (541e635)
  • [claude-skills] Revert “feat(resume): agent-native rewrite — markdown-driven skill, minimal scripts (1.0.0) (#27)” (4f689cf)
  • [claude-skills] feat(resume): agent-native rewrite — markdown-driven skill, minimal scripts (1.0.0) (#27) (9c9d8c8)