Build a publish gate that fails closed, even when the input is garbage
Shipped
Ghostwriter publishes to a real LinkedIn profile, so a fabricated statistic in a draft ships to a real audience. Version 0.7.0 closes that gap with a code gate: any post that asserts something about the outside world can’t publish until its claims trace to at least three real, independently-hosted, live sources. A companion sidecar file carries the evidence next to each draft, a verifier checks it, and the publish script refuses to run unless the check passes. The same release adds a regression suite for the whole skill, because the two problems turned out to be the same problem: a check that can be fooled into passing is worse than no check. Here’s how to build a gate like this, and how to keep it from lying to you.
What you need
Nothing beyond a python3 install. The whole gate is standard library only: urllib for the network calls, json and pathlib for the sidecar. No dependencies to pin, nothing to pip install in CI.
Separate the evidence from the post
The post body should never carry citation links; a LinkedIn post reads badly with a wall of URLs in it. So the evidence lives beside the draft, not inside it, as a small JSON sidecar saved as post.sources.json:
{
"external_claims": true,
"claims": [
{
"text": "HEAD returns headers without a body",
"sources": ["https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods/HEAD"]
},
{
"text": "403 means the resource exists but access is refused",
"sources": ["https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status"]
}
]
}
A purely first-person post makes no claim about the world, so it declares "external_claims": false and an empty claims list, and the gate should pass it trivially. That distinction matters: the gate only fires when there’s something to verify, so it doesn’t become a tax on every post regardless of content.
Before touching the URLs, validate the shape of what you were handed. This file is agent- or hand-authored JSON with no schema behind it, and the easiest way to end up with a gate that doesn’t gate is to let a malformed sidecar crash the process instead of failing it:
def _validate_shape(data: dict) -> str | None:
external = data.get("external_claims", True)
if not isinstance(external, bool):
return "'external_claims' must be true or false"
claims = data.get("claims", [])
if not external:
return None if not claims else "external_claims:false but claims listed"
if not isinstance(claims, list) or not claims:
return "no claims listed for an external-claims post"
for claim in claims:
if not isinstance(claim, dict):
return f"each claim must be an object; got {claim!r}"
sources = claim.get("sources", [])
if not isinstance(sources, list) or not sources:
return f"claim has no source: {claim.get('text', '?')!r}"
for src in sources:
if not isinstance(src, str):
return f"each source must be a URL string; got {src!r}"
return None
Notice the missing-key default: external_claims defaults to True when absent. An under-specified sidecar has to prove its sources, not skip the gate by omission. An explicit non-boolean value, like null, is rejected outright rather than silently treated as “personal.”
Check that a URL is live
The mechanical core is a liveness check: confirm a URL resolves to something real without downloading the whole page. HTTP HEAD asks the server for the headers a GET would return, with no body, which is exactly what you want here. The wrinkle is that plenty of CDNs and bot-management products challenge or reject a bare HEAD from a script even though the page is fine for a browser, so a script with no browser User-Agent gets treated as a bot before it gets treated as a request. Set a real User-Agent, and treat a 401, 403, or 405 as “the host is up but walled,” not “dead”:
import urllib.error
import urllib.request
from urllib.parse import urlparse
TIMEOUT = 6
ALIVE_EXTRA = {401, 403, 405} # bot-walled, but the resource is real
HEAD_RETRY = {405, 501}
USER_AGENT = (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36"
)
def _status(url: str, method: str) -> int | None:
headers = {"User-Agent": USER_AGENT}
if method == "GET":
headers["Range"] = "bytes=0-0" # ask for one byte; we only want the code
req = urllib.request.Request(url, method=method, headers=headers)
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
return resp.status
except urllib.error.HTTPError as exc:
return exc.code # a 403 is a real answer, not a failure
except (urllib.error.URLError, OSError):
return None # DNS failure, timeout, connection refused
def is_live(url: str) -> bool:
code = _status(url, "HEAD")
if code in HEAD_RETRY:
retry = _status(url, "GET")
if retry is not None: # a dead retry must not downgrade a live HEAD
code = retry
if code is None:
return False
return 200 <= code < 400 or code in ALIVE_EXTRA
MDN’s status code reference confirms all three of those codes: 401 and 403 both mean the resource exists and the request was refused, and 405 means the method itself was rejected while the resource stands. None of them mean “not there.”
Require distinct hosts, not just live URLs
One live URL is weak evidence. Three URLs from the same domain is one source wearing three hats. So the gate counts distinct hosts, normalized to strip a leading www., and only passes when there are at least three of them:
import json
from pathlib import Path
MIN_SOURCES = 3
def _host(url: str) -> str:
host = (urlparse(url).hostname or "").lower()
return host[4:] if host.startswith("www.") else host
def verify(sidecar_path: str) -> tuple[bool, str]:
try:
data = json.loads(Path(sidecar_path).read_text())
except OSError:
return False, f"no sidecar at {sidecar_path}"
except json.JSONDecodeError:
return False, "sidecar is not valid JSON"
if not isinstance(data, dict):
return False, "sidecar must be a JSON object"
shape_error = _validate_shape(data)
if shape_error:
return False, shape_error
if data.get("external_claims", True) is False:
return True, "first-person post, nothing to verify"
urls = {u for c in data["claims"] for u in c["sources"]}
live_hosts = {_host(u) for u in urls if is_live(u)}
live_hosts.discard("")
if len(live_hosts) < MIN_SOURCES:
return False, f"only {len(live_hosts)} distinct live host(s); need >={MIN_SOURCES}"
return True, f"{len(live_hosts)} distinct live sources verified"
Wire it into publish, and fail closed on purpose
The gate only matters if it sits in the one path that counts: publishing. It should run after any dry-run or author checks and before any media upload, so a failed gate never leaves an orphaned upload behind. The default on any failure, including a missing sidecar, is refusal:
import sys
def publish(draft_path: str, allow_unverified: bool = False) -> None:
sidecar = str(Path(draft_path).with_suffix(".sources.json"))
if allow_unverified:
print("WARNING: --allow-unverified set, human-only bypass in effect.")
return
ok, reason = verify(sidecar)
if not ok:
sys.exit(f"BLOCKED: source gate failed ({reason})")
print(f"Source check passed: {reason}")
# upload_media_and_post(draft_path) only runs once the gate is green
That’s the shape you want: the exception path is the default, and publishing is the narrow branch you reach only by passing. A bare publish call with no sidecar at all lands in the same refusal as a failed check, which is the fail-securely posture: on error, take the same path as denying the operation, never the same path as allowing it. allow_unverified is the one deliberate escape hatch, and it should be something only a human sets, never something the code sets on its own to clear its own gate.
Give it a command-line entry point so you can run it directly against a sidecar:
import argparse
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("--file", required=True)
args = ap.parse_args()
ok, reason = verify(args.file)
print(f"[{'OK' if ok else 'FAIL'}] {reason}")
sys.exit(0 if ok else 1)
Use it and verify
Save the code blocks above, in order, into verify_sources.py, write the sidecar from the first section to post.sources.json, add a third source URL so it has three distinct hosts, and run it:
python3 verify_sources.py --file post.sources.json
[OK] 3 distinct live sources verified
Now break it on purpose. Change one sources field from a list to a bare string, the mistake an LLM authoring this JSON will make sooner or later:
python3 verify_sources.py --file broken.sources.json
[FAIL] claim has no source: 'HEAD returns headers without a body'
No traceback, a specific reason, exit code 1. And the first-person case:
python3 verify_sources.py --file personal.sources.json # {"external_claims": false, "claims": []}
[OK] first-person post, nothing to verify
Three real outcomes, all deterministic, all explainable from the exit code and one line of text.
Gotchas
Fail-closed does not mean fail-clean, unless you make it. The first version of this gate treated a source as {"text": ..., "sources": [...]} and moved straight to claim.get("sources", []). Feed it a source written as a bare string instead of a one-element list and Python happily iterates the characters of that string as if each one were a URL, or raises AttributeError deeper in the pipeline. Nothing unsafe ever published, since a crash also blocks the publish path, but “crashed” and “refused with a clear reason” are not the same failure mode, and only one of them is debuggable at 11pm. Validate the shape before you touch the content.
A retry can downgrade a result that already succeeded. The HEAD-then-GET fallback exists because some hosts reject HEAD outright. The bug: if that fallback GET hits a transient network error, a plain code = _status(url, "GET") overwrites a HEAD result that had already proven the host is up, turning a live source into a dead one. The fix is one line, if retry is not None: code = retry, but the lesson generalizes: a fallback check should only ever improve on a result, never silently replace a good one with “unknown.”
A verifier that never asserts anything passes every time. This release also added an eval harness for the whole skill’s guardrails, and one of the harnesses had the same disease as the sidecar bug: it invoked the skill without the flag needed to capture streaming output, checked nothing, and reported success on every run regardless of what happened. A test suite with full line coverage can still verify zero behavior if nothing in it can fail. Before trusting any verifier, including the one in this post, break it on purpose the way the “Use it” section above does, and confirm it reports the failure.
Sources
- HEAD request method — MDN — HEAD returns the headers a GET would, with no body, which is why it checks liveness without downloading the page.
- HTTP response status codes — MDN — 401, 403, and 405 all confirm the resource exists; only the request was refused.
- Bot Fight Mode — Cloudflare docs — bot-management products may challenge non-browser traffic, including legitimate scripted requests.
- Fail securely — OWASP — on error, a security check should follow the same path as denying the operation, never the same path as allowing it.