Guardrails: hunting for hallucinated URLs

The premise: JEV as a cheap verification pass over LLM extraction outputs, scored against deterministic ground truth. The result: the errors we came to catch barely exist - and that null result is the finding.

12,199LLM-extracted URLs scored against deterministic truth

~0.03%true hallucination rate found

0/192false flags (JEV and Luna, on correct extractions)

$0.0004cost to verify 200 rows

What the data says

There was nothing to catch

The experiment went hunting for hallucinated URLs a guardrail could flag. Of 12,199 extractions, only 8 missed the strict host check - and most were HN defanging ("poly gains.com", "easel,games"), which the extractor correctly normalized. True error rate: roughly 0-0.03%.

String-matching is the wrong guardrail

The strict check flags correct normalizations - it would cry wolf on legitimate rows and train a human to ignore it. JEV and Luna both read the comments the way a human would ("yes, this references that site") on all 192 controls.

Guardrails belong where errors are

Exp1 found a real 15% disagreement zone; exp6 found a 0.03% error rate. Cheap verification pays on the first and is waste on the second. Measure the error rate, then point the gauntlet at it.

Verification cost is roundoff

200 verifications at 345ms median and $0.0004. When errors ARE common (exp1), the same mechanism prices out at dollars per hundred thousand checks - the economics only ever favor building it after the measurement.

Bottom line

The careful experiment is the one that comes back empty: the prior extraction pass was careful with URLs, so a guardrail there would have been theater. Null results are results - this one cost $0.001 and redirected the guardrail budget to surfaces with real errors.

How the verification works

One Choice question per row - the claimed URL and the comment are the state, the criteria carry the judgment:

state: "Comment: <full comment text> Claimed project URL: https://polygains.com" questions: url_referenced: choice { yes: "The comment mentions the URL, its domain, or an obvious name of that site", no: "The URL or its domain does not appear in the comment" } -> "yes" at 345ms, $0.000002 - and semantically right: the comment says "poly gains.com" (HN defanging), so the strict string check was wrong, not the extraction.

The 8 strict misses, inspected

extracted URL	what the comment actually said	verdict
polygains.com	"https://poly gains.com"	correct normalization (HN defanging)
easel.games	"https://easel,games"	correct normalization (punctuation)
dailybaffle.com	"https://daily baffle.com/truthsorting"	correct normalization (spacing)
savreel.app	"savreel(.) app"	correct normalization (dot trick)
lightningestimates.com	tool named, no link in truncated text	borderline - name-based inference
+ 3 more of the same shapes