NdovuNdovuExperiment 1 · HN mining

HN mining: Jev vs chat LLMs on 400 Show HN comments

Binary classification (does a comment share a project?) over a stratified sample of 18,417 previously-processed Show HN comments. Every number below is from lab-measured runs through OpenRouter on 2026-09-20: typesafe/jev-1.13 decisions and four chat models under strict structured output.

85.5%JEV agreement w/ prior pass (99% on negatives)
95.2%JEV agreement w/ 5-model consensus (2nd of 5)
$0.0093JEV cost, 400 calls (Luna $0.048, mini $0.124, Muse Spark $2.21)
347msJEV median latency (Luna 1,635ms, mini 1,131ms)

What the data says

Quality: a statistical tie

All five models agree with each other 88-95% of the time, and JEV sits at the center of that consensus. No model is meaningfully "smarter" at deciding whether a comment shares a project - so quality cannot decide this comparison. Cost and speed can.

Cost: not close

400 classifications: JEV $0.009, cheapest chat model $0.048 (5x), frontier-sized Muse Spark $2.21 (238x). Across the full 18,417-row corpus: $0.43 with JEV, $2.20 with the cheapest chat model, $102 with Muse Spark.

Speed: not close

Median 0.35s per decision vs 1.1-12.8s. Full corpus, 8 workers: JEV ~14 minutes; the historical interactive LLM pass took 26 hours for the same rows and cost $3.14 - JEV is 7x cheaper and ~110x faster than it.

Confidence you can automate on

JEV's stated confidence tracks its real accuracy (93% correct when it says 95%+, 60% when unsure). That means: automate the confident 69%, send the uncertain 31% to a human or a pricier model. Chat models expose no comparable confidence.

Where JEV loses

It cannot write a single word - project names and pitches still need a generating LLM, so it complements extraction instead of replacing it. And on subjective 6-way tasks (exp2) its accuracy edge disappears; there you buy it for cost, speed, and determinism only.

Expensive ≠ better

Muse Spark, the largest and priciest model tested, had the lowest consensus agreement (93.5%) and the slowest answers (12.8s). Price bought size, not fit: a small decisions model matched or beat it on this workload.

Bottom line

For bulk classification, JEV delivers chat-model-quality decisions at 5-238x lower cost and 3-37x faster - with calibrated confidence that tells you which rows deserve human attention. Use it as the filter; keep an LLM for the rows it flags and for anything that needs writing.

How JEV does this

One call per comment. The comment is the state; two questions ride on it - a Choice with described labels and a Score with described levels (never numeric scales). The reply carries the decision, full probabilities, and a calibrated confidence:

state: "Show HN: I built modelrouter, a tool that routes each request to the cheapest model that has historically passed your evals..." questions: is_project: choice { project: "concrete project shared", not_project: "discussion, question, intent" } usefulness: score [ "low: no real idea...", "medium: common...", "high: genuinely new, specific..." ] -> { is_project: "project", probabilities: { project: 0.93, not_project: 0.07 }, confidence: 0.9 }, usefulness: 1.4 - one call, 0.00002 dollars

Calibration explorer

JEV returns a confidence with every answer. Below: observed agreement with the prior pass, bucketed by stated confidence. Drag the threshold to see what a production filter would keep and how much it would cost.

confidence threshold: 0.70rows passing: -

Five models, one task

5-238xcheaper per classification than the chat models
3.3-37xfaster median latency than the chat models
Bars are log-scale: small differences are real too. On a binary task every model finds the same signal (pairwise agreement 88.5-94.7%) - JEV is the most central voice at the table (highest average pairwise agreement, 91.6%) and the only one that ships calibrated confidence, which is what the filter above runs on. Quality: ~parity. Cost and speed: not close.

The 15% disagreement zone

All but one of JEV's disagreements with the prior pass are prior-pass-project → JEV-not-project. Spot checks show they cluster on "working on X" intent comments where the prior pass appears title-biased ("Show HN" ≠ shareable artifact). JEV's confidence marks the genuinely ambiguous ones - filter the sample below by disagreement and confidence.

Browse the sample

show:
idsamplepriorJEVconf