Kenyan business news: what is each story for?
400 business stories from a public Kenyan news API (source anonymized), classified by JEV into what each story is FOR: trading signal, market analysis, opportunity, policy watch, consumer info, or awareness - with direction and urgency. Five models compared on the identical task.
What the data says
Kenyan business news is mostly watching, not acting
27% policy_watch and 26% awareness: half the feed is context. True trading signals are 12.5% - and of those, 34 of 50 point upward. The usable output: a daily "what's actionable" slice that is 5x smaller than the feed.
Confidence is task-shaped
Only 20.8% of stories got JEV's high confidence, vs 68.5% on exp1's binary task. Multi-class classification over abstract categories is genuinely harder: question design drives confidence. Binary decompositions would buy certainty back at 2-3x the questions.
Direction is the weak field
JEV and GPT Luna agree on direction only 49.8% of the time, and 3 of JEV's 50 signal rows answered "none" (~1%/question flip rate). Direction claims need human labels before being published - use use-category and urgency with confidence.
Model personalities differ on subjectivity
On this 6-way task pairwise agreement collapses to 62-78% for every pair. JEV stays the most central voice (highest average pairwise, 71.7%) - the strictest, cheapest grader. Luna/mini label more "opportunity"; JEV reserves "awareness".
Bottom line
For cents per day, a calibrated model turns a firehose into a prioritized briefing: what each story is for, which direction it pushes, and how soon it matters - with confidence that tells you which rows to double-check. Cheapest model here ($0.0125) beat the most expensive one ($2.74) on consensus fit.
Cost and speed (log scale)
How JEV does this
One call per story: the title plus the site's own summary is the state, and three questions ride on it - a 6-way use Choice with each label described ("trading_signal: actionable now for an investor..."), a 4-way direction Choice ("answer none unless this is a trading signal"), and a 3-level urgency Score. All three evaluate in parallel on the same call:
Browse the classified feed
| story | use | dir | urg | conf |
|---|