Browser agents on a real trip: JEV vs GPT Luna

One browser-automation harness, two brains: JEV (typed decisions) and GPT Luna (structured chat), driving real (headed) Edge on Windows Server across a plan-a-trip-to-New-York task set. Every run recorded - successes and failures alike.

6 tasksreal sites: flights, hotel, transit, eSIM, events, shopping

3/6completed by each brain (identical harness)

0.5s vs 4.9sper-step brain time (JEV vs Luna): even at 2-3x the steps, JEV finishes faster

12recorded runs, published below

What the data says

Parity where the web is kind

On the three tractable tasks both brains finish; JEV costs 30-70% less per run. A 1-step JEV "done" on the eSIM store (prices visible on landing) is goal-awareness, not luck.

Hard sites defeat both, differently

Google Flights' combo-boxes and Booking's date-picker eat JEV's step budget and break Luna's clicks. Site hostility is the adversary - not model IQ.

The impedance mismatch is workable

JEV cannot generate text, so each step splits into action → target → value calls with human-authored fill vocabularies. 2-3 calls per step still cost fractions of a cent.

Unfinished runs oscillate, not starve

Step logs show the same 5-6 actions cycling (a dropdown the text state never captures). More steps would loop; the fix is richer state - a harness v2 applied to both brains.

Bottom line

A decisions model can drive a real browser on well-structured flows at ~30% lower cost and 5-14x lower latency than a cheap chat LLM - until sites get hostile, where both need better state engineering. Fairness note: both brains ran one fixed harness written by us; JEV is the more harness-dependent brain (decomposed calls, human-authored fill texts). The full harness is published.

How does a text-only decisions model drive a browser?

Not directly - and the translation is the engineering. JEV cannot see pixels, click, or write text. The harness gives it three things: a text snapshot of the page (every visible interactive element, listed as [e18] input placeholder:'Search'), a fixed action menu (click / fill / press_enter / done / stuck), and for fill actions a human-authored vocabulary of allowed texts (search terms come from the task spec). Each step is then 2-3 sequential decisions, all selection-by-menu:

step 1 - harness extracts page state (45 elements as text) -> POST /api/alpha/decisions Q1 "What is the single next browser action?" criteria: { click: ..., fill: ..., press_enter: ..., done: ..., stuck: ... } -> JEV: click (conf 0.9) Q2 "Which element should be clicked next?" (only clickable ids offered) criteria: { e1: "button 'Accept all'", e7: "a 'Sign in'", ... } -> JEV: e1 (conf 0.88) -> harness clicks the Accept-cookie button

That repeats: extract state → choose action → choose target → (choose text) → execute → repeat until JEV answers done or the budget ends. Everything JEV "decides" is a menu pick; everything physical is the harness. This is why JEV's answers carry probabilities at every step - and why the same design lets a chat model take the identical task in one free-form JSON call, which is the fairness comparison.

Real run, T6 eBay (every step JEV made):

Q: next action?  click -> Q: which element?  e1 "Accept all"      (cookie banner)
Q: next action?  fill  -> Q: which input?    e18 "Search" -> Q: which text? "iPhone 13"
Q: next action?  press_enter                     (submit the search)
Q: next action?  click -> e31 (close a popup)
... results list with prices visible ...
Q: next action?  done                          (goal reached, 6 steps, $0.0005)

Recorded runs

Every run - successes and failures - was recorded on real (headed) Edge on Windows Server. Click a run to watch it and replay its decision log.