The code-judgment boundary: bug-fix discrimination on Django

Chat LLMs are known to be strong at code. Reframed as a typed decision - given two diffs, pick the bug fix - how far does a decisions model get before quality collapses? Real Django history, certain labels.

60real Django commit pairs, certain ground truth

68-73%diff-only accuracy, all models (random = 50%)

80-87%with commit messages visible

~$0.05total experiment cost

What the data says

The boundary is real but narrow

On pure diff semantics every model lands within 5 points (68-73%); JEV trails chat LLMs by only 2-5 while being 3-7x cheaper and 3.5-7x faster. This is the widest relative gap in the campaign - code semantics is where the decisions model starts to pay for what it cannot do.

Commit messages are a big lever

Messages add +8 to +13.4 points for chat models and +11.7 for JEV. The message is the signal human reviewers use too - a model that reads it approximates the review workflow.

Generation is the hard wall

Reframed as a decision ("which diff fixes the bug?"), JEV competes. Actual review - writing the analysis - is generation, where JEV by design cannot go. The boundary experiments map what to delegate.

The hybrid is the deployable shape

JEV triages every diff at 350ms (risky vs safe, which-of-these-two), the LLM reviews what it flags. Same pattern as exp1, applied to code: decisions everywhere, generation only where decisions point.

Bottom line

Decompose a code task into typed decisions and JEV lands within a few points of cheap chat LLMs at a fraction of the cost - until the task requires real semantic understanding, where the gap widens. That edge is exactly where the boundary sits, and this experiment measures it with certain labels.

How JEV reviews code (it doesn't - it decides)

JEV cannot read a diff and write "here are three issues." The task is reframed into what it can do: a forced choice between two diffs with certain ground truth. One real pair from the run, exactly as sent:

state:
  === DIFF A ===  (Django: "Fixed #36905 -- Deprecated the safe parameter of JSONResponse")
  @@ -47,7 +47,7 @@ class DjangoJSONEncoder(Colorizer):
  -        if isinstance(o, bool): ...
  +        warnings.warn(...)   <- deprecation handling added
  ...
  === DIFF B ===  ("Refs #35514 -- Cleaned up mail tests")
  -        self.assertEqual(..., ...)   <- test reorganization only

questions:
  fix_pick: choice - "Which one fixes a bug?"
  criteria: { a: "Diff A is the bug fix", b: "Diff B is the bug fix" }

-> JEV: a   probabilities: { a: 0.84, b: 0.16 }   confidence: 0.68   cost: $0.00004
ground truth: a  (correct)

Same diff pair, same question, to every model. The chat models answer through strict json_schema on the identical state text. The comparison is the brains, not the plumbing.

Accuracy by model and condition

Source: django/django at commit dd6f6b1531 - fix commits are those Django itself labels "Fixed #<ticket>", paired with same-repo non-fix commits of similar size, order randomized. Ground truth is certain; the full pair set is published in the internal records.