Judgment Models (System 1 AI)
Judgment Models (System 1 AI)
A model class that does fast, cheap, structured micro-judgments at volume — pick-one, score-on-a-scale, yes/no — and does not generate. Named by Typesafe after Kahneman's System 1 (fast, instinctive pattern-matching) vs System 2 (slow, deliberate reasoning) — the frontier LLM being the System 2 half. Jev is the first prominent example. Coined framing via How People Are Actually Using Jev (AI Daily Brief).
The core idea
Repeatedly read something → make a small judgment → take a predictable next step.
LLMs made judgment possible but priced it at frontier rates, one item at a time. A judgment model makes each judgment near-free and sub-second, so you can ask every question about every item — and that turns a difference of scale into a difference in kind (e.g. checking every sentence against a style guide, not one generic review of the document).
Where it sits in the AI stack
| Layer | Job | Example |
|---|---|---|
| Rules / code | Deterministic, known-format decisions | "It's a PDF" |
| Judgment model (System 1) | Semantic classification, scoring, routing at volume | "It's an invoice for project X; someone must see it" |
| LLM (System 2) | Generation, multi-step reasoning, nuanced trade-offs | Draft the reply, explain the decision |
The interesting designs compose the layers:
- Router in front of LLMs — pick the model, reasoning effort, skill, or tool per step (Model Routing, Skills (Claude Code)).
- Judge behind agents — online evals over production traces; a cheap complement to LLM as Judge and a natural executor for Binary Eval Assertions.
- Rank then review — judgment model scores the pile, LLM reviews borderline/high-stakes cases (the pattern Whittemore recommends for hiring).
- Learn then harden — a harness that watches repeated work and migrates steps from LLM calls to judgment calls to code (one compliance-alert example: ~$2.95 → ~$0.25/alert over 10,000 alerts).
Task-fit test (four criteria)
- Answers can be enumerated in advance (categories, yes/no, scale).
- Volume — a pile or a stream.
- Cheap errors — wrong answers are low-cost or easy to catch.
- Evidence fits as text within the context limit (<32K for Jev).
Question design
One judgment per question (decompose "is this a good lead?" into fit, size, intent); describe every scale level in words; over-ask because extra questions are cheap; hand-label ~50 items as a baseline before production.
Enterprise relevance
Service-desk ticket routing, incident severity triage, compliance-alert handling, lead routing, content moderation, malicious-URL detection, and eval infrastructure for agent fleets — the most repeatable, highest-volume decisions in an IT org. Echoes the Narrow Agents thesis: value concentrates in narrow, repeated, painful judgments.
Cautions
No reasoning is returned — unsafe as a sole decider in hiring, money, security. Weak at multi-hop, arithmetic/dates, consistency, and inferring intent.
note Evidence status One source so far (an AI Daily Brief round-up of X demos and vendor claims). Treat the cost/speed multiples as vendor-reported until independently benchmarked.