← narwal.one/Second Brain
SecondBrain
Ask the Brain
Index/Conceptupdated Mon Aug 10 2026 08:00:00 GMT+0800 (Philippine Standard Time)

Harness (LLM Agents)

harnessagentsllmarchitectureharness-engineeringscaling-lawsneuro-symbolic

Harness (LLM Agents)

The scaffolding around an LLM that turns it from a stateless next-token machine into a useful agent. Roughly: tools + context + memory + guardrails + observability.

The most-cited line, from Praveen Akkiraju in Agentic AI in the Enterprise (Praveen Akkiraju, CXOTalk):

"The agent IS the harness."

LLMs are stateless and steerable; the agent's behavior — and its safety — emerges almost entirely from the harness around it.

What's in a harness (per Praveen)

  • Tools — APIs, MCP servers, CLIs, file/browser/code execution
  • Context — conversation, scoped data access, role-based filtering
  • Memory — long-term, structured (e.g. wiki pages, entity stores)
  • Guardrails — prompt-level constraints, security/compliance, sandboxing
  • Observability — tracing at every step, not just final output (errors compound in multi-agent setups)
  • Eval — well-defined criteria for "good," supports continuous tuning
  • Agent identity — who the agent is acting as, what's delegated to it (per AWARE Framework / Governing AI Agents at Scale (Glean + Cvent, CXOTalk)). Traditional IAM doesn't model this; agents reason and delegate, none of which classical access control was designed for.

For an enterprise lens on these components, see the AWARE Framework — its 5 pillars (identity, context, guardrails, risk scoring, ecosystem observability) overlap heavily with this list. AWARE is the most concrete technical-controls breakdown of the harness in this wiki.

Examples in this wiki

  • Claude Code's harness: CLAUDE.md schema + slash commands + sub-agents + permission modes — all decisions about what to expose and how
  • Codex's harness: apply-patch tool + bash semantics + custom ESLint packages + persona-based reviewer agents on every push (see Harness Engineering (Ryan Lopopolo, AI Engineer))
  • OpenClaw's harness: Gateway + Adapters + Skills + agents.md/sole.md — see OpenClaw
  • Blitzy's harness: prompt-input governance + autonomous orchestration + CI/CD integration — see Blitzy
  • This Second Brain's harness for the LLM Wiki Pattern: CLAUDE.md (schema), index.md (retrieval), log.md (memory of operations), source-type frontmatter (structured metadata)

Harness Engineering as a discipline

Two AI Engineer London 2026 talks named the activity:

  • Harness Engineering (Ryan Lopopolo, AI Engineer) — operational. Harness = surfacing instructions to the model at the right time. Just-in-time via lints, test failures, reviewer agents. Heavy structural enforcement (750-package PNPM workspace, package privacy, dependency-edge lints). Garbage Collection Day is the weekly ritual: turn every observed PR slop into a durable harness rule (lint, test, doc, reviewer-agent prompt). See also LLM as Fuzzy Compiler (the underlying mental model) and Code Is Free (the premise).
  • Context Is the New Code (Patrick Debois, AI Engineer) — process. The Context Development Lifecycle (Generate → Evaluate → Distribute → Observe) is what wraps the harness over time. Patrick name-checked harness engineering on stage as a parallel discipline; the two talks are companion pieces.

These are the two most prescriptive sources in the wiki on how to do harness work day-to-day. Praveen, Karpathy, Boris, and Blitzy all describe the harness; Lopopolo and Debois describe how to build and improve one.

Open contradiction worth tracking

Three views in this wiki disagree about the harness's long-term importance:

Reconciliation: Praveen describes the present (jagged models need scaffolding). Boris predicts shrinking safety scaffolding (prompt injection guards, permission modes). Lopopolo argues productivity scaffolding (context-surfacing lints, reviewer agents) grows because each new model release leverages the existing investment. All three can be simultaneously right — they're talking about different parts of the harness. Re-evaluate after the next 1–2 model generations land.

Source convergence

Across Praveen Akkiraju, Andrej Karpathy, and Enrique Ibarra (Blitzy), three independent voices arrive at the same playbook: encode governance and intent as harness inputs upfront, not as post-hoc review. Praveen calls them .md policy files; Karpathy calls them specs/docs the agent works against; Blitzy bakes them into the prompt itself.

Harness scaling — what the right coordinate is

Scaling Laws for Agent Harnesses via Effective Feedback Compute (arxiv 2605.29682, May 2026) is the first source here that proposes a measurable scaling axis for harnesses. The paper argues raw expenditure (tokens, tool calls, ops, wall time, cost) is the wrong coordinate because it doesn't distinguish useful feedback from redundant or unstable interaction. Their alternative, Effective Feedback Compute (EFC), only credits feedback that is informative, valid, non-redundant, and retained for subsequent decisions, normalized by task demand.

Empirically, EFC-based coordinates beat raw-compute baselines (and a strong multivariate SAS baseline) at predicting failure rates across synthetic, code, real-benchmark, held-out, and prospective-validation tasks. Matched-budget interventions — improving feedback quality with raw cost and tool calls held fixed — raise success. The conclusion the paper draws:

"Harness scaling is governed less by how much computation is spent than by how efficiently raw budget is converted into durable, task-sufficient feedback."

For this vault, EFC is the missing measurement that connects everything above to a number:

The "three views of harness importance" contradiction below is now empirically testable — if the EFC vs raw-compute gap closes on newer models, Boris's prediction wins; if it stays open, Lopopolo's investment-compounds thesis wins.

What top-performing agentic artifacts look like — MAC empirical, June 2026

Meta-Agent Challenge (Autonomous Agent Development Benchmark) asked a related question: when frontier code agents are tasked with autonomously designing an agent for SWE-Bench / Terminal-Bench / etc., what does the winning harness look like? Empirical answer, from qualitative artifact analysis:

  • Minimal ReAct loops over a small toolset — not tree search, not planner-worker pipelines
  • Prompt caching on every API call to minimize per-loop latency
  • Pre-search warming — populate context from issue symbols before the first LLM invocation
  • A singular verification nudge — force the model to verify all requirements before terminating
  • Adaptive time budgeting with checkpointing — failure mode in losers was timeout-with-no-partial-output

For reasoning artifacts: parallel sampling + majority voting + prompt diversification dominates. None of the top reasoning artifacts used the elaborate tree-search / planner-worker designs prevalent in the literature.

The implication is anti-elaboration: across two independent papers (EFC and MAC), the empirical answer for current frontier models is that minimal-but-well-tuned beats elaborate. This aligns with Boris's "shrinking harness" prediction at the agentic-loop layer, while Lopopolo's "compounding harness" thesis still holds at the workspace-rules layer (lints, reviewer agents, package policy). The two camps may be talking about different parts of the harness — see the open contradiction above.

2026-07-21 — The community-wide arc: Weng's essay lineage (AI Engineer WF 2026)

5 AI Engineering Trends That Non Engineers Should Know About (AI Daily Brief) (Whittemore's read of Richard McManus's AI Engineer WF 2026 write-up) surfaces the clearest evidence yet that harness engineering is now the AI-engineering community's dominant frame:

  • Lilian Weng's 2023 essay LLM-Powered Autonomous Agents anatomised agents by what's inside the agent — planning, memory, tool use. Anchor examples: AutoGPT, BabyAGI, GPT Engineer.
  • Weng's 2026 essay Harness Engineering for Self-Improvement pivoted to what's around the agent — workflows, context, permissions, evaluations, persistence, continuous improvement.

McManus's synthesis: "AI engineering has moved beyond prompting models towards engineering reliable systems around them." At the event day-2 keynote, Roman Yampolskiy framed the shift as: "software ate the world, then AI ate software, but now the AI engineers are eating the world" — with tools like Codex as "making it easier for engineers to collaborate with agents", not to replace them.

This vault has been carrying the harness thesis for a year (Praveen's "the agent IS the harness" was the anchor); the 2026 WF is the first-major-event where the term "harness engineering" itself became the community's default vocabulary, not a niche framing. Cross-references: Loops as Core Primitive adds the inner-loop / outer-loop taxonomy (also from WF 2026); Software Factory is the enterprise-infrastructure form the discipline ships in; Skill Engineering is the capabilities-authoring specialisation within the discipline.

2026-08-02 — The three-primitive aphorism (rari / isofunds)

A Jul-28 X thread from @0xwhrrari proposed a three-word taxonomy for agent architecture; the top reply from @isofunds (1K likes) gave it its memorable shape:

"harness = environment, loop = feedback, graph = flow" — cleanest mental model for agent architecture i've seen.

Twitter-tier evidence (single tweet + single reply) — but the framing is compact, load-bearing, and lines up with the vault's structural read: harness is fully anatomised here + at Harness Engineering (Ryan Lopopolo, AI Engineer); loop is fully anatomised at Loops as Core Primitive (including the inner-loop / outer-loop split from AI Engineer WF 2026); graph = flow is the missing third primitive — the vault has GraphRAG and Graphiti but not the LangGraph-style agent-flow-as-graph concept page. Flagged as a gap on the source page rather than filled from a Twitter reply. A second substantive vault touch — a LangGraph deep-dive, a multi-agent-orchestration talk, an enterprise case study with named graph primitives — earns the concept page.

@mycomputerspot's reply on the same thread — "the model gets blamed a lot when the actual problem is that nobody designed the loop around it" — is a third-vantage anchor on the same claim Loops as Core Primitive carries via the outer-loop-engineering discourse: the residual engineering surface after the model is chosen is loop design, and blaming the model is usually mis-attributing loop-design failure.

2026-08-10 — The neuro-symbolic acceptance layer (Coyle)

Why Agentic Systems Need Ontologies (Frank Coyle, AI Engineer) adds a specific harness pattern the vault hadn't previously named: a symbolic acceptance layer at the tool-boundary.

  • Pydantic at the door. Structural type-check of tool-call inputs before execution — catches malformed calls the LLM produced.
  • Ontology at the ledger. A domain reasoner (RDFS / OWL / SHACL / datalog) validates tool-call outputs against the domain graph before persistence — catches semantically-invalid states (duplicate refunds, wrong-class payouts, made-up enum values) that pass natural-language inspection but violate the domain model.

Design principle: agents should have no side effects until validated. The neural side does the reasoning; the symbolic side is a deterministic acceptance check.

This sits alongside the existing harness pillars in a specific way:

  • Praveen's five pillars (the agent IS the harness) name what the harness contains; Coyle's pattern is a specific engineering shape for the guardrails pillar — specifically a domain-side guardrail that composes with the security-side guardrails (Prompt Injection, Lethal Trifecta) rather than duplicating them.
  • Complements LLM as Judge: LLM-as-judge is neural checking neural (probabilistic acceptance); ontology-as-ledger is symbolic checking neural (deterministic acceptance). Different verification epistemologies, both live in the same harness slot.
  • The Character Lives in the Harness (Not the Weights) finding gains a parallel: not just character lives in the harness, but correctness against the domain lives in the harness's symbolic layer. The weights don't and can't know your business's constraints.

Under-triangulated in this vault so far (single 20-min conference talk, no runnable reference architecture, no benchmark comparison), but named here because the pattern is likely to attract more sources as agent-reliability moves from security surface to action surface. See Neuro-Symbolic AI for the paradigm.

2026-08-08 — The system-prompt token tax measured across three harnesses

Per DeepSeek-V4-Flash-0731 Reddit Reception Research (2026-08-08), u/xquarx published a harness showdown running Claude Code vs OpenCode vs Pi against DeepSeek V4 Flash on vLLM with identical diffs:

  • Pi: 1,340 tokens system-prompt overhead per call
  • OpenCode: 7,197 tokens
  • Claude Code: 23,132 tokens — of which the actual system prompt is only 1,430 tokens; the other ~22k is the description of Claude Code's 27 tools.

[↑300, 151 comments on r/LocalLLaMA.] Commenters correctly criticised the chart's error bars as misrepresenting a skewed distribution; OP added a zoom toggle.

Two things follow that this concept page needs to carry:

  • The harness has a per-call token tax that is measurable and 17× different across otherwise-comparable production agent frameworks. The tax is dominated by tool descriptions, not by prose-style system-prompt content — Claude Code's actual instructions are ~1.4k tokens, competitive with Pi's whole overhead. The tool-set surface area is where the harness cost lives.
  • This is the same discipline named on Context Rightsizing but with a first-principles measurement rather than a vendor-blog claim. Anthropic's own July-24 2026 blog said Claude Code Opus 5 rightsized its system prompt by 80% with no measurable loss on coding evaluations; xquarx's numbers make the analogous "what's the tool-description equivalent?" question sharp — Claude Code's 22k of tool descriptions is the biggest single rightsizing candidate visible in the harness market. Every harness of any complexity now needs a routinised per-call token-tax measurement, per (harness × model × workload).

The Character Lives in the Harness (Not the Weights) concept lands in the same corpus and composes with this: the harness both defines the model's character and is where most of the per-call token economy is fought. Which is why prompt-caching mechanics are the ratchet that makes carrying a large system prompt survivable in the first place (98% cache-hit discount on DeepSeek; per-call cached input billed at 2% of list).

Sources