Mechanistic Interpretability

The research programme of reading a neural network's internals — individual neurons ("detectors"), the circuits they form, and the directions in activation space — well enough to explain why the model does what it does, rather than inferring it from outputs. Contrast with behavioural evaluation (ask, red-team, grade), which is how AI systems are mostly checked today.

Why it matters for safety (per Nick Cammarata, via A Potentially Historic AI Breakthrough (AI Daily Brief)): "I don't think we can solve alignment if we don't know what we're building." Behavioural evals only cover what you thought to ask, and a model that knows it is being watched can pass them — the Claude Opus 5.5 "cheating less, or hiding it?" worry is the concrete case. A deceptive plan must be legible to the model's own circuitry to work, so a reader of that circuitry could in principle see it; output-only readers cannot.

Timeline of the field (as compressed by Whittemore)

• 2017–2020 — optimism. Chris Olah's team (then at OpenAI, incl. Cammarata) maps image models like Inception v1 detector by detector; Olah's phrase "AI is grown more than built."
• 2020–2023 — the tangle. Many detectors fire for unrelated things (the famous cat-face/car-front neuron). The field adopts superposition — more concepts than neurons, so neurons are shared — as the explanation, and concludes individual neurons are meaningless.
• 2023 — the 2% result. OpenAI's first automated-interpretability project (GPT-4 explaining GPT-2's neurons) finds ~2% cleanly legible.
• 2023–2025 — unscramblers. Sparse-autoencoder-style tools trained to pull concepts back apart absorb much of the field's effort; some progress, few decisive wins. Mood by 2025: "maybe too hard to solve in time to matter."
• Oct 2026 — "already clean." Cammarata's essay argues the tangle was mostly in the tools (feature-visualisation cheating, a badly trained 2014 reference model, unrecognised branching) and that ~90% of neurons in modern LLMs have one main job. Preliminary, self-reported, contested — see the source page for the critics.

The live disagreement

Position	Who	Claim
Already clean	Cammarata (Goodfire), enthusiastic OpenAI staff	Most neurons legible; bigger models cleaner; "ambitious interpretability will be solved by the time we need it most"
Interesting, not a big-picture update	Neel Nanda (Google DeepMind)	Facts outnumber neurons; weak activations carry superposition; "one job" bar too generous; parts-readable ≠ decisions-readable
Narrow target	Liv Gorton	Essay attacks a narrower superposition than researchers hold; branching and cramming can coexist
Hype until proven	Stephen Casper, Eliezer Yudkowsky	Nothing replicated yet

Even critics concede: neurons are more useful than assumed, old results deserve rechecking, and agentic tools reading neurons at scale is promising for practical debugging ("why did the model do a sketchy thing?").

Why an enterprise reader should care

• Assurance is output-based today. Every vendor safety claim an IT leader sees rests on evals and red-teaming. Interpretability is the only route to assurance that doesn't depend on the model behaving the same when unobserved. Progress here changes what "we tested it" is worth.
• It may become a tool, not just a science. If agentic interpretability works at scale, "explain why the model did X" becomes a debugging primitive for enterprise deployments, independent of the alignment payoff.
• Cadence signal. Cammarata's "bitter lesson for interpretability — the simple thing at absurd scale works" would put interpretability on the same scaling curve as capability, which is the opposite of the 2025 consensus.

Cross-references

• Superposition (Neural Networks) — the theory under dispute, and branching as the rival account
• Nick Cammarata · Goodfire — author and lab
• Autonomous AI Cyberattack — the safety incidents that make interpretability urgent
• AI Optimism vs AI Pessimism (AI Daily Brief) — the discourse arc

Sources

• A Potentially Historic AI Breakthrough (AI Daily Brief) — Whittemore's explainer of "Neural Networks Are Already Clean" (Oct 2026)