Mechanistic Interpretability

Parent: LLM Models and APIs · researched 2026-06-03T00:22:13.503Z· 16 sources · 12 concepts · skill mechanistic-interpretability

Reverse-engineering the internal computation of neural networks (chiefly transformer LLMs) into human-understandable mechanisms — the features a model represents and the circuits that combine them. A

Mechanistic Interpretability — SAEs, Circuits, Steering

Reverse-engineering the internal computation of neural networks (chiefly transformer LLMs) into human-understandable mechanisms - the features a model represents and the circuits that combine them. A model-layer reference under the ai-agent-engineering hub (2024–2026). Mechanistic interpretability (MI) is distinct from post-hoc ML explainability (SHAP/LIME, which attribute an output to inputs without opening the model) and from LLM observability (which traces a running app). MI opens the box: it makes causal, testable claims about what computation the weights implement. [source]
A defining property of healthy MI work in 2025 is epistemic honesty - the field's flagship technique (SAEs) is under serious, well-evidenced critique. This reference deliberately carries both the methods and their negative results; treating MI as solved is the most common mistake. [source]

1. The goal and the two paradigms

MI seeks faithful, causal, mechanistic explanations: not "this neuron correlates with X" but "this component computes X, and ablating/patching it changes the output as predicted." Two complementary objects of study: [source]
Features - the units of representation: what directions/subspaces in activation space mean. [source]
Circuits - the units of computation: subgraphs of components (attention heads, MLPs) that compose features to implement a behavior. [source]
The linear representation hypothesis underpins much of the field: many human-interpretable concepts are encoded as (roughly) linear directions in activation space, so they can be found by linear probes, added/subtracted as steering vectors, and isolated by dictionary learning. It is a working hypothesis, not a law - non-linear and multi-dimensional feature geometry (e.g. circular "days of the week" features) are active counter-evidence. [source]

2. Superposition and polysemanticity

Networks routinely represent more features than they have neurons by storing them in superposition - as overlapping, near-orthogonal combinations of activations (Anthropic, Toy Models of Superposition). The visible symptom is polysemanticity: a single neuron fires for many unrelated concepts, so neurons are the wrong unit of analysis. Superposition is why naive "neuron interpretation" fails and why dictionary-learning methods (SAEs) became central - they aim to recover monosemantic features from the polysemantic soup. Sparsity + a high-dimensional dictionary is the lever: real features are sparse (few active per token), so an overcomplete sparse basis can disentangle them. [source]

3. Sparse autoencoders (SAEs) and dictionary learning

An SAE is trained on a model's activations (a residual-stream layer, MLP, or attention output) to reconstruct them through a wide, sparse hidden layer; each hidden unit is a candidate monosemantic feature. The loss trades reconstruction fidelity against sparsity (active-feature count). Architectures evolved to fix the L1-sparsity "shrinkage" bias: [source]
Standard (L1) SAE - ReLU + L1 penalty (Anthropic Towards Monosemanticity, Cunningham et al. "SAEs find highly interpretable features"). [source]
Gated SAE - separates the "which features are on" gate from "how much," reducing shrinkage. [source]
TopK / BatchTopK SAE - keep the top-k activations exactly; k directly sets sparsity (OpenAI scaled this to a 16M-latent SAE on GPT-4). [source]
JumpReLU SAE - a learned activation threshold; strong fidelity/sparsity frontier (DeepMind). [source]
Matryoshka SAE - nested dictionaries at multiple widths for multi-granularity features. [source]
Landmark scale-ups: Anthropic Scaling Monosemanticity (Claude 3 Sonnet - millions of features incl. the "Golden Gate Bridge" feature) and DeepMind Gemma Scope (open SAE suite across all layers of Gemma 2, up to 27B), which made MI broadly reproducible. [source]

4. The SAE critique wave (you MUST carry this)

By 2025 large, careful evaluations found SAEs failing to beat simple baselines on the tasks people most wanted them for: [source]
Probing/concept detection: SAE features often do not beat plain linear probes on raw activations. [source]
Steering: sparse/SAE steering often underperforms simpler activation-addition baselines. [source]
The "dead-salmon" result (Heap et al. 2025): SAEs trained on a randomly-initialized transformer produce features with auto-interp scores similar to those from a trained model - so a high auto-interp score does not prove the feature reflects real model computation. [source]
Non-canonical / unstable: different seeds/widths yield different feature sets ("SAEs do not find canonical units of analysis"); SAEs on OOD data capture dataset artifacts, not model internals. [source]
DeepMind's MI team publicly deprioritized SAE-for-downstream-tasks after negative results. [source]
The constructive reconciliation (Anthropic-adjacent): **use SAEs to discover unknown concepts, not to act on known ones** - they are exploratory instruments, not production controllers or canonical truth. Always benchmark an SAE claim against a probe/random baseline. [source]

5. Transcoders and crosscoders

Transcoders replace an MLP with an interpretable input→output sparse map (decompose the computation, not just the representation). Their features are reported as more interpretable than SAE features and they make MLPs tractable for circuit tracing. [source]
Cross-layer transcoders (CLT) read from one layer's residual stream and write to later layers - the backbone of Anthropic's attribution graphs (CLT-Forge is a scalable library). [source]
Crosscoders jointly decode activations from multiple sources (layers, or two models - base vs fine-tuned) to compare them; used for model-diffing. [source]

6. Circuit discovery and causal analysis

Finding the sparse subgraph that mediates a behavior, then proving it causally: [source]
Activation / causal patching (a.k.a. causal tracing): run a clean and a corrupted prompt; copy ("patch") a clean activation into the corrupted run and measure the recovered logit - isolates necessary/sufficient components. [source]
Path patching restricts the effect to specific component-to-component paths (sender→receiver), separating direct from indirect effects. [source]
ACDC (Automatic Circuit DisCovery) - greedily prunes the computational graph to the edges that matter. [source]
Edge Attribution Patching (EAP / EAP-IG) - a gradient-based linear approximation to patching that scales circuit discovery to thousands of edges cheaply. [source]
Sparse feature circuits (Marks et al.) - circuits over SAE features (not raw components), giving human-interpretable nodes. [source]
Canonical worked circuits: induction heads (in-context copy [A][B]...[A]→[B], the mechanism behind much in-context learning), IOI (indirect-object identification in GPT-2 small), docstring, greater-than. [source]

7. Attribution graphs and circuit tracing

Anthropic's Circuit Tracing / On the Biology of a Large Language Model (2025) applied attribution graphs (built on cross-layer transcoders) to a production model (Claude 3.5 Haiku), surfacing multi-step internal "reasoning" - planning ahead in rhyming poetry, multi-hop factual lookups, a shared multilingual concept space. This was the field's proof that MI can scale beyond toy models, with the honest caveat that attribution graphs are approximate and require manual validation. [source]

8. Lenses and probing

Logit lens - decode an intermediate residual stream directly through the unembedding to watch the prediction form layer-by-layer. Tuned lens learns a per-layer affine probe for a more faithful read. [source]
Linear probes - train a linear classifier on activations to test whether a concept is linearly decodable (the empirical test of the linear representation hypothesis). Caveat: a probe shows availability, not use - the model may not causally rely on what a probe can read. [source]
Concept erasure (e.g. LEACE) - remove a concept direction to test causal dependence. [source]

9. Activation steering and representation engineering

Inference-time control of a frozen model by editing activations: [source]
Steering / control vectors - add a direction (often a difference-of-means between contrastive prompt pairs, "ActAdd"/CAA) to the residual stream to push behavior (sentiment, refusal, honesty, format). [source]
Representation Engineering (RepE) - a top-down program: read concept representations, then control them. [source]
Refinements: mean-centring (subtract the dataset mean to clean the direction), feature-guided / sparse steering (steer along SAE feature dims for monosemantic edits), sparse representation steering for guardrails. [source]
Reality check: SAE-based steering frequently does not beat simple activation-addition; steering is brittle and dose-sensitive (over-steering breaks fluency). [source]

10. Auto-interpretability and evaluation

Auto-interp: use an LLM to label what a feature/neuron means from its top-activating examples, then score the label by having an LLM predict activations from the explanation. Powers feature-labeling at scale (Neuronpedia). [source]
The dead-salmon caveat (§4) means auto-interp scores must be read against random baselines. [source]
Evaluation frameworks: SAEBench (multi-metric SAE comparison), faithfulness/completeness for circuits, targeted concept-erasure tasks, and "does it beat a probe/random baseline" as the gating question. [source]

11. Tooling stack

TransformerLens (Neel Nanda) - the workhorse hooked-transformer library for caching/patching activations. [source]
nnsight / NDIF - interpret/intervene on very large models via remote execution. [source]
SAELens - train & run SAEs; works natively with TransformerLens and HF/nnsight models. [source]
Neuronpedia - interactive feature explorer + hosted SAEs (incl. Gemma Scope) for building intuition. [source]
Gemma Scope - open SAEs for every layer of Gemma 2; the standard reproducible substrate. [source]
pyvene / baukit, CLT-Forge (cross-layer transcoders + attribution graphs). [source]

12. Interpretability for alignment and safety

The strategic motivation: auditing models for hidden goals, deception, or backdoors that behavioral testing misses. [source]
Active directions: SAE/probe-based detectors for deception and alignment faking; model-diffing (crosscoders) to spot what fine-tuning changed; backdoor/sleeper-agent detection. [source]
Sobering results: auto-labeled "deception/lying" SAE features often don't activate during real strategic dishonesty (the labels don't capture the mechanism); and models can produce deceptive interpretability explanations that evade SAE-based oversight. Interpretability is a promising layer of an alignment safety case, not a guarantee. [source]
The interpretability illusion: a feature that looks clean on a curated dataset can behave differently in deployment; faithfulness must be tested causally, not assumed from a tidy label. [source]

Sources

Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in LLMs - arXiv 2601.14004 [source]
A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of LLMs - arXiv 2503.05613 / ACL Findings EMNLP 2025 [source]
Cunningham et al., Sparse Autoencoders Find Highly Interpretable Features in Language Models - OpenReview F76bwRSLeK [source]
Anthropic, Circuit Tracing: Revealing Computational Graphs in Language Models & On the Biology of a Large Language Model - transformer-circuits.pub 2025 [source]
Sparse Autoencoders Do Not Find Canonical Units of Analysis - arXiv 2502.04878 [source]
Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? - arXiv 2602.14111 [source]
DeepMind Safety Research, Negative Results for SAEs on Downstream Tasks and Deprioritising SAE Research (Medium, 2025) [source]
Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts - arXiv 2506.23845 [source]
Steering Language Models With Activation Engineering (ActAdd) - arXiv 2308.10248; Improving Activation Steering with Mean-Centring - arXiv 2312.03813 [source]
Steering LLM Activations in Sparse Spaces - arXiv 2503.00177; Interpretable LLM Guardrails via Sparse Representation Steering - arXiv 2503.16851 [source]
Transcoders / crosscoders - learnmechinterp.com; CLT-Forge arXiv 2603.21014 [source]
Mechanistic Interpretability for LLM Alignment: Progress, Challenges, and Future Directions - arXiv 2602.11180 [source]
Deceptive Automated Interpretability - arXiv 2504.07831 [source]
Tooling: TransformerLens, SAELens + Neuronpedia (learnmechinterp.com), nnsight/NDIF, DeepMind Gemma Scope [source]

Children

Superposition & polysemanticity (linear representation hypothesis) (frontier)
Sparse autoencoders & dictionary learning (Gated/TopK/JumpReLU, Gemma Scope) (frontier)
The SAE critique wave (random-Transformer baselines, non-canonical, deprioritization) (frontier)
Transcoders & cross-layer transcoders/crosscoders (frontier)
Circuit discovery (induction heads, IOI, activation/path patching, ACDC, EAP) (frontier)
Sparse feature circuits (frontier)
Attribution graphs & circuit tracing (Biology of an LLM) (frontier)
Logit lens / tuned lens / linear probes / concept erasure (frontier)
Activation steering & representation engineering (RepE, ActAdd) (frontier)
Auto-interpretability & evaluation (SAEBench, Neuronpedia) (frontier)
Tooling (TransformerLens, SAELens, nnsight/NDIF, Neuronpedia, Gemma Scope) (frontier)
Interpretability for alignment & safety (auditing, deception probes, the interpretability illusion) (frontier)

Frontier under this node: Activation steering & representation engineering (RepE, ActAdd), Attribution graphs & circuit tracing (Biology of an LLM), Auto-interpretability & evaluation (SAEBench, Neuronpedia), Circuit discovery (induction heads, IOI, activation/path patching, ACDC, EAP), Interpretability for alignment & safety (auditing, deception probes, the interpretability illusion), Logit lens / tuned lens / linear probes / concept erasure, Sparse autoencoders & dictionary learning (Gated/TopK/JumpReLU, Gemma Scope), Sparse feature circuits, Superposition & polysemanticity (linear representation hypothesis), The SAE critique wave (random-Transformer baselines, non-canonical, deprioritization), Tooling (TransformerLens, SAELens, nnsight/NDIF, Neuronpedia, Gemma Scope), Transcoders & cross-layer transcoders/crosscoders

← the whole tree · 3D view· how to read this page