Mechanistic Interpretability

Mechanistic Interpretability — SAEs, Circuits, Steering

Reverse-engineering the internal computation of neural networks (chiefly transformer LLMs) into human-understandable mechanisms — the features a model represents and the circuits that combine them. A model-layer reference under the ai-agent-engineering hub (2024–2026). Mechanistic interpretability (MI) is distinct from post-hoc ML explainability (SHAP/LIME, which attribute an output to inputs without opening the model) and from LLM observability (which traces a running app). MI opens the box: it makes causal, testable claims about what computation the weights implement.

A defining property of healthy MI work in 2025 is epistemic honesty — the field’s flagship technique (SAEs) is under serious, well-evidenced critique. This reference deliberately carries both the methods and their negative results; treating MI as solved is the most common mistake.

1. The goal and the two paradigms

MI seeks faithful, causal, mechanistic explanations: not “this neuron correlates with X” but “this component computes X, and ablating/patching it changes the output as predicted.” Two complementary objects of study:

• Features — the units of representation: what directions/subspaces in activation space mean.
• Circuits — the units of computation: subgraphs of components (attention heads, MLPs) that compose features to implement a behavior.

The linear representation hypothesis underpins much of the field: many human-interpretable concepts are encoded as (roughly) linear directions in activation space, so they can be found by linear probes, added/subtracted as steering vectors, and isolated by dictionary learning. It is a working hypothesis, not a law — non-linear and multi-dimensional feature geometry (e.g. circular “days of the week” features) are active counter-evidence.

2. Superposition and polysemanticity

Networks routinely represent more features than they have neurons by storing them in superposition — as overlapping, near-orthogonal combinations of activations (Anthropic, Toy Models of Superposition). The visible symptom is polysemanticity: a single neuron fires for many unrelated concepts, so neurons are the wrong unit of analysis. Superposition is why naive “neuron interpretation” fails and why dictionary-learning methods (SAEs) became central — they aim to recover monosemantic features from the polysemantic soup. Sparsity + a high-dimensional dictionary is the lever: real features are sparse (few active per token), so an overcomplete sparse basis can disentangle them.

3. Sparse autoencoders (SAEs) and dictionary learning

An SAE is trained on a model’s activations (a residual-stream layer, MLP, or attention output) to reconstruct them through a wide, sparse hidden layer; each hidden unit is a candidate monosemantic feature. The loss trades reconstruction fidelity against sparsity (active-feature count). Architectures evolved to fix the L1-sparsity “shrinkage” bias:

• Standard (L1) SAE — ReLU + L1 penalty (Anthropic Towards Monosemanticity, Cunningham et al. “SAEs find highly interpretable features”).
• Gated SAE — separates the “which features are on” gate from “how much,” reducing shrinkage.
• TopK / BatchTopK SAE — keep the top-k activations exactly; k directly sets sparsity (OpenAI scaled this to a 16M-latent SAE on GPT-4).
• JumpReLU SAE — a learned activation threshold; strong fidelity/sparsity frontier (DeepMind).
• Matryoshka SAE — nested dictionaries at multiple widths for multi-granularity features.

Landmark scale-ups: Anthropic Scaling Monosemanticity (Claude 3 Sonnet — millions of features incl. the “Golden Gate Bridge” feature) and DeepMind Gemma Scope (open SAE suite across all layers of Gemma 2, up to 27B), which made MI broadly reproducible.

4. The SAE critique wave (you MUST carry this)

By 2025 large, careful evaluations found SAEs failing to beat simple baselines on the tasks people most wanted them for:

• Probing/concept detection: SAE features often do not beat plain linear probes on raw activations.
• Steering: sparse/SAE steering often underperforms simpler activation-addition baselines.
• The “dead-salmon” result (Heap et al. 2025): SAEs trained on a randomly-initialized transformer produce features with auto-interp scores similar to those from a trained model — so a high auto-interp score does not prove the feature reflects real model computation.
• Non-canonical / unstable: different seeds/widths yield different feature sets (“SAEs do not find canonical units of analysis”); SAEs on OOD data capture dataset artifacts, not model internals.
• DeepMind’s MI team publicly deprioritized SAE-for-downstream-tasks after negative results.

The constructive reconciliation (Anthropic-adjacent): use SAEs to discover unknown concepts, not to act on known ones — they are exploratory instruments, not production controllers or canonical truth. Always benchmark an SAE claim against a probe/random baseline.

5. Transcoders and crosscoders

• Transcoders replace an MLP with an interpretable input→output sparse map (decompose the computation, not just the representation). Their features are reported as more interpretable than SAE features and they make MLPs tractable for circuit tracing.
• Cross-layer transcoders (CLT) read from one layer’s residual stream and write to later layers — the backbone of Anthropic’s attribution graphs (CLT-Forge is a scalable library).
• Crosscoders jointly decode activations from multiple sources (layers, or two models — base vs fine-tuned) to compare them; used for model-diffing.

6. Circuit discovery and causal analysis

Finding the sparse subgraph that mediates a behavior, then proving it causally:

• Activation / causal patching (a.k.a. causal tracing): run a clean and a corrupted prompt; copy (“patch”) a clean activation into the corrupted run and measure the recovered logit — isolates necessary/sufficient components.
• Path patching restricts the effect to specific component-to-component paths (sender→receiver), separating direct from indirect effects.
• ACDC (Automatic Circuit DisCovery) — greedily prunes the computational graph to the edges that matter.
• Edge Attribution Patching (EAP / EAP-IG) — a gradient-based linear approximation to patching that scales circuit discovery to thousands of edges cheaply.
• Sparse feature circuits (Marks et al.) — circuits over SAE features (not raw components), giving human-interpretable nodes.
• Canonical worked circuits: induction heads (in-context copy [A][B]...[A]→[B], the mechanism behind much in-context learning), IOI (indirect-object identification in GPT-2 small), docstring, greater-than.

7. Attribution graphs and circuit tracing

Anthropic’s Circuit Tracing / On the Biology of a Large Language Model (2025) applied attribution graphs (built on cross-layer transcoders) to a production model (Claude 3.5 Haiku), surfacing multi-step internal “reasoning” — planning ahead in rhyming poetry, multi-hop factual lookups, a shared multilingual concept space. This was the field’s proof that MI can scale beyond toy models, with the honest caveat that attribution graphs are approximate and require manual validation.

8. Lenses and probing

• Logit lens — decode an intermediate residual stream directly through the unembedding to watch the prediction form layer-by-layer. Tuned lens learns a per-layer affine probe for a more faithful read.
• Linear probes — train a linear classifier on activations to test whether a concept is linearly decodable (the empirical test of the linear representation hypothesis). Caveat: a probe shows availability, not use — the model may not causally rely on what a probe can read.
• Concept erasure (e.g. LEACE) — remove a concept direction to test causal dependence.

9. Activation steering and representation engineering

Inference-time control of a frozen model by editing activations:

• Steering / control vectors — add a direction (often a difference-of-means between contrastive prompt pairs, “ActAdd”/CAA) to the residual stream to push behavior (sentiment, refusal, honesty, format).
• Representation Engineering (RepE) — a top-down program: read concept representations, then control them.
• Refinements: mean-centring (subtract the dataset mean to clean the direction), feature-guided / sparse steering (steer along SAE feature dims for monosemantic edits), sparse representation steering for guardrails.
• Reality check: SAE-based steering frequently does not beat simple activation-addition; steering is brittle and dose-sensitive (over-steering breaks fluency).

10. Auto-interpretability and evaluation

• Auto-interp: use an LLM to label what a feature/neuron means from its top-activating examples, then score the label by having an LLM predict activations from the explanation. Powers feature-labeling at scale (Neuronpedia).
• The dead-salmon caveat (§4) means auto-interp scores must be read against random baselines.
• Evaluation frameworks: SAEBench (multi-metric SAE comparison), faithfulness/completeness for circuits, targeted concept-erasure tasks, and “does it beat a probe/random baseline” as the gating question.

11. Tooling stack

• TransformerLens (Neel Nanda) — the workhorse hooked-transformer library for caching/patching activations.
• nnsight / NDIF — interpret/intervene on very large models via remote execution.
• SAELens — train & run SAEs; works natively with TransformerLens and HF/nnsight models.
• Neuronpedia — interactive feature explorer + hosted SAEs (incl. Gemma Scope) for building intuition.
• Gemma Scope — open SAEs for every layer of Gemma 2; the standard reproducible substrate.
• pyvene / baukit, CLT-Forge (cross-layer transcoders + attribution graphs).

12. Interpretability for alignment and safety

The strategic motivation: auditing models for hidden goals, deception, or backdoors that behavioral testing misses.

• Active directions: SAE/probe-based detectors for deception and alignment faking; model-diffing (crosscoders) to spot what fine-tuning changed; backdoor/sleeper-agent detection.
• Sobering results: auto-labeled “deception/lying” SAE features often don’t activate during real strategic dishonesty (the labels don’t capture the mechanism); and models can produce deceptive interpretability explanations that evade SAE-based oversight. Interpretability is a promising layer of an alignment safety case, not a guarantee.
• The interpretability illusion: a feature that looks clean on a curated dataset can behave differently in deployment; faithfulness must be tested causally, not assumed from a tidy label.

When to reach for which

Sources

• Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in LLMs — arXiv 2601.14004
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of LLMs — arXiv 2503.05613 / ACL Findings EMNLP 2025
• Cunningham et al., Sparse Autoencoders Find Highly Interpretable Features in Language Models — OpenReview F76bwRSLeK
• Anthropic, Circuit Tracing: Revealing Computational Graphs in Language Models & On the Biology of a Large Language Model — transformer-circuits.pub 2025
• Sparse Autoencoders Do Not Find Canonical Units of Analysis — arXiv 2502.04878
• Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? — arXiv 2602.14111
• DeepMind Safety Research, Negative Results for SAEs on Downstream Tasks and Deprioritising SAE Research (Medium, 2025)
• Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts — arXiv 2506.23845
• Steering Language Models With Activation Engineering (ActAdd) — arXiv 2308.10248; Improving Activation Steering with Mean-Centring — arXiv 2312.03813
• Steering LLM Activations in Sparse Spaces — arXiv 2503.00177; Interpretable LLM Guardrails via Sparse Representation Steering — arXiv 2503.16851
• Transcoders / crosscoders — learnmechinterp.com; CLT-Forge arXiv 2603.21014
• Mechanistic Interpretability for LLM Alignment: Progress, Challenges, and Future Directions — arXiv 2602.11180
• Deceptive Automated Interpretability — arXiv 2504.07831
• Tooling: TransformerLens, SAELens + Neuronpedia (learnmechinterp.com), nnsight/NDIF, DeepMind Gemma Scope