Mechanistic Interpretability

What you’ll learn

Almost all participants should understand:

• The Goals of MechInterp
• What is meant by feature & circuit
• Have an overview of currently used methods for feature and circuit discovery
• Have an overview of our current capabilities/frontier of knowledge in MechInterp
• Be able to engage with/understand common critiques of MechInterp

Some participants should be able to:

• implement common mechinterp methods
• Know when to apply what method
• read and fully understand new MechInterp publications
• Be a useful discussion partner to MechIntep researchers

Why

Mechinterp is a central field within AI safety. I think everyone should have a rough understanding of the current state of MechInterp, to have an idea where it intersects with their own research direction.

For example: how whitebox methods are useful for Ai controls, or how mechinterp findings compare with devinterp findings.

I hope that some participants will take this day as a kickoff point to pursue a project or a career in MechInterp. For these participants, this day should give them enough of a taste of what MechInterp looks like to wake their interest and an overview to orient them towards currently active research directions and open problems

How

We go through a long lecture, getting to know the major developments in MechInterp so far, particularly looking at recent results and methodological developments

We take turns between lectures, and technical exercises, ending the day in a discussion if the time suffices.

Lecture notes and materials are self-contained within this repository: https://github.com/iliad-team/iliad-intensive-C.2

Prerequisites

• Knowing the basics of Machine Learning
• Familiarity with the Transformer Architecture
• Basic maths / linear algebra

Content

Fast Track

Go through the slides below.

Main Content

Material:

Slides

Exercises and external links (in lecture order)

• Feature Visualization
• Logit Lens: normal hard
• Neuronpedia — SAE features
• Sparse Autoencoders
• Neuronpedia — Attribution graphs
• Induction Heads: normal hard
• Neuronpedia — Natural Language Autoencoders

Discussion reading

• Nanda et al., A Pragmatic Vision for Interpretability
• Ségerie, Against Almost Every Theory of Impact of Interpretability
• Hendrycks, The Misguided Quest for Mechanistic AI Interpretability
• Chughtai, Activation Space Interpretability May Be Doomed

Intent:

• The lecture gives an overview of the current state of MechInterp, developments over the last years, and commonly used methods
• The technical exercises are intended to deepen the understanding of the techniques and give a flavour for how mechinterp work looks like
• The neuropedia exercises are supposed to de-cherrypick mechinterp results and give a flavour for the average feature/circuit/NLA-oputput and show that a lot of the mechinterp results are still confusing, hard to interpret, and noisy

Content:

• Complex features and circuits (the car detector circuit)
• Universality in image models (curve and high-frequency detectors)
• Natural abstractions discourse (why certain features might be universal)
• Polysemanticity in CNNs
• Features in language models and the lack of a privileged basis
• Word embeddings encoding human concepts (word2vec gender direction)
• Linear probes for classifying text properties
• Probing world models (Othello-GPT board state)
• Intervening on world models
• Probing LLMs for truthfulness (attention head activations)
• Steering models with contrastive activation vectors
• Ablating features and the refusal direction (activation addition, direction ablation, weight orthogonalization)
• The logit lens
• Superposition (privileged vs non-privileged bases)
• Johnson–Lindenstrauss lemma and the scale of superposition
• Anthropic's toy model of superposition
• Feature geometry and phases (sticky regions, dimensions per feature)
• Semantic feature geometry (circular representations of time / non-linear features)
• Dictionary learning theory (Arora et al.)
• Compressed sensing and L1 minimization (Candès/Donoho)
• Sparse autoencoders (SAEs) and their training objective
• Interpreting SAE features (activation, logit effects, steering)
• Neuronpedia exploration
• SAE problems: feature absorption
• SAE problems: feature shrinkage
• The "Cambrian explosion" of SAE variants
• Gated SAEs
• JumpReLU SAEs
• Top-K / Batch Top-K SAEs
• Matryoshka SAEs
• Staircase SAEs
• Comparing/evaluating SAE architectures (loss recovered, auto-interp, absorption, SCR, k-sparse probing, RAVEL)
• SAEs scaling to production models (Sonnet 3.5)
• Bigger models yielding more specific features (golden gate bridge feature)
• Reading Claude's mind / safety-relevant feature steering (eval-awareness features)
• Transcoders and cross-layer transcoders (expanding SAEs)
• Model diffing
• Transformer circuits
• Reverse-engineering modular addition (the grokking / mod-113 circuit)
• Fourier analysis of the embedding matrix
• Plotting attention patterns
• Logit attribution and induction heads
• Q-composition and K-composition
• Path patching and the IOI (indirect object identification) circuit
• Automated circuit discovery (ACDC)
• Circuit tracing with cross-layer transcoders and replacement models
• Attribution graphs and feature suppression
• Circuit tracing on production models (Haiku 3.5 math circuit)
• Using circuit tracing for alignment (misaligned/reward-hacking model)
• Generality findings across circuits (layer roles, default pathways, shortcuts, special tokens, hedging features)
• Fuzzy/less-faithful interpretability approaches
• Activation oracles
• Natural language autoencoders (verbalizer + reconstructor)
• Open problems in mech interp (Sharkey et al.): decomposition, description, validation, automation, application
• Critiques of mechanistic interpretability (discussion of critical papers)