Nick Cammarata argues neural networks are already legible, not tangled by superposition

Early knots look ordinary

Nick Cammarata of Goodfire argues that the famous view of neural networks as tangled by superposition was mostly wrong. The problem, he writes, was not a second learning principle that folds unrelated concepts together. It was recursive generalization, where a neuron grows related offshoots much like a branch grows sub branches.

In InceptionV1, once treated as the field’s model organism, weak architecture and weak interpretability tools made late layers look arbitrary. After ordinary architectural improvements, with no change to model size or training data, he reports that 99 to 100 percent of neurons in the last layer are easily interpretable, and versions trained across a 300 times size range stayed broadly legible. With less room, neurons cover coarser concepts; with more room, they split into finer ones.

The canonical cat and car neuron, 4e:55, is better read as a V shape detector. It responds to pointed inputs such as flames, water jets from a fireboat, fox and coyote faces and martini glasses, while round objects push it negative. The car into dogs circuit from Zoom In reflected shared pose across cars and dogs, not a high dimensional hiding place for unrelated features. Cammarata says he has not seen another published circuit with that shape.

Modern models look cleaner

Earlier language model studies looked bleak. The 2023 Anthropic work on a small one layer model gave the median neuron 0 on a 14 point rubric, while an OpenAI automated interpretability project from the same year found about 1.7 percent of neurons interpretable by its most generous measure. With better tools and modern models, preliminary audits of Qwen3-32B estimate that roughly 90 percent of neuron poles have one main job, about 7 percent have two related jobs, and only 2 to 4 percent are not yet understood. Dense models and mixture of experts models look similar, and the author says comparable results appear in models up to trillions of parameters, including Kimi K3.

He describes the gap as compounding ordinary problems: explainer models in 2023 could not run conjecture and test well, metrics rewarded explanations that simulated activations rather than correct ones, researchers often looked for the wrong neuron motifs, and many studies used small or simplified models. As understanding fills in, each neuron is constrained by its neighbors, upstream construction, downstream steering effects and, in mixture of experts models, its specialist expert. The bad news is scale, he says, but larger models also learn narrower neurons and give the map tighter crossings. The essay’s claim is not that language models are solved, but that rapid normal science on their own mechanisms may be far more tractable than assumed.

Sources: already-clean.goodfire.com