evaluationsIn brief · · · Written with AI
A reply questions whether neural networks are already 'clean'
A Goodfire essay argues that neurons are easier to interpret than believed. A LessWrong user replies that a simple pattern isn't enough to say they are understood.
Nick Cammarata of Goodfire argues in an essay that the neurons of neural networks are easier to interpret than previously thought. In a preliminary audit of Qwen3-32B, he says, about 90% of neuron poles have one main job, 7% have two related jobs, and 2–4% are not yet understood. He presents it as a working hypothesis and announces a companion technical paper.
On 17 October, the user leogao replied on LessWrong. The result, they argue, justifies updating, but a simple pattern in the examples that activate a neuron most doesn't prove the neuron is understood, because weak activations may also matter. They propose replacing real activations with simulated ones and measuring whether the model gets worse. It is an opinion, not a study.
Sources
Written by AI from these sources.
- Neural Networks Are Already CleanAccessed 17 October 2026 · Image: Goodfire
- Neural networks are not already cleanAccessed 17 October 2026 · Image: LessWrong
