Patching predicted neuron activations based on Neuron Scientist descriptions into an ablated model recovers 74% of the damage measured by Kullback-Leibler divergence

Overview Debates Related claims Topics

The Neuron Scientist agent generates paragraph-length hypotheses describing the specific conditions under which an individual neuron activates

Ablating 80% to 90% of neurons in the Qwen 3.6-35B A3B language model improved performance on ARC-AGI benchmark puzzles

Nick Cammarata authored the essay Neural Networks are Already Clean

Noam Shazeer proposed the SwiGLU gated multi-layer perceptron architecture in a 2020 paper

The Neuron Scientist agent utilizes experimental tools including generating counterfactual sentence pairs and steering activations to refine neuron hypotheses

Specific reasoning neurons in language models fire to modulate confidence, induce delays, trigger self-checking, or pivot problem-solving directions

Kurt Tigges and Nick Cammarata co-authored the research paper Neurons as an Interpretable Basis

Ablating approximately 90% of individual neurons by retaining only extreme activations causes minimal performance degradation on standard language model benchmarks

Individual neurons in language model multi-layer perceptrons can express a compact set of distinct meanings rather than hundreds of unrelated meanings

Selecting the top 64 activating neurons per expert in Qwen 3.6-35B A3B ablates approximately 88% of all neuron activations

Individual neurons in language models frequently function as verbs or control flow operators rather than static entity detectors

Swapping double-negative activations from an unrelated text sequence can restore reasoning capabilities in an ablated Qwen 3.6-35B A3B model

Neural networks employing ReLU or GELU activation functions are more difficult to interpret than networks using SwiGLU activation functions

ReGLU gated multi-layer perceptrons use the ReLU activation function instead of SiLU, preventing the occurrence of double-negative activation values

Double-negative activations occur in SwiGLU when the gate projection and up projection both produce negative values that multiply into a positive product

Ablating individual neurons within attribution circuits can identify whether specific input tokens cause a persona steering vector to fail

In SwiGLU multi-layer perceptrons, distinct sets of downstream reader neurons activate depending on context to parse polysemantic neuron outputs

Transcoders applied to multi-layer perceptron layers predominantly recapitulate individual neurons or sparse combinations of co-activating neurons

Individual neurons differ in the activation percentile thresholds at which their semantic coherence dissolves into noise

In SwiGLU multi-layer perceptrons, each neuron receives input via two separate parameter matrices consisting of an up projection and a gate projection