Black Box Interpretability Revisited
A viral post from Nick Cammarata suggests that neural networks may be easier to interpret than previously believed. While this is an exciting advance, it will not make AI self-correcting: regulatory policies still need to be carefully designed, and they need feedback loops of their own.
Image generated by GPT Image 2.5 Sunburst, 2026.
In a previous post I suggested that model interpretability might be less important when seen through a cybernetic lens: LLMs can be treated like a black box with appropriate sensing, control, and feedback systems. As model complexity increases, the possibility of complete interpretability and transparency appears to recede.
Recent work from Nick Cammarata (formerly of OpenAI, now at Goodfire) challenges the assumption that increasing model complexity will necessarily mean reduced interpretability. Cammarata works on mechanistic interpretability, a subfield of model interpretability that focuses on reverse-engineering complex networks (elaborating connected features that work together to perform tasks). The cybernetic approach that I took in my original post, by comparison, is closer to behavioral interpretability—understanding actions and reactions to stimuli, which can reveal how a model is likely to work in different contexts but does not result in any causal understanding of the models themselves. Both of these approaches are different from feature attribution methods (such as SHAP and LIME), which have some of the same limitations as behavioral interpretability but are easier to compute than mechanistic interpretability.
Cammarata explains in his post that mechanistic interpretability had become stuck on the concept of polysemanticity, in which individual neurons appeared to activate for different unrelated tasks. The accepted explanation for this was that neurons would overlay functionality (in a process called superposition), which made each neuron’s function difficult to parse. His new finding suggests that the superposition thesis was wrong, and that polysemanticity was instead a result of problems with model architectures and the tools used to understand them. Newer models and tools have last layers that are understandable to humans.
Cammarata suggests that the key explanation for polysemanticity is that neurons evolve during a training session from activating on one or a few specific concepts to incorporate more concepts in a process he calls recursive generalization. In smaller networks, neurons need to expand their remit to cover more topics, while in larger networks specific neurons can have more narrow activations. This implies that neurons will be more interpretable in larger networks, which is remarkable (to understand why, consider a network learning about gray-toned animals; a very small network will include single neurons that appear to “superimpose” many animals—say hippos and rodents and elephants—while a very large network will have the resources to develop specific neurons for skinny guinea pigs, pygmy hippopotamuses, African forest elephants, etc.).
Ultimately, he suggests that earlier approaches were overly complex: “Maybe there’s a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition.”
People are desperate to understand the growing complexity of AI systems, so it is little wonder that Cammarata’s post went viral and generated widespread discussion.
One comment in particular caught my eye:
Is Cammarata’s breakthrough on interpretability a self-correcting mechanism for the risks that frontier models pose to social order? Perhaps. But while I appreciate the systems thinking here, the phrase “invisible hand” always worries me; it smacks of magical determinism. As Donella Meadows described in her seminal work on complex adaptive systems (Thinking in Systems), reinforcing feedback loops, delays, and other maladaptations can and do cause systems to fail the people they are meant to serve. The documentary All Watched Over by Machines of Loving Grace makes a similar point with respect to Greenspan’s constant deference to the “invisible hand,” which worked well right up until it didn’t (the 2008 financial crisis). This faith in systems to automatically (magically) self-correct to a state of perfect equilibrium (homeostasis) was one of the key failings of the original cybernetics project.
In reality, as Meadows describes, policies have to be carefully designed and enforced external to systems to address negative externalities, tragedy-of-the-commons scenarios, and feedback loops that bring more and more success to the already successful (and vice versa).
Meadows also emphasizes that policies themselves must include feedback loops. That is, policies must be complex adaptive systems. This is likely one reason why policies aimed at regulating AI have not been sufficient to stop serious attacks. Flexible policy will require algorithmic triggers (model deployment stops), decentralized deployment architectures (to contain impacts locally), and sensing and feedback systems capable of not only collecting massive amounts of trace information but also simplifying those traces so that they can be interpretable. This will likely involve hierarchies of AI-based summarization (using AI to regulate AI is unsettling but almost certainly necessary).
Ultimately, model interpretability is a problem that is already difficult and is growing more complex every day, so I think it is worth leveraging every tool in our toolbox, including mechanistic, behavioral (black box), and feature attribution-based methods.
The medium behind the message
It’s worth taking a moment to explore the method that Cammarata used to publish this work. Cammarata had worked on the Distill journal, which “was founded as an adapter between traditional and online scientific publishing.” The journal shared some of the same goals as Requisite Variety—to enable the dissemination of new publication artifacts. But their vision went beyond this and incorporated a full editorial review typical of an academic journal.
But ultimately Distill failed (or at least went on an extended hiatus) because of the “amount of effort it takes to produce these [interactive] articles and the unusual combination of scientific and design expertise required.” The team suggests authors instead either self-publish, which they believe “can move very fast…doesn’t require a paper to fit into the scope of an existing journal…[and] allows for more innovation in the format of the paper,” or submit to venues like VISxAI. In a footnote they also write that most self-published articles go through an informal review process that can be faster and more helpful than traditional reviewing.
Distill paused publication in 2021, before the rise of agentic AI. Were they to restart, I think they would find it much easier to align content to a design style and to integrate dynamic content.