Black Box, Then and Now

For a decade, the standard response to AI opacity has been to try to peer inside the black box. But frontier models are now opaque even to their builders, and explainability is hitting a ceiling. What if instead of trying to explain how the box works, system builders instead focus on designing control architectures that make opacity tolerable?

Image generated by GPT Image 2, 2026.

Black Box theory is, however, even wider in application than these professional studies. The child who tries to open a door has to manipulate the handle (the input) so as to produce the desired movement at the latch (the output); and he has to learn how to control the one by the other without being able to see the internal mechanism that links them. In our daily lives we are confronted at every turn with systems whose internal mechanisms are not fully open to inspection, and which must be treated by the methods appropriate to the Black Box.

— Ashby, An Introduction to Cybernetics, p. 86

Explainability and its limits

In contemporary AI discourse, a “black box” refers to deep models with many hidden layers where inputs and outputs are known, but the internal decision-making is hidden, making the models hard to audit and hard to trust. Explainable AI (known as XAI, not to be confused with the AI company “xAI”) offers techniques, such as LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017), to peek inside these boxes, revealing why a decision was made. These techniques are perceived as crucial for fairness, debugging, and compliance, especially in high-stakes areas like medicine or finance, even though explainability sometimes comes at the cost of peak performance. For example, a SHAP analysis might show how much each input feature (e.g., salary, credit score) contributes to a prediction (e.g., likelihood of repayment of a loan); and a LIME analysis might approximate aspects of a more complex model with simpler, more interpretable one (e.g., a linear regression model).

But this focus on explainability has begun to hit structural limits. Modern models keep growing in complexity, exceeding the reach of any post hoc rationale, while models with built-in interpretability (Rudin, 2019) are increasingly uncompetitive with state-of-the-art models. Meanwhile, most modern AI innovation is concentrated in commercial labs that are increasingly unforthcoming about the inner workings of their models (Stanford HAI Index). On the other hand, many chat systems that interact with foundation models now include “reasoning traces” feedback meant to help the user have some understanding of the tools, lines of thought, web pages, etc., that the system uses in deriving a response to a user prompt. However, this feedback typically amounts to short, rapidly cycling phrases that provide only the most cursory information about the true set of executions under-the-hood.

A new kind of black box

From a cybernetics perspective, the effort to fully explain these systems was always going to hit a ceiling. Arguably around 2023-24, frontier AI models crossed a threshold: they became classic black boxes, not just to their users, but in important respects to their builders. This is a stronger claim than Ashby’s. The door handle is opaque to the child but transparent to the engineer who made it; a frontier model can be opaque to everyone at once. After all, who can explain precisely how their phone, car, laptop, monitor, watch, etc., works, much less the far more complex and interacting systems that govern everyday infrastructure that supports the banal cadence of everyday life (e.g., power and water procurement and distribution)? Concerns focused on the inner workings of those systems are important, but only to the extent to which they impact relevant external (and monitorable) systems.

At first blush, this stance could appear worrisomely indifferent. What if, for example, these systems, executing some unknown algorithm, act in ways that are inappropriate when given some new input? In fact the cybernetic stance demands more caution, not less. Precisely because these systems are never going to be fully knowable from the inside, they must always be embedded in architectures of control from the outside. There must always be a regulatory system in place that mediates between a system’s outputs and the others it interfaces with.

This is, in fact, the direction production AI is already moving. Serious deployments rarely expose a single model directly to users. Instead, they nest models inside one another: a narrowly-scoped classifier polices the output of a more capable generator; a separate model checks for policy violations before anything reaches the interface; logs and rate-limits sit outside the whole stack. None of these guardrails require understanding why the underlying model produced what it produced. They only require monitoring what it produced and intervening when needed.

A complement to XAI

This is the cybernetic complement to the XAI program. The goal of XAI is to make the box transparent enough to trust. Cybernetics instead focuses on a regulator around the box so that its opacity doesn’t matter.

The irony may be that designers who give up on explanation will feel the need to build systems with more regulatory controls. When you cannot inspect the engine, you add instrumentation and sensors, design in redundancies, improve logging and feedback, etc. You define acceptable outputs and enforce them.

In the end, presuming less control may force better overall accountability.