When Explanation Stops Explaining
On what both humans and AI may be quietly losing
The Edge of Understanding
In recent years, artificial intelligence has become extraordinarily good at explaining things. It can summarise complex arguments, justify decisions, generate coherent models, and produce persuasive reasoning at scale. In many professional contexts, this explanatory fluency is now taken as a proxy for understanding. I want to suggest, cautiously, that this assumption deserves closer scrutiny — not because explanation is unimportant (it is indispensable), but because explanation may be becoming the only form of understanding we trust. That narrowing carries risks, for humans and machines alike.
Across science, technology, education, and policy, we increasingly reward clarity, explicit justification, formal reasoning, and clean models. These values have driven genuine progress and remain the backbone of empirical work. But they are not the whole of cognition. Much human understanding operates prior to, or alongside, explanation. We often recognise that something is wrong, incomplete, or misaligned before we can say why. We rely on metaphor, image, rhythm, and a felt sense of coherence long before we can produce an argument. These pre-formal signals are not irrational; they are how cognition maintains calibration when formal reasoning begins to outrun reality. When explanation is treated as the sole admissible form of understanding, these signals are quietly downgraded or ignored.
This is not foreign to science itself. Many major scientific advances did not begin as deductions from established theory, but as shifts in how a problem was seen. Newton’s insight was not simply a calculation but a re-framing of motion and force; Einstein’s breakthroughs relied on thought experiments and imaginative reorientation long before formal derivation. In such cases, articulation followed insight — it did not generate it. Explanation came later, as a way of stabilising and communicating something that first appeared in a non-propositional form. This is not romanticism; it is a sober reading of scientific history.
Large language models are trained on human language and human symbolic practices. That is precisely why they are so effective. They operate inside the same semantic ecosystem that humans use to reason, explain, and justify. But that shared semiosphere also imposes limits. Language, however refined, presupposes grammar, propositions, and closure. Even the deepest verbal metaphors remain substitutions within language. They can move meaning sideways, but not indefinitely downward. At a certain point, additional structure does not clarify; it obscures. Many readers will recognise the experience: a diagram, model, or formally correct explanation that somehow makes the underlying issue harder to grasp rather than easier. This is not a failure of intelligence. It is a sign that explanation has exceeded its appropriate domain.
This has practical consequences. Diagrams and formal models are excellent at representing mechanisms and processes, but they are far less effective at representing epistemic conditions: what is assumed rather than observed, what merely appears valid, and where interpretation quietly substitutes for evidence. When an argument concerns how we may be misled by our own methods, adding structure can increase cognitive load rather than reduce it. In such cases, carefully staged prose — hedged, reflective, and explicit about uncertainty — often does more work than visual abstraction. This is not an argument against models or diagrams, but for recognising when they are the wrong tool.
The concern here is not mystical, and it is not about attributing inner lives to machines. It is about epistemic balance. Human cognition has always relied on layers that are difficult to formalise: intuitive unease, metaphorical resonance, aesthetic coherence, and pre-verbal judgement. These are not decorative additions to reasoning. They are how we detect misfit, overconfidence, and false precision. If we train ourselves — and our machines — to trust only what can be explicitly justified, we risk losing access to those diagnostic layers. We become very good at explaining systems that no longer quite correspond to human experience.
In artificial systems, this shows up as models that are internally coherent, statistically impressive, and operationally brittle. In human systems, it shows up as confidence that outpaces understanding. None of this implies abandoning rigour or embracing obscurity. On the contrary, it is an argument for protecting the conditions under which rigour remains meaningful. Scientific progress has always depended on pre-formal ways of seeing: orientations that made explanation possible rather than the other way around.
As we increasingly build machines that mirror our own explanatory habits, we should pay attention to what those habits exclude. The question is not whether AI can explain — it already can. The question is whether we are preserving the forms of understanding that tell us when explanation is no longer enough. If we lose those forms in ourselves, artificial intelligence will not recover them for us. If we preserve them, AI may help us notice their absence more clearly. That is not a prediction. It is a caution — and cautions, by their nature, are easiest to dismiss and hardest to replace once ignored.
© John Rust, December 2025, All Rights Reserved