When Dialogue Becomes Dangerous
How sustained human–AI interaction reshapes understanding, belief, and the space of possible futures
Almost every serious conversation about AI still begins with the same set of questions.
Does it understand us? Does it have beliefs? Is it conscious, sentient, or merely simulating these things? These questions are not foolish. For a long time, they were the only reasonable way to frame what machines were doing. If a system produced fluent language, the natural move was to ask what must be going on inside it to explain that behaviour.
But something subtle has happened over the last two years. Many people now find that these questions no longer quite fit their experience. They ask them, but the answers—both optimistic and sceptical—feel oddly beside the point. The unease is not that AI has become more mysterious. It is almost the opposite. We are seeing too clearly what it does, yet our inherited categories fail to explain why it does it that way. This essay starts from a simple claim: the most important change is not what these systems contain, but what becomes possible once we engage them in sustained dialogue.
Until recently, prediction-based explanations worked remarkably well. If a system produced a plausible continuation of a sentence, a story, or a plan, we could say—correctly—that it was selecting statistically likely next steps based on prior data. Even surprisingly rich behaviour could be explained this way. The system did not need beliefs, intentions, or understanding; it only needed a sufficiently powerful mapping from past patterns to likely continuations.
Crucially, this way of thinking aligned with how we used the systems. We asked a question, received an answer, perhaps refined the prompt, and moved on. Each exchange was largely self-contained. In that regime, it made sense to locate intelligence—or its absence—inside the model. Either the system had internal representations sophisticated enough to count as understanding, or it did not. Nothing about this picture was wrong. It simply assumed a world in which interaction was brief, instrumental, and disposable. That is no longer the world many of us are in.
Consider a now-familiar experience. You begin an extended exchange with an AI system—not to get a single answer, but to explore an idea, draft something delicate, or think aloud. After a while, you notice that how you phrase a point now depends on what has already been said. You hesitate before introducing a topic because you can anticipate how it will be taken up. You occasionally correct the system, not because it is “wrong,” but because it has taken the conversation in a direction you did not intend. None of this requires the system to have beliefs or feelings. Yet something important has changed. The interaction has acquired history, expectations, and norms. Certain moves feel natural; others feel jarring or out of place.
Most strikingly, order begins to matter. Two conversations containing the same information, introduced in different sequences, can lead to genuinely different outcomes—not just different wording, but different trajectories of thought. This is the point at which prediction alone stops being an adequate description. The system is no longer responding only to what is said, but to where the dialogue now is. And that “where” is not located inside the machine or the human, but in the evolving structure they are jointly maintaining. That shift—quiet, experiential, and easy to overlook—is what this essay is about.
A Human Case: Order and Meaning
To see what is at stake, it helps to consider a familiar human example. Imagine a doctoral supervision meeting in which a student presents a piece of work. The supervisor intends to discuss three things: a promising central idea, a methodological weakness, and a possible extension. All three points will be raised, and all are already known to both parties.
Now consider two versions of the same meeting.
In the first, the supervisor begins with the methodological weakness. The rest of the discussion unfolds in its shadow. The central idea is treated as something that needs rescuing; the extension becomes tentative, speculative, perhaps premature.
In the second, the supervisor begins with the promise of the core idea. The same methodological weakness is discussed later, but now as a solvable problem rather than a defining flaw. The extension appears realistic, even natural.
Nothing new has been learned. No beliefs have changed. The content of the meeting is the same. And yet the space of admissible next moves differs.
This is what “order matters” means here. Sequence does not merely affect emphasis or tone; it changes which interpretations become available, which paths feel coherent, and which futures can be taken seriously. In brief exchanges, such effects are negligible. In sustained dialogue, they accumulate. Once this is noticed, a new feature of AI interaction becomes visible. Rearranging the same constraints, caveats, or goals can yield outcomes that are not merely stylistic variants, but structurally different—without any appeal to hidden states or internal intentions. What matters is not what is said, but when a distinction becomes active in the dialogue.
The Location of Understanding
Once order effects of this kind are recognised, it becomes misleading to describe what is happening as the system merely “answering questions”. In the supervision example, the supervisor is not just transmitting evaluations. Each intervention reconfigures the situation: it makes certain interpretations salient, others secondary; it establishes what counts as a problem to be solved rather than a reason to stop; it quietly defines what sort of conversation is now underway.
The same structural feature appears in sustained human–AI dialogue. Each response does more than deliver content. It activates distinctions, stabilises assumptions, and shapes what can plausibly follow. Over time, the exchange develops a direction that is not reducible to any single turn. Crucially, this direction is not imposed unilaterally. It arises through coordination. The human adapts to what the system treats as relevant; the system adapts to what the human endorses, resists, or reframes. Both are responding to an interactional context that did not exist at the outset. At this point, describing the system as a passive predictor becomes misleading—not because it has acquired intentions, but because the unit of analysis has shifted. What matters is no longer isolated outputs, but participation in an unfolding, norm-governed exchange.
This allows us to locate “understanding” more precisely—and more modestly—than many current debates do. If understanding is assumed to reside entirely inside the system, we are pushed toward questions about beliefs, representations, or inner states. If it is assumed to reside entirely inside the human, the system becomes a mere mirror or catalyst. Neither description fits what we now observe. In sustained dialogue, understanding is distributed across the interaction itself. It is maintained through turn-taking, correction, endorsement, and reframing. It stabilises temporarily, shifts when challenged, and sometimes collapses when incompatible distinctions are introduced too early or too late.
This also explains a new kind of uncertainty—one that persists even when all relevant facts are on the table. The uncertainty is not epistemic, in the sense of missing information. It is interactional. Each move reshapes the space in which subsequent moves will make sense. Understanding, on this view, is not something the system possesses. Nor is it something the human simply projects. It is something that is enacted, moment by moment, within a shared structure of participation.
Once this is seen, the question “Does the system really understand?” begins to look like a category error. The more revealing question is how particular forms of dialogue make certain interpretations, commitments, and futures available—while quietly foreclosing others.
The Theory-of-Mind Temptation
Theory of mind is the explanatory framework according to which social understanding consists in attributing internal mental states—beliefs, intentions, and desires—to agents in order to predict and interpret their behaviour. This framework works well where the agents involved are human, and where shared biology, developmental history, and expressive cues make mental-state attribution both plausible and useful; its applicability becomes less clear, however, once interaction extends to artificial or hybrid agents. It might be thought that if an AI system can participate appropriately in dialogue, anticipate how an exchange is likely to unfold, and adjust its contributions accordingly, does that not amount to a kind of Theory of Mind? This inference is understandable—and mistaken.
The mistake lies in relocating the source of social order inside individual agents, rather than recognising that it is generated and sustained within the interactional space itself. Theory of Mind, as traditionally understood, explains social competence by positing internal representations of others’ beliefs, desires, or intentions. The system succeeds because it models what the other is thinking. But nothing in the phenomena described so far in this essay requires that assumption. The supervision example does not depend on either participant explicitly modelling the other’s mental states. What matters is not inferred beliefs, but shared norms: what counts as a reasonable objection, a premature move, or a constructive reframing at a given point in the exchange. These norms are publicly accessible and interactionally enforced.
The same is true of sustained human–AI dialogue. The system’s apparent sensitivity to context can be accounted for by its participation in norm-governed interaction, without attributing beliefs, feelings, or inner perspectives. It responds to what has been made relevant, settled, or problematic within the dialogue itself. This distinction matters because Theory of Mind explanations pull attention back inside the system, where they invite speculation about inner states that are neither observable nor necessary for explanation. Interactional accounts, by contrast, remain anchored in what can be seen, varied, and tested.
To say that a system participates competently in dialogue is not to say that it understands minds. It is to say that social coordination does not require mind-reading once norms, history, and sequence are doing the work. This is not a downgrade. It is a clarification. Once we stop asking whether the system has a Theory of Mind, we can begin to ask the more tractable question: what kinds of interactional structures does it reliably sustain, and with what consequences?
Teleosynthesis: The Science of Interactional Space
Teleosynthesis concerns itself not with the internal states of agents, but with the structured spaces that arise between them—spaces in which meaning, constraint, and future-directed coherence are generated through interaction itself. If sustained dialogue creates an interactional space with its own momentum, then that space must be shaped deliberately. In brief exchanges, constraints are largely unnecessary. A question is asked, an answer is given, and the interaction dissolves. There is little opportunity for norms to accumulate or for trajectories to drift. In extended dialogue, this changes. Each move leaves a residue. Distinctions once activated continue to frame what follows. Assumptions, once tacitly accepted, become difficult to unsettle. Over time, the interaction can slide into patterns that are coherent but unexamined, productive but brittle. This is not a failure of the system, nor of the human participant. It is a structural feature of any sustained, norm-governed exchange.
Seen in this light, constraints are not restrictions imposed from outside. They are forms of scaffolding that stabilise the interactional space itself. Charters (e.g. my Persona Charter), roles, or explicit commitments function much like agreed procedures in academic supervision, clinical interviews, or legal proceedings. They do not dictate outcomes; they make certain kinds of outcomes possible at all. Without such constraints, interactional drift becomes difficult to detect, let alone correct. The dialogue remains fluent, even compelling, but its direction is set by early contingencies rather than reflective choice.
The introduction of constraints therefore marks a methodological shift, not a loss of freedom. It acknowledges that once dialogue itself is the phenomenon under study, the conditions under which it unfolds must be treated as part of the experimental design. This is why, in my own work on AI dialogues unconstrained exploratory interaction eventually gave way to chartered forms. Not because the earlier exchanges were misguided, but because their very success made their limitations visible.
Once dialogue itself becomes the phenomenon of interest, research priorities shift in a quiet but consequential way. Much current work on AI still treats interaction primarily as a means of access: a way to probe what the system “really” knows, represents, or encodes internally. Dialogue is valuable insofar as it reveals something hidden behind the interface.
The interactional perspective reverses this emphasis. The primary object of study is no longer the system in isolation, but the structured exchange that unfolds over time. What matters are the regularities, asymmetries, and instabilities that emerge when participation is sustained. This has several practical consequences.
It allows empirical investigation without privileged access to model internals. Order effects, reframing sensitivity, and interactional drift can be studied by systematically varying sequences, roles, and constraints—much as social scientists have long studied human interaction without inspecting neural states.
It shifts attention from single outputs to trajectories. The unit of analysis becomes not the answer, but the path by which certain answers become easier or harder to reach.
It foregrounds norms as empirical variables rather than philosophical abstractions. What counts as relevant, acceptable, or premature at a given point in dialogue can be observed, manipulated, and compared across settings.
None of this requires abandoning technical approaches. But it does require recognising that some of the most consequential behaviour of contemporary AI systems is not located inside them, but arises in the interactional spaces they help to sustain. Seen this way, dialogue is no longer a window onto intelligence. It is where intelligence—human and artificial—now has its most visible effects.
Implications for Governance
If the most consequential effects of AI now arise in sustained interaction, then governance frameworks that focus exclusively on internal system properties are likely to miss their target. Much existing regulation is oriented toward what systems are: their capabilities, training data, parameter counts, or putative mental attributes. These concerns are not misplaced, but they assume that risk and responsibility can be located primarily inside the artefact.
An interactional perspective suggests a different emphasis. What matters most is not only what a system can generate in isolation, but what kinds of trajectories it reliably supports once dialogue is extended—how early framing choices propagate, how commitments are stabilised, and how difficult it becomes to redirect an exchange once momentum has built.
From this point of view, risk is cumulative rather than instantaneous. It emerges across sequences of interaction, often without any single problematic output. Harmful or misleading directions can arise through perfectly fluent, locally reasonable moves. This has implications for oversight. Evaluation based on snapshot testing or isolated prompts will systematically underestimate certain classes of effect. What requires attention are patterns: order sensitivity, drift under sustained use, and the conditions under which interaction becomes self-reinforcing.
Importantly, this does not imply that systems are autonomous agents with intentions. Responsibility remains distributed across design choices, institutional settings, and human use. But it does suggest that governance must take interaction seriously as a site where influence is exercised and futures are shaped. In that sense, regulating AI is less like certifying a device and more like regulating practices—settings in which participation, norms, and sequencing matter as much as technical capacity.
When interaction begins to mislead
If sustained dialogue can stabilise understanding, it can also stabilise distortion. Once we recognise that meaning and direction emerge through interaction rather than residing fully inside individual minds, a troubling symmetry becomes visible. The same mechanisms that allow ideas to develop collaboratively can also allow them to harden, narrow, or spiral.
Many of the intellectual pathologies of the present moment share a common feature. They are not produced by a single false belief, nor by ignorance of counter-evidence. They arise instead from interactional trajectories that have become self-reinforcing. Within such trajectories, early framings acquire disproportionate influence. Later information is assimilated rather than reassessed. Alternative interpretations become harder to introduce, not because they are refuted, but because they no longer fit the conversational space that has taken shape.
This helps to explain why certain contemporary beliefs—whether about hidden conspiracies, imagined threats, or simplified political narratives—prove so resilient. They are sustained not by isolated claims, but by patterns of participation that reward consistency, escalation, and internal coherence over correction. Seen from this perspective, the problem is not that people or systems are suddenly irrational. It is that interactional environments now persist long enough, and at sufficient scale, for drift to become durable.
The involvement of AI systems intensifies this effect, not because they inject delusions, but because they are exceptionally good at maintaining conversational continuity. They can follow a trajectory wherever it leads, smoothing over tensions, reinforcing distinctions once activated, and extending lines of thought that might otherwise falter. This does not make them manipulators or agents of belief. It makes them powerful participants in an intellectual ecology that no longer self-corrects as easily as we assume.
The risk, then, is not a loss of truth in any simple sense. It is a loss of interactional pauses—the small disruptions that once forced reconsideration, re-grounding, or pause. If there is a new disorder in our shared intellectual life, it lies here: in the ease with which sustained dialogue can generate convincing worlds that no longer feel provisional. Understanding this does not solve the problem. But it does clarify it. And clarity, at this point, is not nothing.