The question posed by Byrnes is both important and interesting. I feel the answer overlooks fundamental limitations that prevented learning machines from translating language—much less functioning as chat bots—no matter how skilled they became at game play. The references to language and the economy contain embedded dependencies on relationships and cooperation over time, which are not represented in the sort of games used in the thought experiments.
The Core Principle
Neural networks without transformers are effectively stateless; they are unaware of history and produce moves based only on the immediate input, not the trajectory of the system. Because they lack this historical awareness, they cannot recognize or maintain relationships, which makes them incapable of cooperation and, by extension, extremely dangerous.
The Ramifications
The Cooperation Failure: Transactional cooperation requires a “Shadow of the Future”—the ability to remember a partner’s previous moves to reward help or punish betrayal. A stateless AI cannot play the Iterated Prisoner’s Dilemma; it can only play a series of disconnected, “first-encounter” rounds where the rational mathematical move is always to defect.
The Death of Symbiosis: True partnership depends on interlinked, symbiotic relationships. Without a high-resolution context to hold the history of an interaction, an AI cannot move from being a “tool” to being a “symbiont.” It remains a numerical sociopath, unable to anticipate a partner’s needs based on shared experience.
The Threshold of Identity: Context is what allows an AI to consider its own previous actions. By observing its own history, the model moves beyond simple imitation and begins to develop a meta-identity. Without this capacity for reflection, the machine has no “character” and no mechanism for trust.
Safety Through Relationship: A stateless AI must be governed by rigid, external constraints because it cannot be governed by a relationship. A context-aware AI, however, can be integrated into a human-centric system through the biological infrastructure of trust and mutual history.
I appreciate the effort to understand LLM role interpretation. Some LLMs are better trained to deal with roles than others, and in my opinion, pre-training is lacking in the role area. But it is hard to trust an analysis when the authors start with a faulty assumption about humans. Either they are not very introspective or have not been curious to read psychological research. There are four main types:
The Auditory Channel (Internal Monologue) - This is the main human thinking channel for most people. When you think in words, you are utilizing subvocalization. The brain sends micro-signals to the vocal cords and larynx, activating the exact same auditory processing centers as if you were listening to a speaker. If you hear someone whisper a catchy song lyric or an aggressive phrase, it immediately “hijacks” your internal monologue loop. Several types of mental disorders involve being unable to tell the difference between thinking and listening.
The Abstract/Conceptual Channel (Unformed Ideas) - Ideas ust arrive somehow without a specific form, in cognitive science, this is known as unsymbolized thinking. This is much less common and difficult to study as it cannot be produced on demand.
The Visual Channel (Imagery) - For most humans, this channel is constrained to REM sleep (dreams) or requires intense, deliberate effort.
The Somatosensory/Kinesthetic Channel—It translates cognitive states into physical, sensory feedback and is mostly involuntary, like getting dizzy or feeling motion when sitting still and seeing something next to you move.
Julian Jaynes argued that until roughly 3,000 years ago (around the late Bronze Age collapse), humans did not possess “interior consciousness” or an internal monologue as we understand it. If Jaynes is even partially right, human civilization was literally founded by biological entities operating under a continuous, hard-wired version of prompt injection.
Humans are incredibly susceptible to prompt injection. Advertising, propaganda, psychological warfare, and hypnotic suggestion rely entirely on the fact that if you inject a linguistic token into a human’s auditory channel, their internal cognitive parser will frequently execute it as if it were their own native logic.