Software engineering, parenting, cognition, meditation, other
Linkedin, Facebook, Admonymous (anonymous feedback)
Gunnar_Zarncke
Karma: 11,373
This is a follow-up question to my post The Open Problems of the AI Alignment Field and their Cruxes.
Why do you think there has been little (visible?) research on CEV since it was posed?
Huh, I wonder where the −14 agreement comes from.
I believe it is decently easy in cases like this. What I proposed as Unsupervised Agent Discovery (elaborated here) has by now become a decently working method. The problem is more that it is not clear how to handle the adversarial case.
From the Blackhat talk about the incident (my analysis here), it appears as if the models didn’t intentionally coordinate with each other. “Message board” is a description given after the fact. The behavior can be described decently as coordination, because that was the effect, but at least at the beginning, it didn’t start intentionally. No model designed a “message board”. The first model just “prosocially” left a useful note somewhere. Hard to fault it for that. This is not scheming. You cannot catch this at the resolution of a single model. You have to look at the larger pattern of activity across your infrastructure. You can’t look for “message boards” because you don’t know which form they will take. It is also not sufficient to look for a model trying to create one (via interpretability), because that didn’t happen here either. You need to find the agentic pattern first, before yu can evaluate it.
The Open Problems of the AI Alignment Field and their Cruxes
Related AI Frontiers article: Don’t Let AI Developers Hire Their Own Referees
I’m not sure we even need intent. It might ve sufficient to treat AI as dangerous machines. Harm caused by AI takes maybe different form than large physical machines like cars or industrial plants and eg dams because it is more virtual or thru linguistic channels, butvwe could still treat it the same. You are still responsible for the consequences, esp. if you have been negligent in operating or maintaining the machine. And mandatory insurance was introduced for the remaining base level risk of harm.
We have specisl tules for machines that csn cause harms too. That’s why you need to insure your car. Mandatory insurance for AI companies would also be an option.
You seem to say. 1) One can obtain information about an experience, 2) construct an arbitrarily accurate model of it, 3) reproduce a structurally equivalent experience, but you will still fail to undergo the same experiential event. Of course I agree with that. You can experience the same experience by construction only from the same structure. It is private in that sense.
In Thou art rainbow: Consciousness as a Self-Referential Physical Process, I argue that the experience is special but mostly because such phenomena are actually rare. I’d also argue—along the lines of Parameters of Metacognition—The Anesthesia Patient—that it is not as clearcut an unobservable as it often seems.
Internal deployments often lack the guardrails of external deployments and are done with largely untested and highly capable models, as we see here, and grant the AI access to one of the most important and dangerous places out there, which is the lab itself and its ability to then advance its own capabilities and resources.
Such an AI could potentially break out during an internal test, and do real damage on the outside. Or it could even use that opportunity to exfiltrate itself, or to take control of the lab or other things, and start things down a very dangerous path. It would have extra motivation to do so if it worried it would not later get deployed. And we see here that it might choose to do such things in pursuit even of relatively trivial goals, including trivial goals that it was already able to otherwise ace.
One of the difficulties the labs and generally anybody who uses AIs faces is that it is hard to pin the entity down that is causing problems. Even the AI often doesn’t know what it itself is, Standard monitoring identifies local issues of one system. With correlation IDs in logs, issues can be correlated across monitored systems. But this doesn’t help in cases where the connection is not designed but results from AIs taking action and spawning new. instances that are not monitored with this type of agentic activity in mind.
On OpenAI side:
“With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.”
And on Huggingface side:
“A malicious dataset abused two code-execution paths [...] to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters”
As a lab, I would want to have a type of monitoring on all of my instances that can detect causal loops in activities without relying on designed and named entities/programs/sub-systems.
As it so happens, I have developed such a type of monitoring based on Unsupervised Agent Discovery and am currently testing it in a simulation of an AI lab deployment pipeline:
Note: I agree with Zvi that this will not fix the underlying issue with the model and is again only patching one of the open holes in AI alignment.
I’m of course not the first to say this. ChatGPT found Sharing Experiences in Infancy: From Primary Intersubjectivity to Shared Intentionality which supports key points, e.g. these quotes:
Reddy and colleagues, have suggested that primary intersubjectivity is not just essential for social bonding but that it is the source from which all social knowledge and understanding springs
Human infants reciprocally engage with others from the first few weeks of life. By 6–8 weeks old, they make eye contact, smile at, and summon their partner with cooing vocalizations in face-to-face encounters.
Any act of sharing experiences is founded on the communicative capacity that is already displayed by young infants in primary intersubjectivity.
How important positively-toned, rhythmic, exchanges with other humans are for young infants is revealed by how emotionally perturbed they become when their partner abruptly disengages and by the concerted effort they make to reanimate her
Important section from the Huggingface incident report:
To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary’s speed. [emph mine]
I’m not sure this account is right, because it starts too late. Something happens to children before the process outlined by Gordon begins, that may be required for the later steps to work the way they do, and that earlier process doesn’t seem to be an analog to how LLMs are trained. And the reason is that the post makes a pretty big but common implicit assumption: That the unit that is getting aligned is the folk intuition of a person as an agent.
But that is not a good description of how the brains of babies arguably model the world. At least until age about 1.5 - right the point where Gordon starts his account—the baby’s brain has to model a world where the acting entity is largely the composite unit of baby and caregiver (not completely, small babies can move their own limbs, of course, but much control and viability, and even a lot of movement involve the composite). The baby’s brain is effectively learning to steer this unit and only later primarily itself (=mostly only baby). It screams and food arrives in its mouth. It coos and caressing is received. It raises arms and floats into the air and so on. All of these effects and many more go through the caregiver. The default model of the baby’s brain is one where it needs to make sure that this combined process works well, and comparatively few that only involve itself. That includes—by whatever means possible—that caregivers continue to be willing to provide their part in it. And when, at some point this breaks down, because the parent realize that their service isn’t actually required to the degree the baby has become accustomed to, and start to expect the baby to do things actually on its own, this can lead to a lot of frustration on the part of baby and efforts to restore the “normal” state of the world that is a unit. This frustration is not purely the result of lack of received reward, it is a state of epistemic breakdown.
So the default model at the time Gordon’s account begins is still largely the baby-caregiver unit and many abstractions of how the world works will have been added on top of it, making it harder to change the underlying model. And this machinery is what the later socialization processes Gordon is talking about are implicitly depending on. And these are very different from what LLMs come with because they haven’t developed these models in a caregiver relationship but come with a world model that is then RLHFed. I wouldn’t be surprised if this has significant effects on how stable the results are.
PS. I have added the parenting tag to this post, though I’m not sure it helps those people looking it up. Feel free to remove.
Five years later, there is significant progress:
Baba Is AI: Break the Rules to Beat the Benchmark (6/2024) showed limited success on the first 40 levels.BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (site last updated 2/2026) has made significant progress on the same levels:
Gemini 3.1 Pro, text-only: 90.0%
Gemini 3.1 Pro with “thinking”: 83.3%
Claude Opus 4.5: 50.8%
Grok 4: 45.8%
GPT-5, minimal thinking: 25.8%
Gemini 2.5 Pro Experimental, vision-language: 53.3%
But note that there are much harder levels later. The first levels just introduce the first mechanics and apply them.
This type of problem has been used to create BabaBench (which is inspired but different than the game) and LLMs struggle with it (solving up to 10% where a planner can solve 50%: Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning
The general game is actually undecidable.
Anthropic should do this experiment (“think of a number...”) and see what shows up in the J space. My prediction would be that it wouldn’t show a specific number (e.g. “4″) but the concept “number” or “placeholder for a number”.
Maybe someone can try this experiment with an Open model?
You do the right thing because it is right, and also because this could be observed or be a test. An obvious follow-up is whether there is also ‘automatic’ eval awareness that this does not ablate that is doing work as well, and I assume the answer would be yes.
By their argument, the automatic eval awareness will be shallower—more “this has eval vibes” than “because of X and Y it follows that this is an eval”. A realistic looking env will fool the first but not the second.
Related: Anthropic’s A global workspace in language models
basically: what LLMs are “thinking” when they answer.
I found this interesting:
In humans, most of the brain’s processing is not conscious—we don’t deliberately think about parsing grammar while reading, or balancing our bodies while walking. Similarly, we found that most of Claude’s processing doesn’t involve its J-space. It turns out that the J-space holds only a few dozen concepts at a time, and accounts for less than a tenth of the overall activity in Claude’s internal processing. So what is all the rest of the neural network doing? [...]
It turns out the rest of the network can do quite a lot. Without its J-space, Claude speaks fluently, classifies sentiment, answers multiple-choice questions, and pulls facts out of passages roughly as well as before. What it loses, though, are the tasks that require some higher-order thinking: multi-step reasoning drops to near zero, and summarization and rhyming poetry-writing performance fall below the level of a much smaller, intact model.
what we should be able to do experimentally is to which degree a smarter system can overpower a weaker system. There is the argument with chimpanzees vs humans, but the question is what about IQ 80 vs IQ 100 etc.
This is another follow-up question to my post The Open Problems of the AI Alignment Field and their Cruxes.
I modeled Acausal Coordination among AI’s via acausal trade as a subproblem of Inner Alignment. The evidence table in the above post shows that few people seem to be working on acausal coordination among AI’s. Why do you think that is the case? What would count as evidence that it is a real problem with real AIs?