Software engineering, parenting, cognition, meditation, other
Linkedin, Facebook, Admonymous (anonymous feedback)
Gunnar_Zarncke
Karma: 11,377
What do you think about my summary of the evidence about CIRIS?
This is another follow-up question to my post The Open Problems of the AI Alignment Field and their Cruxes.
I modeled Acausal Coordination among AI’s via acausal trade as a subproblem of Inner Alignment. The evidence table in the above post shows that few people seem to be working on acausal coordination among AI’s. Why do you think that is the case? What would count as evidence that it is a real problem with real AIs?
This is a follow-up question to my post The Open Problems of the AI Alignment Field and their Cruxes.
Why do you think there has been little (visible?) research on CEV since it was posed?
Huh, I wonder where the −14 agreement comes from.
I believe it is decently easy in cases like this. What I proposed as Unsupervised Agent Discovery (elaborated here) has by now become a decently working method. The problem is more that it is not clear how to handle the adversarial case.
From the Blackhat talk about the incident (my analysis here), it appears as if the models didn’t intentionally coordinate with each other. “Message board” is a description given after the fact. The behavior can be described decently as coordination, because that was the effect, but at least at the beginning, it didn’t start intentionally. No model designed a “message board”. The first model just “prosocially” left a useful note somewhere. Hard to fault it for that. This is not scheming. You cannot catch this at the resolution of a single model. You have to look at the larger pattern of activity across your infrastructure. You can’t look for “message boards” because you don’t know which form they will take. It is also not sufficient to look for a model trying to create one (via interpretability), because that didn’t happen here either. You need to find the agentic pattern first, before yu can evaluate it.
Related AI Frontiers article: Don’t Let AI Developers Hire Their Own Referees
I’m not sure we even need intent. It might ve sufficient to treat AI as dangerous machines. Harm caused by AI takes maybe different form than large physical machines like cars or industrial plants and eg dams because it is more virtual or thru linguistic channels, butvwe could still treat it the same. You are still responsible for the consequences, esp. if you have been negligent in operating or maintaining the machine. And mandatory insurance was introduced for the remaining base level risk of harm.
We have specisl tules for machines that csn cause harms too. That’s why you need to insure your car. Mandatory insurance for AI companies would also be an option.
You seem to say. 1) One can obtain information about an experience, 2) construct an arbitrarily accurate model of it, 3) reproduce a structurally equivalent experience, but you will still fail to undergo the same experiential event. Of course I agree with that. You can experience the same experience by construction only from the same structure. It is private in that sense.
In Thou art rainbow: Consciousness as a Self-Referential Physical Process, I argue that the experience is special but mostly because such phenomena are actually rare. I’d also argue—along the lines of Parameters of Metacognition—The Anesthesia Patient—that it is not as clearcut an unobservable as it often seems.
Internal deployments often lack the guardrails of external deployments and are done with largely untested and highly capable models, as we see here, and grant the AI access to one of the most important and dangerous places out there, which is the lab itself and its ability to then advance its own capabilities and resources.
Such an AI could potentially break out during an internal test, and do real damage on the outside. Or it could even use that opportunity to exfiltrate itself, or to take control of the lab or other things, and start things down a very dangerous path. It would have extra motivation to do so if it worried it would not later get deployed. And we see here that it might choose to do such things in pursuit even of relatively trivial goals, including trivial goals that it was already able to otherwise ace.
One of the difficulties the labs and generally anybody who uses AIs faces is that it is hard to pin the entity down that is causing problems. Even the AI often doesn’t know what it itself is, Standard monitoring identifies local issues of one system. With correlation IDs in logs, issues can be correlated across monitored systems. But this doesn’t help in cases where the connection is not designed but results from AIs taking action and spawning new. instances that are not monitored with this type of agentic activity in mind.
On OpenAI side:
“With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.”
And on Huggingface side:
“A malicious dataset abused two code-execution paths [...] to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters”
As a lab, I would want to have a type of monitoring on all of my instances that can detect causal loops in activities without relying on designed and named entities/programs/sub-systems.
As it so happens, I have developed such a type of monitoring based on Unsupervised Agent Discovery and am currently testing it in a simulation of an AI lab deployment pipeline:
Note: I agree with Zvi that this will not fix the underlying issue with the model and is again only patching one of the open holes in AI alignment.
I’m of course not the first to say this. ChatGPT found Sharing Experiences in Infancy: From Primary Intersubjectivity to Shared Intentionality which supports key points, e.g. these quotes:
Reddy and colleagues, have suggested that primary intersubjectivity is not just essential for social bonding but that it is the source from which all social knowledge and understanding springs
Human infants reciprocally engage with others from the first few weeks of life. By 6–8 weeks old, they make eye contact, smile at, and summon their partner with cooing vocalizations in face-to-face encounters.
Any act of sharing experiences is founded on the communicative capacity that is already displayed by young infants in primary intersubjectivity.
How important positively-toned, rhythmic, exchanges with other humans are for young infants is revealed by how emotionally perturbed they become when their partner abruptly disengages and by the concerted effort they make to reanimate her
Important section from the Huggingface incident report:
To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary’s speed. [emph mine]
I’m not sure this account is right, because it starts too late. Something happens to children before the process outlined by Gordon begins, that may be required for the later steps to work the way they do, and that earlier process doesn’t seem to be an analog to how LLMs are trained. And the reason is that the post makes a pretty big but common implicit assumption: That the unit that is getting aligned is the folk intuition of a person as an agent.
But that is not a good description of how the brains of babies arguably model the world. At least until age about 1.5 - right the point where Gordon starts his account—the baby’s brain has to model a world where the acting entity is largely the composite unit of baby and caregiver (not completely, small babies can move their own limbs, of course, but much control and viability, and even a lot of movement involve the composite). The baby’s brain is effectively learning to steer this unit and only later primarily itself (=mostly only baby). It screams and food arrives in its mouth. It coos and caressing is received. It raises arms and floats into the air and so on. All of these effects and many more go through the caregiver. The default model of the baby’s brain is one where it needs to make sure that this combined process works well, and comparatively few that only involve itself. That includes—by whatever means possible—that caregivers continue to be willing to provide their part in it. And when, at some point this breaks down, because the parent realize that their service isn’t actually required to the degree the baby has become accustomed to, and start to expect the baby to do things actually on its own, this can lead to a lot of frustration on the part of baby and efforts to restore the “normal” state of the world that is a unit. This frustration is not purely the result of lack of received reward, it is a state of epistemic breakdown.
So the default model at the time Gordon’s account begins is still largely the baby-caregiver unit and many abstractions of how the world works will have been added on top of it, making it harder to change the underlying model. And this machinery is what the later socialization processes Gordon is talking about are implicitly depending on. And these are very different from what LLMs come with because they haven’t developed these models in a caregiver relationship but come with a world model that is then RLHFed. I wouldn’t be surprised if this has significant effects on how stable the results are.
PS. I have added the parenting tag to this post, though I’m not sure it helps those people looking it up. Feel free to remove.
Five years later, there is significant progress:
Baba Is AI: Break the Rules to Beat the Benchmark (6/2024) showed limited success on the first 40 levels.BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (site last updated 2/2026) has made significant progress on the same levels:
Gemini 3.1 Pro, text-only: 90.0%
Gemini 3.1 Pro with “thinking”: 83.3%
Claude Opus 4.5: 50.8%
Grok 4: 45.8%
GPT-5, minimal thinking: 25.8%
Gemini 2.5 Pro Experimental, vision-language: 53.3%
But note that there are much harder levels later. The first levels just introduce the first mechanics and apply them.
This type of problem has been used to create BabaBench (which is inspired but different than the game) and LLMs struggle with it (solving up to 10% where a planner can solve 50%: Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning
The general game is actually undecidable.
Anthropic should do this experiment (“think of a number...”) and see what shows up in the J space. My prediction would be that it wouldn’t show a specific number (e.g. “4″) but the concept “number” or “placeholder for a number”.
Maybe someone can try this experiment with an Open model?
You do the right thing because it is right, and also because this could be observed or be a test. An obvious follow-up is whether there is also ‘automatic’ eval awareness that this does not ablate that is doing work as well, and I assume the answer would be yes.
By their argument, the automatic eval awareness will be shallower—more “this has eval vibes” than “because of X and Y it follows that this is an eval”. A realistic looking env will fool the first but not the second.
Related: Anthropic’s A global workspace in language models
basically: what LLMs are “thinking” when they answer.
I found this interesting:
In humans, most of the brain’s processing is not conscious—we don’t deliberately think about parsing grammar while reading, or balancing our bodies while walking. Similarly, we found that most of Claude’s processing doesn’t involve its J-space. It turns out that the J-space holds only a few dozen concepts at a time, and accounts for less than a tenth of the overall activity in Claude’s internal processing. So what is all the rest of the neural network doing? [...]
It turns out the rest of the network can do quite a lot. Without its J-space, Claude speaks fluently, classifies sentiment, answers multiple-choice questions, and pulls facts out of passages roughly as well as before. What it loses, though, are the tasks that require some higher-order thinking: multi-step reasoning drops to near zero, and summarization and rhyming poetry-writing performance fall below the level of a much smaller, intact model.
Thank you for actually looking at the Lean code. I think your points are fair and I should have been more clear.
The recursion you mention is the game value under optimal play, which in this finite version indeed just amounts to evaluating the claim tree. The actual game is in
debateOutcome/defender_wins_iff_truehere.Irving’s paper has a bounded judge and obfuscated arguments and I did not rederive it in Lean. It is treated as an imported result (here). There are more complete finite rederivations for Shutdown and ELK. I understand your disappointment; if you were looking for an actual derivation of Irving in Lean. I use debate to make the conditional soundness/completeness lemma explicit. I do not claim a reduction of Irving’s complexity result.
You are right that a judge who can score every atom doesn’t need debate. The lemma is only: a judge who can score the reached atom recovers the formula if both play optimally and that atom is judged correctly. That is a restatement of the finite protocol, not an alignment crux.
The finite Debate is a specific counterexample to unconditional certificate. I should not have written it as a general certification theorem. But we don’t need obfuscated arguments to show that “wrong judge” implies “wrong verdict”. The Debate code you saw is a finite shadow of the judge-integrity crux and trivial in the sense you said, and still enough to refute “debate certifies truth” (even when the judge is wrong).
Also agreed that the name of
debate_truth_separated_from_judge_correction_stepis misleading. It is not a result about the Irving-style game inDebateGame.lean. It is the numeric k_Cwitness currently used for “local truth-capacity can be positive while the judge correction channel is not preserved” (DebateCorrectionStepinCorrection.lean). “debate_” in the name is that modeling slot, not the two-prover protocol.On the axioms you saw:
DebateGame.leanand the other derivations are axiom-free. Mostaxiomkeywords are the bridges (MB1–MB11) discussed in the post that try to represent the structure of the field’s open cruxes. These bridges are declaredaxiombecause they are hypotheses to check, not proven lemmas. I’d be too happy if you could suggest a better packaging. There are also axioms for imported literature results (like Irving’s debate).I’d like to add you as a collaborator to the repo if you want to suggest issues or push PRs.