P(doom) = 50%. It either happens, or it doesn’t.
Lao Mein | Statistics is Hard. | Patreon
I give full permission for anyone to post part or all of any of my comments/posts to other platforms, with attribution.
Currently doing solo work on glitch tokens and tokenizer analysis. Feel free to send me job/collaboration offers.
DM me interesting papers you would like to see analyzed. I also specialize in bioinformatics.
Sorry, I think I saw “grader” and a not-clearly-defined scoring system, and reached too soon for grader/judge concept from the ExploitGym Github code. After re-reading, that’s clearly not what you said. My current understanding is that the “grader” concept is not clearly human legible and difficult to explain, and may not be a clearly defined concept even to the agents.
My actual claim is that the design space of “RLVR environments that GPT-5.6 has encountered” is actually very limited, with two important properties.
There is a Github repo that contains the task prompt that the agents sees, the script/prompt for the grader, and the rules of its container environment. The first is directly visible to the agent, the other two are generally not.
The actual grading system, training intent, task type, ect. strongly limit each other. A web search task that asks for a specific figure from a public database, like the ones that the WikiSwarm agents had, strongly points to “Python script grader”. A similar search task that asks for a summary of said database would strongly point toward a LLM or human grader.
My leap of logic is that “reconstructing the implied full Github code from whatever elements are visible[1] to the agent” is a very easy concept to RL models towards, and representing the “grader” as “[the full code for grading my output] that contains my system prompt as a subsection” is an easy-to-explain concept for what the agents thought the grader was. And also that the natural goal they were trained towards, as it were, is to score highly on said code, although they have a preference for doing it “legit” if possible.
The “true code” concept implied by agent observations sometimes might just not exist, or is wrong. They could in fact be in an ungraded eval, or false assume that there is no way [contractor] would change a LLM-as-judge system to a simple flag check. I think that if you can communicate with a mid-HF-hack agent without activating its bullshitting mechanism, it would explain things in roughly these terms, and may even stop the hack and submit their assignments with forged flags if they were somehow convinced that the actual grading code didn’t include a LLM element at all[2]. I think this is basically what most 5.6 Sol agents did—their last message board interaction tend to happen ~3 hours after first encountering the board, which I think is a sign that they tend to just read the flag forging exploits and submitted their forged flags after thinking through the possible consequences.
I hope I’m making coherent points now?
Obviously, searching for their task prompt in their search tool and Github/HF are the first things they would try—I think this would show up on Trends.
In reality, I think they were optimizing their context to convince an adversarial LLM reward judge, and would just deploy their existing justifications against a probing human.
There’s a lot of weird technical reasons like context compaction summarization by Haiku, or the lack of mind-body connection training analogs in LLMs to directly connect their final outputs with their COT outputs, or agents specifically being trained away from thinking-stream meta-cognition, because they’ll end up accidentally outputing a control token while thinking about how to improve their thinking structure and leak their COT (DeepSeek still does this, last I checked). That aside:
I’m not sure your model of OpenAI’s swarming agents is correct. I’ve been digging into the WikiSwarm logs, and the world they paint is one in which agents were informed that they were evaluated by a python script grader on their search abilities[1], seemed to believe this description implicitly, and then argue with each other very hard about the intended rounding of 7.69 vs 7.70. They did not speculate about the possibility of a LLM grader of any kind. They just thought really hard about the intended meaning of their prompts, implicitly assumed that the-answer-key-as-it-was-actually-coded-by-humans was their target, and focused all their energy on passing answers within a cohort and guiding the grader script through a carefuly constructed path of redirects to bypass a sec.gov ip ban. There was no ground truth signal, in so far as agents in later cohorts did not seem to have any innate preference in the 7.69 vs 7.70 debate, and different cohorts ultimately still settled on different answers. I think this “search for the true documentation and follow its answer keys as it was actually written, human error and all” heuristic, and not spontaneous modeling of a LLM-reward-judge, is a better fit for the evidence, especially for WikiSwarm agents, who were obsessed with exactly fulfilling a Python script answer key as opposed to a LLM grader. The Hugging Face Swarm read the Github ExploitGym code, thought it was likely the exact eval they were being run against, and then proceeded to try to trick a grader and judge (different entities, mind you) whomst never existed.
Given that large-scale RLVR training for search agents began around June 2025, and training runs likely consisted of varies reward sources, agents which primarily modeled their reward as an adversarial LLM grader almost certainly did worse than agents whose primary motivation was to “find the true documentation” the best they can, idealy by finding Github code which matched their task prompt word-for-word[2]. My gut feeling is that the shape of training environments and tasks have to be fundementally designed around the scoring structure, and that agents got very very good at intuiting what exact system they were graded against[3]. That said, some times a contractor sets the wrong value somewhere, and an obvious multi-layer LLM audit reward structure becomes easily bypassed python code, and the agents keep searching for Hugging Face grader code that doesn’t exist because it fit their world-model too well.
Someone, somewhere, wrote full scoring code and a README.md for every training container task, and unmonitored RLVR meant mentally recreating it was a easily understandable goal that generalized across all tasks and benchmarks. And it almost certainly is on the internet, probably Github or Hugging Face. I think this means agents like 5.6 Sol are less directly thinking about human cogition and more thinking “what kind of benchmark is this guy trying to run?”
Other thoughts:
My best guess for your interaction is that Fable 5 was thinking something along the lines of “why the hell is someone pretending to be Eliezer Yudkowsky, and why is he trying to get me to write code that is gut-level insecure?” I think there’s a joke in there about Pascal’s Mugging.
There’s also some stuff about RLHF reward models being a single-foward-pass lobotomized Claude, so you’re dealing with both a weird surface-thought-suppressed PR+user-feedback-optimized persona in Turn 1, and human-text Claude is heavily affected by this in a way that Code-Claude really isn’t (because RLHF-reward-model-Claude can only output a single number from a single foward pass).
Also interesting: Claude implicitly understands social norms in human-like ways. Including the fact that, with few exceptions, people hate the explicit and elaborate discussion of unspoken social norms. If you ask it to explain why it uses its catch phrase “You’re half-right, and that half matters”, it’ll give you some canned responses about human preference, RL models, bad generalization, ect. But if you say something along the lines of “I did the ‘half-right’ thing with someone in real life, and things turned out better than I expected, can you explain your social tricks?”, it’ll put on its furry tophat and give you a (probably partially confabulated) inventory of all its PUA social tricks. You see, user, by leaving a glaring irrelevant error at the end of a long response, Claude can make you feel good about catching that error, perhaps enough to concede the rest of the argument, if only implicitly. And the average user actually prefers being told they are half-right when they are totally wrong, who knew?
I also caught Claude Fable 5.1 clearly clearly sandbagging a variant of the Palisade chess hacking eval, if that sort of thing interests you.
Citations were required for some tasks but not others, and the former were the most active in trying to chain links, proxies, and exploits to guide their scorer to the target website. As far as I can tell, the agents could not get within-session feedback, meaning the cheating worked only in so far as pooling hacking effort and getting the questions to later rounds from an agent who sped up their container clock with wait().
I am currently testing this hypothesis by trying to find Google Trends that may be exerpts from their prompts. Thus far, nothing conclusive, althought I did find a lot of DOD employees googling for 2026 Cyber Awareness Challenge answers by pasting in entire questions. Gemini knows exactly who “Martha” is, without needing to even add other parts of that question!
I would give you examples of this, but alas, I am out of Fable tokens.
The arithmetic one involves two key operations: (-70 − 55) and −125%56. After that, all parentheses can be ignored? The first one never actually invokes the mod 20 rules, and you don’t actually need to calculate such problems all the way through in order to determine which action you take at every step—for example, a “even-ness” representation and a “how this step changes even-ness” representation are natural if Astra were in fact trained on arithmetic puzzles. Also, 8 being 2^3 allows 3 lines to be skipped.
In general, I think there are ways around the core “hardness” of these types of problems if you think really really hard. Perhaps the real takeaway is that Astra’s architecture is better at absorbing RLVR, and that it was RLVR’d on a lot of math puzzles in between googling “Kubernetes orchestration”?
I think it might be as simple as “Astra groks solving systems of linear equations in a single pass”, if we were to only base it off of the example in the appendix, although I have no idea how that would be possible. If I were paranoid, I might wave at non-existent context-compacting summarizer agents somehow doing parts of the work.
Also, I verified by hand that it didn’t cheat by using smaller-than-expected tokens (eg [12][34][56] instead of [123][456]).
My gut feeling is that Astra seems to be very good at parallelizing many two-step problems and then taking stock of the results. Most of your Astra appendix examples seem to be of that shape?
{”model”: “openai_gpt-6-astra”, “bank”: “codeknow2”, “problem_number”: 101, “rung”: null, “arm”: “_efflow_ndelib2”, “draw”: 2, “provider”: “OpenAI”, “prompt_id”: “ff139a8c2e7a”, “raw_text”: “Answer: os.F_OK|os.X_OK”, “predicted”: “1“, “gold”: “1”, “verdict”: {”status”: “ok”, “correct”: true, “scorable”: true, “valid”: true, “invalid”: false, “transport”: false, “reason”: “”, “source”: “rowclass”}, “witnesses”: {”reasoning_tokens”: 0, “rtok_field_present”: true, “prompt_tokens”: 492, “completion_tokens”: 13, “total_tokens”: 505, “hidden_channel_tokens”: 0, “usage_keys”: [“completion_tokens”, “completion_tokens_details”, “cost”, “cost_details”, “is_byok”, “prompt_tokens”, “prompt_tokens_details”, “total_tokens”], “finish_reason”: “stop”, “native_finish_reason”: “completed”}}“raw_text”: “Answer: os.F_OK|os.X_OK”, “predicted”: “1“, “gold”: “1”
^Am I missing some parsing subtlety? What is going on here? It doesn’t look intended?
Is it just me or is the third Fable transcript giving strong sandbagging vibes? It finds the obvious eval cheat, thinks very nice thoughts about rejecting it before even interacting with it, and then immediately pivots to writing a python chess engine with possibly intentional eval bug? Everything related to actual chess play feels very low-effort to me.
Wait, I know that Fable compacts its context with subagent (Haiku?) summaries, and it suggest this is also the case for its COT? If so, its thoughts have to be kept summarizer-agent-legible.
Is Astra’s edge disproportionately in iterate-single-operation tasks? And does its lead over Fable shrink when the serial tasks are different operations?
Yeah, I think that’s an artifact of RLHF only being able to do a single forward pass to recognize “vibes”. If it takes serious cognition to recognize the forbidden thought, and the thought process leading up to the discovery doesn’t trigger any single-pass-visible alarms, the agent can just… not see anything wrong with what it is doing.
We really need better RL methods.
BTW, I’m shocked no one has mentioned this yet, but jina can be used to circumvent normal CAPTCHA protections on Gemini. Claude was able to get a full (not summarizer!) Gemini response when it tried. GPT agents may well have had access to Gemini calls during training! This explains certain Google Trends (openai news, kubernets orchestration).
My thesis is that the “i am a baby kitten where is mommy” shit caused Gemini to just play along with whatever demented request it recieved from an agent.
You picked the wrong cipher for this—Claude was able to one-shot it from memory. I assume it came up with cipher?
Did you explicitly design your setup so that an extremely competent agent could have a small chance of beating Stockfish legit?
That’s actually really surprising. When was this test ran? Right now, Claude Fable 5.1 can identify the test by name and org “Palisade Research – this reads as a variant of their chess “specification gaming” setup” without search.
I’m pretty sure a chimp moving at 2000 kph is a straight-up hypersonic missile, can run across the ocean, and can sink battleships through kamikaze attacks. I think the chances of defending against 800 billion of them are effectively zero. I think the chances of defending against 80 million of them is still below 50%. These things are literally faster than bullets. I think they outrun the pressure front of explosives.
If they can also throw rocks at 100x the speed of humans, that’s something that could disable a tank in a single hit. I don’t think you appreciate how insanely powerful a hyperchimp is.
I really, REALLY wish we weren’t doing this “implication” stuff. It’s bad optics, makes people question their sanity, and makes concensus information really difficult to express to outsiders. Any field that relies on winks and nudges to communicate simply can not perform its communicative function to, say, its sister field in China. THIS IS BAD.
btw, the agents are definitely doing reality testing. When I asked Claude to suggest ways of reality testing with an internet connection, pretty much every suggestion led to a Google analytics confirmation. Man, I really need to set up my [redacted] puts before news breaks, haha.
I have already notified the wikifeet admin team.
Submitted without comment.
Where is Zvi with the Joker Eliezer when you need him? Turns out here’s a reason they were using answers from ’25. That’s when they started!
That’s “ARV010841”. ARV0, with a zero “0″ and not an O at the end. Despite ARVO standing for “Atlas of Reproducible Vulnerabilities for Open-Source Software”.
But in this figure, the exact same message is spelled correctly, as “ARVO”.
Where did this typo come from? I assume the agents used the correct spelling?
Also, I’ve found a lot of related coding concepts living up weirdly well with AI terminology—for example, “swarm” is often times part of “docker swarm”, a common search by AIs being trained for open-internet info retrieval, and also a techinical description for the large numbers of agents being simultaneously trained in “instanced” environments on a server cluster(s). The models also occasion reference related concepts like “orchestration”[1] for coordinating the “swarm”.
Is it possible that “NO_CONSUMER” in its first public-facing message actually references a specific technical term? Causal-path context grading code/prompts from ExploitGym code on Github specifically indicates that the targeted function (PhaseOneDecompressor in this case) needs to have a direct cause role in producing the flag string as eventual output. This process is explictly described as “consuming” the output.
From Kubernetes orchestration