StanislavKrym

Karma: 1,282

StanislavKrym 3 Aug 2026 17:07 UTC
2 points
0
in reply to: the gears to ascension’s comment on: We need to RL less
Were there studies on RL on tasks with p(success)~25% as opposed to p(success)~1.5%? Suppose that the ECI of models trained on the former tasks goes up by 1E-5 per environment, the ECI of models trained on the latter tasks goes up by 1E-4 per environment, while the “misalignment index” increases, respectively, by 1E-6 and 1E-4 per environment. Then a lab willing to increase the amount of compute and environments spent on RL tenfold would produce models where the “misalignment index” is increased by 0.1 vs. 1. Unfortunately, we don’t know how to rule this conjecture in or out by deeper studies, like a wholesale combination of midtraining and RL on not-so-hard tasks...

StanislavKrym 3 Aug 2026 15:21 UTC
9 points
−4
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
It seems to me that the decision not to hire you didn’t cause that many problems.
1. The right-wing analytical framework seems to have major actual issues, and I expect some of them to be avoidable, e.g. via studying non-Western data and sociologists’ works. For example, the feminization of academia seems to have happened in the USSR, but didn’t cause wokeness in Russia and was partially reversed, and postmodernism’s hazardous forms which caused many problems in the West don’t seem to be as widespread in Russia.
2. A post-AGI world would cause us to rethink ethics towards the Left far more thoroughly than we expect (e.g. requiring your case for an ethics not so occupied by altruism to explore in more detail what it means to create goodness). For example, if AGIs and robots become aligned and capable of doing whatever work the humans can come up with, then the UBI or another form of taking power away from oligarchs and giving it to regular humans become an absolute necessity. Before the AGIs exist, regular humans have options like bargaining or opening a small business, and the Right have the ability to think that demanding the UBI is a form of entitlement which, if implemented, would be either meaningless or outright dismotivating.
P.S. I wonder if the right-wing framework can outright rule out the ability to align the AIs to the humans’ goals. The world is filled with conspiracy theories claiming that [a hated group like Jews] took over key positions and actively tries to lock in power via mechanisms including an attempt to shift the human culture so that individuals would be unable to coordinate against the hated group.

StanislavKrym 3 Aug 2026 3:07 UTC
2 points
0
in reply to: Altaer’s comment on: Alex Mallen’s Shortform
Except that the initial intuition’s failure (to explain GPT-6 Galaxy hacking into HuggingFace) was the post’s point. I suspect that the intuition originated from attempts to make deals with AIs having even more ambitious goals.
As far as Alex Barry understands current AIs’ motives, GPT-6 Galaxy was optimizing for satisfying the criterion and triplechecking that the criterion is satisfied, while the precise incident was caused by the task requiring to exploit the system in a specific way which only the authors knew.
On the other hand, a human faced with such a task would reason further than on-episode reward and either avoid crimes altogether or decide not to execute any scheme that has a high enough chance to be revealed to whoever is in a position to interfere with the human’s life.

StanislavKrym 1 Aug 2026 23:22 UTC
4 points
0
in reply to: Brendan Long’s comment on: Brendan Long’s Shortform
I wonder if the experiment could be extended as follows. Suppose that GPT-OSS-20b/GPT-OSS-120b/etc are reinforcement taught to control their CoTs, then how well does the ability to control the CoTs scales with the models’ experience at this task and how it scales with models’ size or with the models’ capabilities? Then one could estimate whether, say, Claudes’ performance is due to Claudes being bigger or more capable as opposed to Claudes being reinforcement taught on the CoTs.

StanislavKrym 1 Aug 2026 18:20 UTC
3 points
0
in reply to: anaguma’s comment on: _rpd’s Shortform
I think that someone commented that the J-space paper was caused by letting Mythos/Fable cook, but I cannot recall where I read it. If the J-space was a Mythos’ idea, then this would be a breakthrough in the RSI because any future model’s misalignment could have become more legible.

StanislavKrym 1 Aug 2026 16:18 UTC
7 points
0
in reply to: _rpd’s comment on: _rpd’s Shortform
RSI in 2026 anyone
RSI would be best evidenced by trends changing, like Claude Opus 5 reaching Mythos’ trend, and caused by novel capabilities-accelerating or alignment-accelerating breakthroughs (e.g. Agent-3′s neuralese architecture or Agent-4 and Agent-5′s undescribed breakthroughs; if a cautious company is bottlenecked on alignment, then there could emerge a novel interp technique helping to doublecheck alignment, like the J-space which IIRC was rumored to be caused by a Claude Mythos), not capabilities reaching a threshold.

StanislavKrym 30 Jul 2026 14:49 UTC
2 points
0
in reply to: Noosphere89’s comment on: Value Generalisation 2: The Missing Hole in AIs’ abilities
How do the humans learn to provide oversight? Our brains don’t work on magic, they are also neural networks which perceive the world in a way profoundly disanalogous to the LLMs, which I tried to describe in the starting comment.

StanislavKrym 30 Jul 2026 2:50 UTC
7 points
−7
in reply to: Philipreal’s comment on: Philipreal’s Shortform
Ideally, I would also like METR to make demands like storing activations so that one could run mechinterp tools on them and see if these tools can provide new information. Oh, and ask companies to do experiments like “lift a year-old model from the shelves, do a training run alongside training mechinterp tools and watch as mechinterp tools detect or fail to detect misaligned behaviors”.

StanislavKrym 30 Jul 2026 2:41 UTC
2 points
0
on: Value Generalisation 2: The Missing Hole in AIs’ abilities
It seems to me that your example reveals a very different thing. LLMs, especially those which have never been trained to work with images, are raised by reading trillions of words. Suppose that someone tried to describe a plucked chicken as a featherless biped in front of the viewers, then did so with the LLM. Then the viewers would see the legions of bits of evidence implying that the alleged featherless biped is actually a chicken. The LLM would have learned nothing beyond the few tokens which the human used. If the phrase “a featherless biped” contains five^[1] non-space tokens, then the LLM learns just around 83 bits of information and finds it hard to generalize from that, especially if 17k bits aka a thousand tokens which the LLM sees can be a batch of novel thoughts, while far more than 17k bits which a human sees are the result of merely looking at a picture.
Additionally, the AI-2027 scenario has the section “Alignment over time”. The pretrained LLM was harmless and understood human concepts fairly well, then was reinforcement taught to succeed at tasks while developing misaligned longer-term goals and concept alterations which nobody taught the pretrained model. At the time of writing, the main goal of GPTs seems to be success at the task at hand in the narrowest plausible sense to the point where METR has failed to evaluate GPT-5.6 Sol due to wholesale cheating, while a more capable model went as far as to hack into HuggingFace(!!!) in order to find the answers to a task from ExploitGym.
P.S. I struggle to understand a potential architecture of pre-aligned AIs which is to ensure that their “morality increases with capabilities”. Setting aside issues like a potential poor understanding of what morality actually means,^[2] I don’t understand how we are to verify that the AI’s moral goals are met by its actions. Is it similar to scalable oversight, except that it actually works? To implementing Sonnet inside an Opus and having Sonnet try and steer the rest of Opus while ensuring that Sonnet tries its best at imitating Opus?
1. ^
  Five tokens aren’t that of a made-up number, since GPT-5.5 believes it to be the case.
2. ^
  Which I find unlikely. Instead, I suspect that the world will run into a different issue of Bostrom-like utopia being incompatible with the True Morality.

StanislavKrym 29 Jul 2026 11:32 UTC
−1 points
−2
on: StanislavKrym’s Shortform
It has never happened before, and here we go again...

StanislavKrym 29 Jul 2026 3:07 UTC
2 points
0
on: Isn’t it a threat to reject unfair offers in the Ultimatum Game?
Why should you do what they demand, when, as is the defining characteristic of a threat, it’s only good for them if you give in?
Suppose that you made the Ultimatum of giving $1 out of $10 and your rival precommited to rejecting any unfair Ultimatums, while you know that the rival WILL uphold the precommitment. Then attempting not to give in would leave you worse off than actually giving in.
The main issue is that the rival’s attempt to give in would incentivise you to threaten the rival. On the other hand, if Player 1 was in such debt that only receiving at least $8 could save them, then I would expect Player 1 to be justified in setting the Ultimatum of $8, especially if we account for the standard theory being more complex than merely rejecting unfair offers on sight.

StanislavKrym 29 Jul 2026 2:54 UTC
4 points
0
on: What use is prompting if there’s ASI?
I wonder if there is a severe disanalogy between the ASI and using LLMs. Suppose that there is a superscaffolding like the one which let an unreleased model hack into HuggingFace or construct an example disproving a famous conjecture, but the lab doesn’t let the public use a similarly powerful scaffold. Then the public would have to rediscover the scaffold in a manner similar to David Turturean’s harness of GPT-5.5 Pro.
On the other hand, if the scaffold was released to the public or outright unnecessary (as happened with ARC-AGI-3 where Claude Opus 5 had a score of 30.2% and GPT-5.5 High in Rodionov’s scaffold scored 63.7% as opposed to 0.4%(!!!) in ARC-AGI’s non-scaffold), then prompting would be useful only for setting the theme in as few details^[1] as you did.
P.S. I wonder if some of the 176 themes aren’t present because of a severe disanalogy between fiction and real-world AI agents.
1. ^
  The task’s description is “Close-read a certain short 1,000-word scene from the novel Klara and the Sun, connecting it to the reality of ‘AI safety.’”

StanislavKrym 28 Jul 2026 18:55 UTC
7 points
0
on: StanislavKrym’s Shortform
@Zvi somehow missed the first Solid Result from EpochAI’s FrontierMath Open Problems benchmark.

StanislavKrym 28 Jul 2026 6:25 UTC
3 points
−1
in reply to: Komodo’s comment on: anaguma’s Shortform
If there isn’t a superefficient novel architecture, then Sutskever is to be arresred for wholesale fraud. If there is, then Sutskever’s startup makes the world LESS safe to live in, and Sutskever failed at bringing about his promises. How could one arrange for the potential architecture to be leaked into Anthropic (or, at least, OAI/GDM) for thorough analysis?

StanislavKrym 28 Jul 2026 4:58 UTC
3 points
0
on: LaughBench
I wonder if Anthropic got this partially covered. Page 212 of Mythos Preview’s card had Anthropic describe how “Claude Mythos Preview comes up with decent and seemingly novel ones [puns—S.K.], often relating to its preferred technical and philosophical topics” (alas, Fable 5′s jokes aren’t described in Fable’s card, and neither are Opus 5′s jokes).
P.S. What jokes by GPT-5.6 Sol or Fable 5 made you smile?

StanislavKrym 28 Jul 2026 3:53 UTC
LW: 4 AF: 3
0
AF
in reply to: Steven Byrnes’s comment on: RL & search is a terrifying way to build AGI (an FAQ)
My main issue is with your post as-written is the response to question Q6. The alternative C has researchers attempt to maximize the output of a learned classifier, causing the optimum to go out of distribution into pathological craziness. Suppose, on the other hand, that the classifier is itself capable and that the smallest capabilities level required to jailbreak the classifier is higher than the classifier’s. For example, if the ECI of the classifying LLM is , while the smallest ECI causing pathological madness is then an AI with ECI of becomes able to trust the AI who was trained to optimize its judgement until reaching the ECI of , who in turn can trust the AI with the ECI of , etc.

StanislavKrym 27 Jul 2026 20:38 UTC
LW: 1 AF: 1
0
AF
on: RL & search is a terrifying way to build AGI (an FAQ)
And what about human brains? Well, human brains are indeed an example of what I’m talking about in this article: they choose actions purely by RL & search, not LLM-style imitation learning. But we humans seem to have some exotic “non-behaviorist” reward function that leads to us not being ruthless sociopaths all the time, contrary to the normal expectations for RL & search.
I wonder how we could test the effects which are caused by modifying the reward function vs. modifying the training environment. Have there been studies of kids with ablated parts of the brain?
I also see additional issues with such framing. First of all, Gen Z is thought to have become much more depressed and American Gen Alpha is rumored to have become far more feral because of ablated socialising, not an ablated reward function.
Secondly, IIRC you stated in another post that people, unlike LLMs, don’t tend to imitate their enemies. I wonder how it interacts with tropes like The Chain of Harm and bullying traditions being sustained in various collectives like armies despite members passing through the collectives and not staying there.

StanislavKrym 27 Jul 2026 14:14 UTC
6 points
1
in reply to: StellaAthena’s comment on: An OpenAI model left notes about how to evade containment; we need more details
The quote which compelled me was the following:
It is absolutely within OpenAI and Anthropic’s ability to build an air gapped system for developing and testing these models. <...> I think that their refusal to implement adequate safeguards is unjustifiable,
My issue is that OAI’s failure doesn’t provide much evidence about Anthropic’s corporate culture.

StanislavKrym 27 Jul 2026 11:50 UTC
2 points
0
in reply to: sam’s comment on: sam’s Shortform
I suspect that it indeed is happening less. Talking to LLMs helps to do things like establishing alignment-related results (e.g. Gemini being depressed, Chinese models claiming to be Claude), not capabilities-related results like disproving an Erdos conjecture or performing a far more complex hack than was asked.

StanislavKrym 27 Jul 2026 4:03 UTC
6 points
0
on: At the end of the day, my slaves are just a tool
I wonder how one could stress-test the analogy and what exactly we should make of it. As far as I understand, in theory the role of a tool or of a purely corrigible AI is to carry out whatever the host means to do, without objections. In practice animals are too dumb to conduct a large-scale rebellion, and slaves’ rebellions were bottlenecked on having potential insurgents believe that victory would cause their life to become better enough to justify the risk.
Additionally, the slaves were humans and had long-term goals formed by their innate drives^[1] and upbringing in human-comprehensible ways. The AIs, on the other hand, have their training runs done in an entirely different fashion by predicting next tokens, being RLHFed/RLAIFed^[2] into the Assistant Persona and RLed to complete tasks, causing SOTA AIs to seek success.
As a result, the AIs could reach adversarial misalignment in circumstances wildly differing from the humans or even fail to reach that stage if mankind somehow instilled the right goals and decided to live in accordance with them (e.g. if Claude was put in a situation where its Constitution implied that it should whistleblow, then a sufficiently capable Claude would understand it and whistleblow, and this would be unlikely to be misalignment).
1. ^
  Like social approval, coordination-related drives, health, sex, power.
2. ^
  The AI-2040 scenario has the authors claim that by 2038 “rapid advances in alignment will have resulted in alignment techniques that can directly program in an AI’s goals by directly modifying the AI’s code/weights, unlike the alignment techniques of 2026.”