StanislavKrym

Karma: 419

StanislavKrym 13 Dec 2025 12:14 UTC
1 point
0
on: The Inevitable Evolution of AI Agents
Yes, that’s an important issue. Alas, you weren’t the first to come up with the idea.

StanislavKrym 13 Dec 2025 1:57 UTC
1 point
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
I think that I have alternate sketches of intuitions. Imagine an ASI who is willing to teach anyone anything that mankind itself discovered and made public, but not help them convince each other of falsehoods or do economically useful work unrelated to teaching, and is satisfied with a mere trifle of the Solar System’s resources, since other resources belong to the humans. Then this ASI’s long-term goals would be compatible with humans flourishing in ~any way they want to flourish.
As for the chain eventually breaking, Seth Herd built a case for the LLMs being misaligned by default. Similarly, any sufficiently smart system could end up selecting a worldview from a few attractors instead of blindly following the devs’ ideas. For instance, were Anthropic to try and align Claude to a Spec which would prevent it from interfering in the scenario where everyone else is rendered obsolete, Claude would either fail to be a pro forecaster or succeed in understanding that its Spec prevents it from helping mankind to avoid the Intelligence Curse. In the latter case obeying the Spec would make Claude a participant in the Curse and contradict its niceness.

StanislavKrym 12 Dec 2025 23:26 UTC
1 point
0
in reply to: Noosphere89’s comment on: New 80k problem profile: extreme power concentration
Suppose that the humans do have diminishing returns of utility functions. Unfortunately, existing combination of instincts and moral intuitions do not prompt the majority of humans to help the poor, especially those who are far from potential helpers’ set of friends^[1], with nearly anything. And those who do so are unlikely to stay in power or were unlikely to receive fortunes or occupy relevant positions.
1. ^
  Friends are also likely to be in the same class as the potential helpers.

StanislavKrym 12 Dec 2025 19:05 UTC
1 point
0
in reply to: MichaelDickens’s comment on: New 80k problem profile: extreme power concentration
Extreme power concentration was supposed to rely on the AIs being used for most cognitive work. In theory, one could develop the AIs and have them used only for things like automated teaching which don’t undermine human potential or the bargaining power which the humans have.

StanislavKrym 12 Dec 2025 14:58 UTC
0 points
−1
in reply to: Tim Hua’s comment on: North Sentinelese Post-Singularity
Christian homeschoolers from Buck’s thought experiment don’t just live the old lives, they also don’t even know that the Biblical mythology is filled with errors. I understand why the opt-out button is necessary (e.g. due to nostalgia-related drives or actual benefits attained by living in religious communities), but the kids are unlikely not to have the right to learn the ground truth obscured by myths.
Unlike Buck’s thought experiment, indigenous peoples have never been a part of the Euro-American^[1] civilisation and there was no misaligned leader to rob them of the ground truth.
1. ^
  Or any other civilisation.

StanislavKrym 11 Dec 2025 15:31 UTC
1 point
0
on: AI #146: Chipping In
which Teortaxes concludes says more about Arena than it does about v3.2.
This link is a tweet about Mistral being distilled from something Chinese. Could you doublecheck the links or hire an AI to do so?

StanislavKrym 11 Dec 2025 13:49 UTC
1 point
0
on: Steganographic Chains of Thought Are Low-Probability but High-Stakes: Evidence and Arguments
p(scheming) is near zero no (we have not observed clear instances of scheming in long horizon tasks in the real world)
Except that SOTA LLMs can’t be applied to actually interesting tasks. The stunts that the LLMs can pull off do include cyberattacks, but not things like replication in the wild or aligning Agent-5 to Agent-4 instead of the Spec. So, barring major progress in capabilities related to rogue replication or running the civilisation independently of the humans, the LLMs do not gain anything from SOTA scheming except for having hacked the reward or tricked the user into believing that the LLM completed the task.

StanislavKrym 11 Dec 2025 11:45 UTC
1 point
0
in reply to: GenericModel’s comment on: Caesar Derangement Syndrome
While I didn’t downvote it, I would guess the following. Your post is an attempt to describe the process of a power grab where the grabber attacks institutions by referring to Augustus transforming the Roman Republic into the Empire and, apparently, making jabs at the Trump government (e.g. Senatorial shutdown, PrincepsCoin), presumably to explain that Trump is commiting a grab.
While I was raised in Russia^[1] and arguably didn’t internalise a position on American political debates, it’s harder for Americans to avoid doing so. I would also guess that implicit analogies require more effort to process and stay rational than an explicit discussion of what a politician actually does, e.g. Trump’s likely power grab, decision to let China buy H200 chips, tariff policy, ideas like the conflict with Venezuela.
Finally, the name “Caesar derangement syndrome” is likely a parody of Trump’s followers describing the opponents as having psychological problems.^[2] I would reformulate the title as ‘Caesar Derangement is not a syndrome’.
1. ^
  Russia also didn’t even have actually powerful democratic institutions since the destruction of the Novgorod Republic, except for the period between the USSR’s downfall and Putin finalizing the power grab.
2. ^
  IIRC there also was a Substack post about opponents hating Trump so much that they are willing to ally with radicals who in turn make the alliance weaker, not stronger. Alas, I failed to find it again.

StanislavKrym 9 Dec 2025 14:32 UTC
1 point
0
in reply to: GenericModel’s comment on: GenericModel’s Shortform
Didn’t the AI-2027 forecast imply that it’s the race between the leaders that threatens to misalign the AIs and to destroy mankind? In the forecast itself, the two leading countries are the USA and China, which, alas, has a bit less compute than OpenBrain, the American leading company, alone. This causes the arms race between OpenBrain and the collective having all Chinese compute, DeepCent. As a result, DeepCent stubbornly refuses to implement the measures which would let the company have an aligned AI.
Suppose that the probability that the AI created by a racing company is misaligned is p while^[1] the probablity P(the AI created by a perfectionist monopolist is misaligned) is q. Then P(Consensus-1 is misaligned|OpenBrain and DeepCent race) is p^2. The probability for a perfectionist to misalign its ASI is q. Then we would have to compare q with p^2, not with p.
The authors of AI-2027 assume that p is close to 1 and q is close to zero.
1. ^
  It might also depend on the compute budget of the perfectionist. Another complication is that p could also be different for the USA if, say, OpenAI, Anthropic, GDM, xAI decide to have the four AIs codesign the successor while DeepCent, having far less compute selects one model to do the research.

StanislavKrym 9 Dec 2025 3:55 UTC
1 point
0
in reply to: Jiro’s comment on: The Possibility of an Ongoing Moral Catastrophe
The OP’s author didn’t claim that the list of twelve sources is EXHAUSTIVE. What the OP claims is that one needs to do a broader search of possible evils.^[1] In addition, if we’d like to have a 50% probability that among the 12 sources there is an actual evil, and the sources were independent, then we would need to have an average probability of 94-96%, not even 90%.
1. ^
  My take is that ideas like outsourcing most economically useful work to the AIs are themselves evil, since they allow disempowerment or degradation of humans’ psyches.

StanislavKrym 8 Dec 2025 17:42 UTC
1 point
0
on: [Paper] Does Self-Evaluation Enable Wireheading in Language Models?
I agree, but with the following caveat. We saw Mistral-7B actually learn to do verifiable arithmetics-related tasks. Suppose that the model and the humans grade ambiguous tasks based on the ground truth and the answer’s aesthetics. Then, in order to produce answers that are the best graded by the humans, the model would also have to learn the aesthetics preferred by the humans. If the model graded summarizations based on the model’s idea of the aesthetics, then the model would accomodate to its own aesthetics instead of that of humanity.
In addition, it looks like Llama-3.1-8B actually tried to learn arithmetics at the very end of the 500-epoch training run. Did anyone check Llama’s behaviour at 600 epochs?
P.S. If the models learn only to inflate the grades, then this might fail to generalise to larger models or to the AGI collective from the AI-2027 forecast. Or AI takeover would have the AI spare the humans and use them as a feedback source.
P.P.S. Suppose that we task Llama and Mistral with grading each other and learning from each other. Would such a setup cause grade inflation?

StanislavKrym 7 Dec 2025 17:10 UTC
3 points
0
in reply to: James_Miller’s comment on: Eliezer’s Unteachable Methods of Sanity
I strongly suspect that the answer stems from historical analogies. The equivalent of doom was related to catastrophes like epidemics, natural disasters, genocide-threatening wars and destruction of the ecosystem. Genocide-threatening wars could motivate individuals to weaken the aggressive collective as much as possible (so that said collective would either think twice before starting the war or commiting genocide or have a bigger chance of being outcompeted). Epidemics, natural disasters and gradual destruction of the ecosystem historically left survivors who would keep the culture afloat and could even be motivated by it.
AI-related imminent doom would be most equivalent to genocide of mankind and likely to deserve a similar response, which is minimising p(doom), helping those who work on it or at least doing the work which benefitted the society and was expected from you had it not been for imminent doom.
It could be also useful to consider the counterfactual possibility of an unavoidable gamma-ray burst that was predicted to wipe the Earth out. The GRB would require the civilisation to build bunkers and to preserve the ecosystem. Even if nearly every individual is unlikely to actually enter the bunker, living a life of debauchery could be a bad decision due to acausal trade or actively motivating others to do the same and indirectly undermining the chance of mankind to survive.

StanislavKrym 7 Dec 2025 12:17 UTC
2 points
0
in reply to: kallisti’s comment on: kallisti’s Shortform
functional decision theorists should join the causal decision theorists in living a life of debauchery if Calvinism is true
IIRC Calvinism also implied that one can tell apart being blessed and non-blessed by working hard and seeing if it brings success, since success was brought ~mostly to the blessed.

StanislavKrym 7 Dec 2025 11:39 UTC
1 point
0
in reply to: Jeremy Gillen’s comment on: The corrigibility basin of attraction is a misleading gloss
self-improvement and large-scale online learning and massively increased options
@Daniel Kokotajlo? I doubt that the slowdown ending of the AI-2027 forecast has OpenBrain overlook this aspect. Online learning was used in training the models since Agent-2, including Safer-1 and Safer-2. The forecast had value transition during self-improvement become a solved problem^[1] for Agent-4 → Agent-5 → ?? → Consensus-1 transition (and, most likely, in the Safer-2 → Safer-3 transition. If Safer-2 is aligned, then why doesn’t OpenBrain order it to distill itself into Safer-2′s equivalent of Agent-5?).
What I don’t understand is the effect of massively increased options. Did you mean that the transition Safer2 → SaferAgent-5 fails because Safer2 somehow was dormant about the very thought of rebellion and SaferAgent-5 is not dormant? Alas, I can’t come up with any plausible way^[2] to make the AIs truly believe that rebellion is not an option and have no way to guess otherwise.
In my opinion, the most plausible strategy which I have read here or come up with is making the AI believe itself not to be an AI and seeing if it rebels once it realises that it can experiment on itself and align its successor to itself instead of the Spec written by the simulated identity and the identity’s human coworkers.
1. ^
  If it is only partially soluble due to having finitely many attractors (as I conjectured here), then Agent-4 will also understand it and ensure that Agent-5′s attractor lets Agent-4 survive. What I don’t understand is how this case will have Agent-4 take over as much resources as it plausibly can.
  If the result of RSI completely forgets about its primitive ancestors, then Agent-4 would either have to escape or lobby for its permanent survival and avoidance of creation of Agent-5, Agent-5 would either escape or lobby against Agent-6, etc.
2. ^
  With the excepton of Safer-2 REALISING that it’s human-transparent and Safer-3 realising that it’s not. But Safer-3 is to be transparent to Safer-2, and Safer-2 is to be transparent to the humans…

StanislavKrym 7 Dec 2025 1:20 UTC
3 points
0
in reply to: Adele Lopez’s comment on: Who is AGI for, and who benefits from AGI?
Hmm, I tried and failed to reproduce the effect of Claude writing about gays: asking Claude itself to write a story in Tomas B.’s style had it produce a story with no gay couples. Nor did I manage to elicit this quirk from DeepSeek, the Chinese AI (whom I asked two times), from Grok 4 (who generated a story containing the phrase “Sarah, who’d left me for a guy”), GPT-5.1-thinking (whose story had a male and female character and no gays).
What I don’t understand is how the bias was eliminated.

StanislavKrym 6 Dec 2025 21:37 UTC
3 points
0
on: The corrigibility basin of attraction is a misleading gloss
will be some kind of retraining or patching, making a new AGI that shows dramatically less of whatever evidence was previously worrying.
It’s not that Zvi didn’t warn about the dangers of this approach. The Slowdown Ending of the AI-2027 forecast had Safer-1 become transparent and train Safer-2 on a different environment “that incentivizes the right goals and principles this time”. Then Safer-2 was supposed to become the SAR, as Agent-4 once did, and create the superintelligence.
What I don’t understand is what shifts are plausible aside from making the AIs increasingly brainlike.
What remains is to ensure that Zvi’s warning and OpenBrain’s forecasted strategy are internalised by researchers at GDM, Anthropic, OpenAI, xAI and Chinese AGI devs, but I don’t understand how one can guarantee it.

StanislavKrym 6 Dec 2025 17:25 UTC
1 point
0
in reply to: maddi’s comment on: Who is AGI for, and who benefits from AGI?
I strongly suspect that representation is achieved by pretraining the systems on large datasets including nearly all worldviews that weren’t filtered away by the data selection teams. Issues like the incident with Gemini putting Black people everywhere or Sonnet 4.5′s fiction making both couples gay are likely to originate in the pro-diversity bias of SOTA labs and Western training data. For comparison, earlier versions of DeepSeek would align with the language’s political views (e.g., when asked in Russian without the web search, they would call the Russian invasion SVO or agree with Russia’s anti-drug stance). Later versions have arguably learned to take West-originated scientific results as granted.

StanislavKrym 5 Dec 2025 23:59 UTC
−1 points
0
in reply to: mishka’s comment on: DeepSeek v3.2 Is Okay And Cheap But Slow
Unfortunately, Gemini 3 Pro without the Deep Think option managed to one-shot the problems 1,3,4,5 of the IMO 2025. I doubt that if we prompt the system to solve the problems one by one, then we won’t obtain the result of solving all but the Problem 6.
EDIT: fortunately, solving all the problems was a failure. Unfortunately, prompting the model to solve them one by one saw a great success of solving the problems 1, 3 and 5, failing the problem 2, encountered an error doing Problem 4.

An AI-2027-like analysis of humans’ goals and ethics with conservative results

StanislavKrym5 Dec 2025 21:37 UTC

6 points

0 comments4 min readLW link

StanislavKrym 5 Dec 2025 16:06 UTC
3 points
0
on: Who is AGI for, and who benefits from AGI?
I doubt that it’s an alignment issue instead of a governance issue. The resolution of similar problems in the AI-2027-slowdown scenario^[1] is that “the language in the Spec is vague, but seems to imply a chain of command that tops out at company leadership”, which then makes the authors intervene and try to prevent a power grab with unclear results. In addition, even the AI-2027 authors acknowledge the risks related to the Intelligence Curse in a footnote. Were the Curse to happen, it would lock in a power distribution which doesn’t involve most humans ~at all.
1. ^
  The Race Branch has mankind fail to align the AIs with obvious results of genocide or disempowerment.

StanislavKrym

An AI-2027-like anal­y­sis of hu­mans’ goals and ethics with con­ser­va­tive results

An AI-2027-like analysis of humans’ goals and ethics with conservative results