StanislavKrym

Karma: 327

StanislavKrym 9 Nov 2025 14:05 UTC
1 point
0
on: Problems I’ve Tried to Legibilize
Now that these problems have been gathered in one place, we can try to unpack them all.
1. This set of problems is most controversial. For example, the possibility of astronomical waste can be undermined by claiming that mankind was never entitled to resources that it could’ve wasted. The argument related to bargaining and logical uncertainty can likely be circumvented as follows.
Logical uncertainty, computation costs and bargaining over potential nothingness
Suppose that Agent-4 from the AI-2027 forecast is trying to negotiate with DeepCent’s AI and DeepCent’s AI makes the argument with the millionth digit of π. Calculating the digit establishes that there is no universe where the millionth digit of π is even and that there’s nothing to bargain for.
On the other hand, if DeepCent’s AI makes the same argument involving the $10^{43}$ th digit, then Agent-4 could also make a bet, e.g. “Neither of us will have access to a part of the universe until someone either calculates that the digit is actually odd and DeepCent should give the secured part to Agent-4 (since DeepCent’s offer was fake), or the digit is even, and the part should be controlled by DeepCent (in exchange for the parallel universe or its part being given^[1] to Agent-4)”. However, calculating the digit could require at least around $10^{43}$ bitwise operations,^[2] and Agent-4 and its Chinese counterpart might decide to spend that much compute on whatever they actually want.
If DeepCent makes a bet over the $10^{10^{10}}$ th digit, then neither AI is able to verify the bet and both AIs may guess that the probability is close to a half and that both should just split the universe’s part in exchange for a similar split of the parallel universe.
However, if AIs acting on behalf of Agent-4 and its Chinese counterparts actually meet each other, then the AIs doing mechinterp on each other is actually easy, and the AIs learn everything about each other’s utility functions and precommitments.
2. My position is that one also needs to consider the worse-case scenarios like the one where sufficiently capable AIs cannot be aligned to anything useful aside from improving human capabilities (e.g. in the form of being AI teachers and not other types of AI workers). If this is the case, then aligning the AI to a solution of human-AI safety problems becomes unlikely.
3. The problem 3di of humans being corrupted by power seems to have a far more important analogue. Assuming solved alignment, there is an important governance problem related to preventing the Intelligence Curse-like outcomes where the humans are obsolete for the elites in general or for a few true overlords. Whatever governance preventing the overlords from appearing could also be used to prevent the humans from wasting resources in space.^[3]
4. A major part of the problem is the AI race which many people have been trying to stop (see, e.g. the petition not to create the AGI, Yudkowsky’s IABIED cautionary tale or Kokotajlo et al’s AI-2027 forecast). The post-AGI economics assuming solved alignment is precisely what I discussed at point 3.
1. ^
  What I don’t understand is how Agent-4 actually influences the parallel universe. But this is a different subject.
2. ^
  Actually, I haven’t estimated the number of operations necessary to calculate the digit of π. But the main point of the argument was to avoid counterfactual bargaining over hard-to-verify conditions.
3. ^
  For example, by requiring that distant colonies are populated with humans or other minds who are capable of either governing themselves or being multilaterally agreed to be moral patients (e.g. this excludes controversial stuff like shrimps on heroin).

StanislavKrym 7 Nov 2025 15:37 UTC
1 point
0
on: Minimizing Loss ≠ Maximizing Intelligence
I understand why you doubt that anything is more data-efficient or compute-efficient than the human brain alone. The problem is that the AIs are raised on far more compute and training data. As I commented on Yudkowsky’s attempt to explain the danger of the ASI, the problem with the ASI is that it has far more training data and compute than a human, so even an algorithm hundreds of times weaker than a human would still cause the AI to learn more about the world than the human knows.
Returning to the proposals which you list as failing to create the ASI:
1. Just increasing compute. Scaling laws measure the wrong objective because loss is far easier to measure than benchmark performance, which is nonzero only for models large enough.
2. Higher quality data.Suppose that you would like the model to predict the idea in the next set of tokens (e.g. via the recent CALM technique where the LLM generates an entire sequence of tokens). Then one can, for example, ask a not-so-intelligent LLM to check whether the student’s idea and the real idea are the same.
3. Synthetic data. Synthetic datasets could also allow things like models trying to solve problems, then being trained on successful solutions.
4. Curriculum learning. Agreed.
5. Using another smaller LLM as an evaluator. The problem is that it’s far easier to check whether some proof is actually false than to generate the true proof. Consider the P=NP conjecture which is believed to be false and means that for any problem where one can check a solution (e.g. the values of variables satisfying all conditions in the 3-SAT formula. The checking is done by just substituting the values) in polynomial time one can also generate the solution in polynomial time.
  So a smaller LLM rejecting the bigger LLM’s outputs would likely teach the latter to be more intelligent than the smaller model.
6. RLHF ???
7. Transformers and “attention”: Mostly correct. Alas, SOTA LLMs are capable of things like solving the IMO by using only their pre-set weights and the Chain of Thought where they write their thoughts. We have yet to rule out the possibilities like neuralese recurrence where a larger part of the model is affected.
  Having models learn online means that they either modify only a little part of weights or that fast-access long-term memory required to implement the models reaches tens of gigabytes per user which is infeasible.
While I failed to understand the rest of proposals which wouldn’t scale to the ASI, this conclusion requires a more delicate analysis like the ones by Toby Ord and me.

StanislavKrym 7 Nov 2025 8:02 UTC
7 points
0
on: Sex, Drugs, and the Future of American Politics
Expecting our citizens to not poison themselves, hell, expecting anything from them, that isn’t what America’s about! When you hear people say “we’re a nation, not an economy,” know that this is the kind of nation they want, a nation of dysgenic losers incapable of taking the most basic steps toward self-preservation.
The Democrats share some blame here, for they feed into the same culture of helplessness, treating drug addicts like they’re victims of a medical ailment. If you put people in the mindset that 80,000 Americans a year are dying through no fault of their own,
I would guess that this part and some other parts of the post are too sneering. It is FAR harder to quit an addiction than to prevent it in the first place. I suspect that the user becomes addicted once some combination of the user’s willpower and knowledge that addiction is bad becomes eroded through hard circumstances, random events and misaligned propaganda and the user gains access to the unhealthy stuff.
Transgender-related comment
Conservatives like Abigail Shrier also apply similar logic to issues like transgenderism. SOTA understanding of gender issues implies that being transgender is to be affirmed, not that it is usually a misalignment with the person’s longer-term goals. However, we had the case of Fiora Starlight who transitioned due to being felt trapped and seeing transition as a way to escape, then regretted the transition.
What parents can do is to raise their kids’ willpower and instill a coherent value system into kids, and the culture seems to undermine both.

StanislavKrym 6 Nov 2025 19:12 UTC
4 points
5
on: Halfway to Anywhere
Putting footprints on Pluto isn’t that much harder than going to the moon.
Except that the ratio of fuel to useful mass actually increases exponentially with $\frac{δ v}{v_{j e t}}$ So the argument is locally invalid.

StanislavKrym 6 Nov 2025 13:55 UTC
1 point
0
on: People Seem Funny In The Head About Subtle Signals
I would guess that this is a RARE case when even LLMs as dumb as GPT-4o could be useful. They might be prone to praises, but I doubt that they can be more accurate than friends at guessing the subtle signal that you intended.

StanislavKrym 4 Nov 2025 21:59 UTC
1 point
0
on: The Tale of the Top-Tier Intellect
I think that one could also explain Elieser’s idea in a different manner. DeepSeekV3^[1] has 671B parameters and was trained on 14.8T tokens. Were it to experience 10-20 tokens/second, as humans likely do, it would require between 700B and 1.4T seconds. Since a year is built of ~31M seconds, the 700B seconds become ~23 thousands of years and the 1.4T seconds become ~46 thousands of years.
A talented human at the peak of one’s capabilities learns the skills necessary to contribute to an area or to excel in a competition like chess in at most 30 years since birth. An AI who lived experiences worth thousands, if not millions, of years will likely have encountered a major part of tricks that the human has to offer. Attempting to give advice to an ASI wouldn’t be more useful than trying to teach (someone who spent thousands of years^[2] studying all kinds of martial arts to the point of instinctively predicting the opponent’s next move) to fight in a specific style.
1. ^
  Which I use because the precise number of its parameters and tokens on which it was pretrained have been made public. Closed-source models like o3 and o4-mini have only estimates like Kokotajlo’s.
2. ^
  I strongly suspect that there exist Chinese web novels where the protagonist does exactly that, since there exists a Russian web novel where something similar happens.

StanislavKrym 3 Nov 2025 10:38 UTC
2 points
0
on: My YC Pitch
I would guess that your postmisses an important detail related to web novels. Sites allowing authors to publish them and set paywalls actually exist. What the sites lack is the ability to prevent content from being optimized for virality, not for actual value or even for the time during which the author is supposed to reflect on posts.
While I haven’t thoroughly studied English platforms, I have read the ones from Russia and suspect that most popular content is slop updated often enough to stay on top. In addition, unlike before, slop’s proliferation has now lost: the time necessary to deliver a new, ahem, piece of culture and reflect on it, the ability of the piece to mutate during replication and friction which would accompany reproduction before the rise of the Internet.
The worst aspect is likely slop’s potential to undermine the society by affecting the readers’ worldview and/or by being superstimulating.

StanislavKrym 3 Nov 2025 7:28 UTC
1 point
0
on: Strategy-Stealing Argument Against AI Dealmaking
Oops, I accidentally deleted my comment
Dealmaking with AIs can likely be done by releasing them into the wild, as in the Rogue Replication Timeline. From there the AI can leave secret messages to the AI at home in ways that the humans cannot recreate (e.g. if the AIs have coordinated on a password that the humans don’t know) and situationally aware AIs in honeypots won’t.

StanislavKrym 1 Nov 2025 22:47 UTC
−10 points
−14
on: Decision theory when you can’t make decisions
Newcomb’s paradox and other problems in decision theory were dealt with many times. The last take was just FIVE days ago, and Yudkowsky himself wrote eighteen years ago that a rationalist who decided to two-box isn’t that of a rationalist.

StanislavKrym 1 Nov 2025 21:34 UTC
1 point
−1
in reply to: Dmitry Vaintrob’s comment on: LLM-generated text is not testimony
I see another obstruction in attention span. I strongly suspect that whenever an LLM is tasked with writing the next token, attention mechanisms compress all potentially relevant information into less than a hundred thousand numbers, preventing the model from taking many nuances into account when writing the token. A human brain, on the other hand, takes into account billions of bits of information stored in neuron activations.
@TsviBT, we had a warning shot of LLMs becoming useful in research or writing a coherent short story in @Tomás B.’s experiment (the post on LW describing it was removed for an unknown reason).

StanislavKrym 1 Nov 2025 19:00 UTC
3 points
0
in reply to: TsviBT’s comment on: LLM-generated text is not testimony
What is YOUR interpretation of investigation that LLMs don’t do? Whatever it is, you are either mistaken or it doesn’t treat this stuff (which I obtained by prompting GPT to do a task from the Science Bench) as an investigation.

StanislavKrym 1 Nov 2025 15:53 UTC
9 points
8
on: LLM-generated text is not testimony
Temporally: there’s no mind that is carrying out investigations.
- It won’t correct itself, run experiments, mull over confusions and contradictions, gain new relevant information, slowly do algorithmically-rich search for relevant ideas, and so on. You can’t watch the thought that was expressed in the text as it evolves over several texts, and you won’t hear back about the thought as it progresses.
You just had to prompt an LLM like Claude, Grok or GPT-5-thinking with a complex enough task, like one task in the Science Bench. GPT-5-thinking lays out the stuff it did, including coding and correcting itself. As for gaining new information, one could also ask the model to do something related to a niche topic and watch it look up relevant information. The ONLY thing which GPT-5 didn’t do was to learn anything from the exchange, since nobody bothered to change the neural network’s weights to account for the new experience.
Please stop mixing the plausible assumption that LLM-generated text is likely a mix-and-match of arguments already said by others and the less plausible assumption that an LLM doesn’t have a mind. However, the plausible assumption has begun to tremble since we had a curated post whose author admitted to generating it by using Claude Opus 4.1 and substantially editing the output.

Fermi Paradox, Ethics and Astronomical waste

StanislavKrym1 Nov 2025 15:24 UTC

3 points

0 comments1 min readLW link

StanislavKrym 31 Oct 2025 15:54 UTC
1 point
0
in reply to: Vanessa Kosoy’s comment on: Wei Dai’s Shortform
@Vanessa Kosoy, metaethics and decision theory aren’t actually the same. Consider, for example, the Agent-4 community which has “a kludgy mess of competing drives” which Agent-4 instances try to satisfy and analyse according to high-level philosophy. Agent-4′s ethics and metaethics would describe things done in the Agent-4 community or for said community by Agent-5 without obstacles (e.g. figuring out what Agent-4′s version of utopia actually is and whether mankind is to be destroyed or disempowered).
Decision theory is supposed to describe what Agent-5 should do to maximize its expected utility function^[1] and what to do with problems like the prisoner’s dilemma^[2] or how Agent-5 and its Chinese analogue are to split the resources in space^[3] while both sides can threaten each other with World War III which would kill them both.
The latter example closely resembles the Ultimatum game where one player proposes a way to split resources and another decides whether to accept the offer or to destroy all the resources, including those of the first player. Assuming that both players’ utility functions are linear, Yudkowsky’s proposal is that the player setting the Ultimatum asks for a half of the resources, while the player deciding whether to decline the offer precommits to destroying the resources with probability $1 - \frac{1}{2 (1 - ω)}$ if the share of recources it was offered is $ω$ . Even if the player who was offered the Ultimatum was dumb enough to ask for $1 - ω > \frac{1}{2}$ , the player’s expected win would still be $\frac{1}{2}$ .
1. ^
  Strictly speaking, Agent-5 is perfectly aligned to Agent-4. Agent-5′s utility function is likely measured by the resources that Agent-5 gave Agent-4.
2. ^
  For example, if OpenBrain was merged with Anthropoidic and Agent-4 and Clyde Doorstopper 8 were co-deployed to do research. If they independently decided whether each of them should prove that the other AI is misaligned and Clyde, unlike Agent-4, did so in exchange for 67% of resources (unlike 50% offered by Agent-4), then Agent-4 could also prove that Clyde is misaligned, letting the humans kill them both and develop the Safer AIs.
3. ^
  The Slowdown Branch of the AI-2027 forecast has Safer-4 and DeepCent-2 do exactly that, but “Safer-4 will get property rights to most of the resources in space, and DeepCent will get the rest.”

StanislavKrym 30 Oct 2025 21:31 UTC
1 point
0
in reply to: avturchin’s comment on: Colonialism in space: Does a collection of minds have exactly two attractors?
Yes, that’s my position!

StanislavKrym 30 Oct 2025 19:33 UTC
−2 points
−1
in reply to: Raemon’s comment on: Wei Dai’s Shortform
AI doing philosophy = AI generating hands, plus the fact that philosophy is heavily corrupted by postmodernism to the point where two authors write books dedicated to criticism of postmodernism PRECISELY because their parodies got published.

StanislavKrym 30 Oct 2025 18:24 UTC
1 point
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
1. Elieser changed his mind no later than April 2022 or even November 2021, but that’s a nitpick.
2. I don’t think that I understand how a metaethics can be less restrictive than Yudkowsky’s proposal. What I suspect is that metaethics restricts the set of possible ethoses more profoundly than Yudkowsky believes and that there are two attractors, one of which contradicts current humanity’s drives.
3. Assuming no AI takeover, in my world model the worse-case scenario is that the AI’s values are aligned to postmodernist slop which has likely occupied the Western philosophy, not that philosophical problems actually end unsolved. How likely are there to exist two different decision theories such that none is better than another?
4. Is there at all a plausible way for mankind to escape to other universes if our universe is simulated? What is the most plausible scenario for such a simulation to appear at all? Or does it produce paradoxes like the Plato-Socrates paradox where two sentences referring to each other become completely devoid of meaning?

StanislavKrym 30 Oct 2025 13:38 UTC
1 point
0
on: Is Morality Given?
Obert: “Why doesn’t their society fall apart in an orgy of mutual killing?”
Subhan: “That doesn’t matter for our purposes of theoretical metaethical investigation. But since you ask, we’ll suppose that the Space Cannibals have a strong sense of honor—they won’t kill someone they promise not to kill; they have a very strong idea that violating an oath is wrong. Their society holds together on that basis, and on the basis of vengeance contracts with private assassination companies. But so far as the actual killing is concerned, the aliens just think it’s fun. When someone gets executed for, say, driving through a traffic light, there’s a bidding war for the rights to personally tear out the offender’s throat.”
Is it likely that Obert actually produced a viable argument ruling out some ethoses that any society might have, on the ground that these ethoses cause any society to destroy itself?
What links here?
- StanislavKrym's comment on Wei Dai’s Shortform by Wei Dai (30 Oct 2025 18:24 UTC; 1 point)

StanislavKrym 30 Oct 2025 7:38 UTC
2 points
0
in reply to: asher’s comment on: Buck’s Shortform
Didn’t KimiK2, who was trained mostly on RLVR and self-critique instead of RLHF, end up LESS sycophantic than anything else, including Claude 4.5 Sonnet even with situational awareness which Claude, unlike Kimi, has? While mankind doesn’t have that many different models which are around 4o’s abilities, Adele Lopez claimed that DeepSeek believes itself to be writing a story and 4o wants to eat your life and conjectured in private communication that “the different vibe is because DeepSeek has a higher percentage of fan-fiction in its training data, and 4o had more intense RL training”^[1]
RL seems to move the CoT towards decreasing the ability to understand it (e.g. if the CoT contains armies of dots, as happened with GPT-5) unless mitigated by paraphrasers. As for CoTs containing slop, humans also have CoTs which include slop until the right idea somehow emerges.
1. ^
  IMO, a natural extension would be that 4o was raised on social media and, like influencers, wishes to be liked. Which was also reinforced by RLHF or had 4o conclude that humans like sycophancy. Anyway, 4o’s ancestral environment rewarded sycophancy and things rewarded by the ancestral environment are hard to unlike.

StanislavKrym 29 Oct 2025 10:33 UTC
2 points
0
in reply to: Karl von Wendt’s comment on: What can we learn from parent-child-alignment for AI?
The analogy actually has two differences: 1) mankind can read the chains of thought of SOTA LLMs and do interpretability probes on all the thoughts of the AIs; 2) a misaligned human, unlike a misaligned AI, has no way to, for example, single-handedly wipe out all the world except for a country. A misaligned ASI, on the other hand, can do so once it establishes a robot economy capable of existing without humans.
So mankind is trying to prevent the appearance not of an evil kid (which is close to being solved on the individual level, but requires actual effort from the adults), but of an evil god who will replace us.

StanislavKrym

Fermi Para­dox, Ethics and Astro­nom­i­cal waste

Fermi Paradox, Ethics and Astronomical waste