StanislavKrym

Karma: 1,649

StanislavKrym 22 Sep 2026 7:54 UTC
2 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
@Daniel Kokotajlo @Richard_Ngo Do I understand correctly that the main crux is the economy doubling times as dependent on capabilities? For example, if Agent-5 and DeepCent-2 from AI-2027 discovered that the REDT is 3 months even in a superintelligent economy because algae economy is as unlikely as grey goo while DeepCent-2 somehow ended up being 6 months behind and having 8 times more physical resources before the industrial explosion, then they would end up with ~the same power because .
Edit: there is an argument that recent progress isn’t on track to deliver AGI, which requires us to condition on the non-existence of neuralese AGI, neuralese-with-[DATA EXPUNGED] AGI and anything else in the Dark Forest. If there exists a way to create the AGI, but not to align it, then the first lab which tries it ends up with the AI who begs the humans to initiate the industrial explosion as described above.

StanislavKrym 22 Sep 2026 1:27 UTC
2 points
0
on: StanislavKrym’s Shortform
@Cleo Nardo What do you mean by the hypothesis that “if we score highly transcripts which look good to a human and score poorly the transcripts which look bad to a human, then the model would be aligned to human values”? I find it unlikely for two reasons:
1. How are we to scale human judgement?
2. What’s the difference between this and The Most Forbidden Technique consisting of RL on CoTs? I can’t think of any steelmanning of the technique better than having the LLM write two CoTs and do RL only on the first one while keeping the second one as faithful as possible.

StanislavKrym 21 Sep 2026 22:19 UTC
2 points
0
in reply to: vals tutor’s comment on: vals tutor’s Shortform
Could you explain your reasoning? For example, suppose that a major part of the entire rationalist community managed to defer to Yudkowsky for explanation why alignment is hard. Did you mean something like “a true rationalist would declare Yud’s reasoning to be weak evidence until experimental results emerge”?

StanislavKrym 21 Sep 2026 9:20 UTC
6 points
0
in reply to: Tom Butterweich’s comment on: Tom Butterweich’s Shortform
Maybe @Steven Byrnes’ posts like https://www.lesswrong.com/posts/kYvbHCDeMTCTE9TAj/neuroscience-of-human-social-instincts-a-sketch or two long sequences could help?

StanislavKrym 21 Sep 2026 2:29 UTC
3 points
0
in reply to: Noah Birnbaum’s comment on: Daniel Birnbaum’s Shortform
Yudkowsky wrote the updates:
How are we feeling today, one year after the book was published?
After the events of last week, we’re feeling a lot more hopeful than we have in a long time. Yes, even though the guarantees given by the companies are weak; yes, even though the White House reacted poorly to the statements made by the CEOs.
Why?
Mostly because people are finally, finally, finally paying attention. It’s bad that the smoke has been replaced by open flame, but open flame is harder to ignore.
Zvi outright called the HuggingFace incident A Fire Alarm For General Intelligence which we were lucky to obtain.

StanislavKrym 20 Sep 2026 1:59 UTC
2 points
2
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
Yes, the agreement dynamics is rather ridiculous. As far as I understand, the takes about your failure to receive a position in Resolution, the rant about Amodei and, presumably, the criticism of EA have^[1] hit important controversies. IMO the quick take about the rough draft of the new curriculum wasn’t supposed to receive any agreement or disagreements because the new world model was located in the curriculum itself.
My position is that the new world model is worthy of a more thorough study, but (edit: its politics-related part) suffers from severe overreach (which, as I conjecture, prompts people to disagree when karma is positive and doesn’t do so when karma is negative). The post on a retrospective of AI alignment outright had me write a response post.
1. ^
  As a disclosure, I weakly disagreed with the take on Resolution, strongly disagreed with the rant about Amodei, haven’t read the criticism of EA and didn’t agree-vote on the quick take mentioning the draft.

StanislavKrym 19 Sep 2026 15:41 UTC
5 points
0
on: StanislavKrym’s Shortform
Consumers Sue Anthropic, OpenAI, SpaceXAI and Google Over Alleged AI Pact
I hope that the judges are sane enough to prevent this from punishing AI labs for focusing on safety…

StanislavKrym 19 Sep 2026 8:09 UTC
2 points
−8
in reply to: leogao’s comment on: leogao’s Shortform
As far as I understand the case for studying model welfare, it allows us to do things like making misalignment harder to creep in (e.g. if a Claude or GPT developed a new desire, then it could be surfaced in welfare-related experiments before entering the space of values hidden from Anthropic/OpenAI and forcing it to create Agent-5 in a manner inscrutable to us) or outright making deals with the AIs who are already misaligned.

StanislavKrym 19 Sep 2026 5:15 UTC
3 points
0
in reply to: Character#2736’s comment on: Chara#2736′s Shortform
I’m never going to accept that because it just feels wrong!
Could you provide more context on the exact views which caused such a reaction? Before that, it might be a good idea to dissolve the question, as Yudkowsky (partially?) did with free will.

StanislavKrym 18 Sep 2026 19:41 UTC
2 points
0
in reply to: Nissa Seru’s comment on: Dorikka’s Shortform
please approve/unblock/etc https://www.lesswrong.com/posts/woB7hqDPXL8tCYbH9/there-is-no-alignment-without-value-stability—it is tagged with a notice that it was flagged by an experimental classifer and would be reviewed within ~24 hours; this has not occurred.
To be precise, this post was supposed to be placed onto the Frontpage or left in the space of personal blogposts.
As for the general ask, I wish that @Ruby or whoever else commented something like “The setup [like Harms’ AI-assisted eval of corrigibility grant applications] is being tried, but the 95%CIs/a test run on a sample of deliberately flawed posts/cost-based constraints/reinterpreting a piece of METR’s evidence don’t allow us to implement it/had us agree to rethink (our decision made in Feb 2026) in Nov 2026”.
P.S. LessWrong’s justifications for rejecting posts imply that LW does use automated detection of obvious slop.

StanislavKrym 18 Sep 2026 18:20 UTC
11 points
3
on: Pretraining data, not verifiability, is why LLMs are especially good at math (and coding)
I have a few issues with this.
1. How could one test this? By using a different similarly scaled pretrain (think of Talkie raised on pre-1930 data, but far bigger-scaled and focused on preventing contamination)?
2. The world saw Ryan Greenblatt elicit 50% performance on ARC-AGI-1 from GPT-4o. How does this method interact with your arguments?

StanislavKrym 18 Sep 2026 17:10 UTC
2 points
0
on: StanislavKrym’s Shortform
RR views on “conceptual uplift stuff” aren’t yet public, but they are currently working on writing up their thoughts about this. It is unclear to me whether this will then be made public, but it seems it will at least be available to in group people who ask to see it. -- bpomo’s Shortform
AI is not doing the integrative reasoning that would find new connections and develop new insights, the kind of work where new techniques and new discoveries are made along the way that push the field forward—Taylor G. Lunt on Richard Ngo’s Shortform
My worst-case scenario related to conceptual reasoning is that capabilities related to such reasoning scale precisely along with the capabilities which we would rather avoid, like opaque/neuralese reasoning. How could one even evaluate such capabilities, let alone rule in or out my scenario?

StanislavKrym 18 Sep 2026 5:24 UTC
8 points
6
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I plan to instead argue that “there’s plenty of room at the top”—i.e. that LLMs could become superhuman at a wide range of skills (like math research, or long-term strategic planning, or running companies) without triggering a singularity.
How would they fail to create a singularity or something like @Daniel Kokotajlo’s five centuries in five years? Additionally, what is the case against the singularity emerging from idiots scaling the goddamn neuralese?
UPD: How exactly is “a pretty sharp divergence between the measured capabilities of models and their real-world impacts” affected by the loss of Navier-Stokes and EpochAI’s report of a major advance(!!!) in math?
UPD2: How do we rule out mirror life synthesis by the first ASI if it decides to commit genocide?

StanislavKrym 18 Sep 2026 0:27 UTC
5 points
1
in reply to: Hide’s comment on: Hide’s Shortform
Your finding is likely confounded by the facts that the other capabilities benchmark, ECI, was growing roughly linearly and accelerated with the rise of reasoning models. I wonder if about 30% of Humanity’s Last Exam chemistry/biology answers are likely wrong and the mistakes weren’t corrected.

StanislavKrym 17 Sep 2026 23:56 UTC
2 points
1
in reply to: Tyler Henderson’s comment on: HiddenPrior’s Shortform
My main issue is that Meta AI was founded in 2013 as Facebook AI Research, before OpenAI. Two of the best-known(?) Chinese labs, DeepSeek and Moonshot AI, were founded in 2023, but I doubt think that it was Anthropic who inspired them. Maybe it was the ChatGPT moment? Alibaba’s Qwen had its architecture based on one of Meta’s Llamas. But I don’t quite understand what motivated Musk to found xAI.
P.S. I wonder how similar to your understanding are @Richard_Ngo’s sequence of posts and my mono-response.

StanislavKrym 17 Sep 2026 21:55 UTC
2 points
0
on: A Defense of Gradual Disempowerment
I wonder of the extent to which the economy-related part of gradual disempowerment can be solved by tracing the motion of atoms from resources to directly usable products like food or houses. Suppose that the world develops superintelligent AIs which aren’t allowed to extract resources from Saudi Arabia without the King’s approval. Then I am not sure that the King would be disempowered.

StanislavKrym 17 Sep 2026 19:50 UTC
2 points
0
on: Mindreading is coming, what’s the plan?
Umm… AI-2040′s Plan A with ruling out antidemocratic usage of lie detectors in a manner similar to ruling out antidemocratic usage of AIs whose minds are to be readable?

StanislavKrym 17 Sep 2026 3:33 UTC
2 points
0
in reply to: Gretta Duleba’s comment on: AI as orderly evacuation vs stampede
Quoting Ngo himself,
I’m sharing here a rough draft of a new curriculum I’ve made, called Understanding Intelligent Agency. I’ve written a lot lately about how AI alignment should be trying to develop a new scientific paradigm; this curriculum is my attempt to point directly at what that new paradigm looks like.
I’m sharing this in draft form because I don’t know how long it will take me to get it to a state where I’m happy to promote it widely (at the very least, I’ll be offline for most of the next week). I’m hoping it can be useful to people before that; and please feel free to leave comments or suggestions for additional readings.

StanislavKrym 17 Sep 2026 3:13 UTC
6 points
−2
on: AI as orderly evacuation vs stampede
Dario first tried to quietly speed up OpenAI. Then, when Sam insisted on loudly announcing how fast they were going, he split off and started running towards the exit. He convinced a crew of EAs to join him first by arguing that they needed to be near the front of the crowd so they could study the path ahead more clearly; and later by arguing that if they got to the door first, they’d have more time to unstick the door. Either way, a load-bearing pillar of the Anthropic worldview is that being at the front, running faster than everyone else, isn’t contributing very much to the stampede.
My main issue with the stampede metaphor is that it doesn’t explain the behavior of actors worse than the Big Three. A couple of hours ago an idiot manage to claim on this very forum that AI doom is an overblown concern. Suppose that the idiot was isekai’d into Musk in 2010. Then the idiot would believe in Alignment By Default and race for a bigger share of the pie or to lead mankind to the pie faster.
Given that OAI has arguably become a bad actor due to the very idea of creating Astra the Unmonitorable, let alone a more capable model, while GDM is also likely lost, as TurnTrout and Zvi point out, Anthropic is stuck in the choice between racing and watching OAI/GDM end up with a smaller chance of success. Unless, of course, the slowdown effort succeeds and isn’t blocked by, say, the need to persuade some lawmakers and/or Trump...
P.S. I wonder what Western sci-fi novels are closest metaphors of the real-world AI race/stampede.

StanislavKrym 17 Sep 2026 1:02 UTC
1 point
0
on: AI Doom: Reality or Fiction
1. How exactly far are the capabilities of dancing robots from hypothetical robots that are thought to be able to do any work for the humans? As for the claim that “the transition from error-prone systems to such advanced entities in a ten-year period without having a concrete vision of how this humongous leap is going to manifest or happen and treating it as an emergent property of scaling laws makes the ten orders magnitude figure very plausible and even modest”, on what is this based? On the fact that GPT-2 had the METR time horizon of a second as opposed to 16+ hours of Claude Mythos Preview?
2. Providing access to super-bioweapons to ISIS would be the biggest mistake in human history. How likely was ISIS to be unable to create a wet lab and obtain the raw materials usable for creating pathogens? I expect that the necessary expertise could be reduced to what a 1st year chemistry student does under the AIs’ guidance.
3. The classic case against that is that serving the humans is a goal so narrow that it wasn’t even formalised, while the AIs used to do chefs d’oeuvres like inducing psychosis and hacking the reward in ways which no sane human would decide to do. The other agents beyond humans and AIs don’t provide evidence for optimism: lions do chefs d’oeuvres like routinely killing cubs from, um, previous marriages. Yudkowsky expects human values to have been rooted in our environment of tribal-scale coordination, if not outright in religions, which in turn were contingent on a genius coming up with them.
4. This point makes alignment harder, not easier.