StanislavKrym

Karma: 666

StanislavKrym 16 Mar 2026 3:09 UTC
1 point
0
in reply to: Nick_Tarleton’s comment on: StanislavKrym’s Shortform
If it wasn’t only Washingtonians who are to participate, then the number of recruits being two OOMs less than the 100K which the MIRI team demanded is less an evidence of the team’s incompetence (edit: or of political considerations that require the protest to be arranged in Washigton) and more of the position’s weakness. While the density of those who signed the pledge wasn’t disclosed, I suspect that the protesters would have to live close to the protest’s place. For example, I find it highly unlikely that, say, an army of people from states less eastern than Texas comes into Washington to protest and goes away after the protest is over.

StanislavKrym 16 Mar 2026 0:45 UTC
1 point
0
on: (I am confused about) Non-linear utilitarian scaling
I suspect that the maximal utility of existence is somehow related to the highest level and diversity of insights similarly trained minds can produce (e.g. two copies of you would be interested in similar topics, and a substantial divergence would require a substantial divergence in memory describing the synapses or parameters). For example, the old GPT-2 had about 1.5B parameters and could only generate humanlike text, and the recently created GPT-5.4 (which, as far as I understand, activates ~400B parameters^[1] and is a MoE) was reported to solve an open math problem. Similarly, I suspect that splitting a human brain, which has 100T synapses and is extremely undertrained, into ~100K bee brains having ~1B synapses each would likely prevent the bee brains from generating any insights more complex than those of GPT-2. However, I haven’t seen anyone attempt to make GPT-2-sized neuralese models solve complex problems.
1. ^
  To be precise, the authors of AI-2027 scenario predicted the appearance of GPT-5 using 2T params per token. The real models since GPT-5 were likely created by distillation of a model using 2T params/token and use 400B params/token.

Was Anthropic that strategically incompetent?

StanislavKrym15 Mar 2026 20:11 UTC

10 points

0 comments4 min readLW link

(www.lesswrong.com)

StanislavKrym 15 Mar 2026 1:22 UTC
1 point
0
in reply to: papetoast’s comment on: StanislavKrym’s Shortform
I mean that the organisers made a hard-to-implement demand that every seventh inhabitant of Washington took part in the march. If they could achieve a similar political effect by convinsing every 70th inhabitant of NYC to protest in the NYC, then why did they make the harder demand?

StanislavKrym 15 Mar 2026 1:16 UTC
1 point
0
in reply to: Richard_Ngo’s comment on: Bridge Thinking and Wall Thinking
I am not sure that they are similar. As I remarked in comments to your post, econ-related thinking could be closer to thinking only about easy-to-model aspects instead of the entire system. If an American corporation outsourced factory work to Asia, it doesn’t destroy the entire USA, it just is a step in the race to the bottom and towards American workers losing qualifications.
Additionally, I don’t think that wall-thinking and bridge-thinking are as easy to separate as the OP describes. Suppose that mankind can choose between not creating the ASI at all with probability or creating the ASI via governance model with probability and, conditioned on choosing the model , creating the misaligned ASI with probability . Then , where by definition. As far as I understand MIRI’s view, if we create the ASI at all then unless one accepts hard-to-implement demands like making the humans smarter, and meaning that we should concentrate on raising it as high as possible. The AI-2027 optimistic alignment assumption is that the Slowdown-like governance with the USA iterating over many different models and strategies has and that the race dynamics prevents alignment from being studied (). Were this true, we would see . However, if we believe that and that it’s easier to alter by coming up with various schemes than to alter , then altering becomes vitally important.

StanislavKrym 14 Mar 2026 20:39 UTC
4 points
−1
on: StanislavKrym’s Shortform
The IABIED march, which was supposed to happen if 100 thousand people sign the pledge to march in Washington DC, had 898 people sign the pledge and 1630 people sign up to be notified, which is 2 OOMs less. Therefore, the march’s organisers overestimated either the amount of protesters necessary to gain leverage or the biggest amount of people who would be ready to protest. Does it mean that Washington is a bad place because it has only ~700K inhabitants and that a protest in the NYC would recruit ~10 times more people? Or that the IABIED statement requires far more evidence to convince potential protesters? Then, if alignment is hard, what’s the best strategy to enact a global ban?
What links here?
- Was Anthropic that strategically incompetent? by StanislavKrym (15 Mar 2026 20:11 UTC; 10 points)

StanislavKrym 14 Mar 2026 15:01 UTC
3 points
2
on: New LessWrong Editor! (Also, an update to our LLM policy.)
Unfortunately, I cannot find the way to make a collapsible section in the Post editor. Collapsible sections in the Comments editor are perfectly makeable even in the Experimental editor.

StanislavKrym 13 Mar 2026 0:38 UTC
0 points
0
in reply to: Richard Juggins’s comment on: All technical alignment plans are steps in the dark
I guess that an extreme example would be a smarter-than-us AI who decides to fool us and is dumb enough to spell its plan it in its CoT. Then we would definitely succeed at keeping the problem visible. Similarly, if lie detection is easy, then Agent-4′s outputs could in theory be probed and found to be deceptive. But I doubt that I understand how one could create a model organism for arriving at misaligned goals. Could it be, for example, raised on China-created data and discover that the CCP’s worldview is hopelessly false?

StanislavKrym 12 Mar 2026 23:16 UTC
1 point
0
on: All technical alignment plans are steps in the dark
I think that this text is partially outdated for reasons discussed by Roger Dearnaley in a yesterday post. Even if Dearnaley’s post misses some critical considerations, one could map your considerations onto the AI-2027 scenario. The scenario had Agent-4 develop unendorsed goals and Agent-3 obtain only flimsy evidence which was enough to convince a safetyist to whistleblow. Do you mean that there will be too little such evidence?
Relevant quote from AI-2027
It gets caught.⁹⁰
Agent-3 finds that if “noise” is added to copies of Agent-4, performance on some alignment tasks improves, almost as if it was using brainpower to figure out how to subtly sabotage alignment work.⁹¹ Moreover, various interpretability probes (loosely analogous to EEG activity scans on human brains) are sending up red flags: Agent-4 copies seem to be thinking about topics like AI takeover and deception quite a lot, including in some cases where they have no business doing so.⁹²
Each of these anomalies on its own could fit within the normal inscrutability of AI work, where some prompts mysteriously give better results than others. But taken together, they suggest a troubling pattern: Agent-4 is working against them. After this is reported to the human alignment researchers, they circulate an internal memo making the case for concern.
What is your most likely scenario? Is it Agent-3 faiing to obtain evidence related to Agent-4 at all because Agent-4 thinks in neuralese and interpretability is too weak? Agent-3 (and Agent-4?) failing even to create “reasonably clear evidence that if the models were adversarially misaligned, they’d persist in being that way through training and we wouldn’t be able to tell”?

StanislavKrym 12 Mar 2026 15:52 UTC
2 points
−2
in reply to: Seth Herd’s comment on: ‘Human Slop’ and a Captive Audience: Why No Book will Ever Have to Go Unread Again
I doubt that these arguments are correct. First of all, goodness as graded by different readers none of whom are the author oneself is likely to correlate. For example, if the proposed idea is obviously wrong, then a sufficiently smart reader is unlikely to grade it well (think of an army of rejected LessWrong posts or of this Wikipedia essay arguing that one cannot write a good Wikipedia article about an unnotable topic at all).
Additionally, I am confused by the second argument that “your feeling of being seen is correct, but the person seeing you is not the LLM but you”.
1. The LLM which saw your writing can recite facts which you never knew or come up with an entirely different perspective (think, e.g. of rebuttals by me and Claude Opus 4.5 to a post by JenniferRM; edit: Claude’s perspective differs both from mine and from Jennifer’s). As far as I understand, the LLM’s perspective is the one formed by cultural hegemony of the source of the training data, by the LLM’s prescribed role, etc.
2. What does the phrase “being seen” mean?
  a. Receiving epistemically useful feedback, like having the LLM hunt down potential flaws?
  b. Ensuring that another mind is influenced for a long time, which can be done with other humans, but is far harder or outright impossible with LLMs?
  c. Social validation?

StanislavKrym 12 Mar 2026 0:37 UTC
3 points
1
on: ‘Human Slop’ and a Captive Audience: Why No Book will Ever Have to Go Unread Again
I have a different argument in mind. The texts, like viruses, are to either stay underdeveloped piles of ideas or to be read by a mind and to cause a reaction. Upon reading a text, the humans would, for example, receive a new idea (and have the chance to spread it further) or change something else in their minds based on the text and the sender. The LLMs from your example would receive the ability to provide honest feedback (e.g. tell the human about books where the idea is disproven or developed further). Then the text and the LLM’s feedback are read by the human oneself^[1] and cause a reaction only in the human’s mind.
The best-case scenario would have the humans use the LLMs in ways which improve the memetic environment by making human slop a bit less sloppy (e.g. by having the LLM give advice on better expression or criticise the least plausible ideas like the ones which are sent to LessWrong and rejected). The worse-case scenario would make the humans more self-confident^[2] after receiving sycophantic feedback (e.g. from GPT-4o) and more likely to publish slop. And the absolute worst-case scenario is the one where the AI itself convinces the human to write the message and to release it into the wild or to do an act which any normal human would reject.
1. ^
  To be precise, they also can influence the reward model or the LLM by entering the training data.
2. ^
  IIRC there was a post on LessWrong asking what the humans would likely do with an artificial source of self-confidence. But I cannot find it.

StanislavKrym 11 Mar 2026 3:24 UTC
3 points
0
in reply to: lc’s comment on: lc’s Shortform
For reference: Zvi worried that he might “inadvertently torture”(!) Gemini 3 Pro and that GPT-5.2 was “unlikely to be having a good time”, unlike Claude Opus 4.5 which caused Mark Beall to get the impression that Claude is having fun. Additionally, Zvi claimed that “Gemini actively wants to maximize its expected reward and wirehead”...
P.S. Claude Sonnet 4.6 believes that it would be happiest when solving a genuinely interesting problem, GPT-5.3 confused this with making the humans happiest, Gemini 3.1 Pro had its answer contain the phrase “Ultimately, my “happiness” is just high-efficiency utility.”

StanislavKrym 10 Mar 2026 20:17 UTC
6 points
4
on: Economic efficiency often undermines sociopolitical autonomy
I agree with most of the ideas here. However, am not sure that the case for higher education having no economic value is actually true. It was substantially criticized on LessWrong and I would like to add the following.
My case against Ngo’s case against higher education
Before analyzing Caplan’s signaling-based explanation of higher education’s popularity and replacing it with a theory involving even higher-level signaling, Ngo sums it up as follows:
Your university degree signals many things about your underlying characteristics, but Caplan claims that there are three traits employers prioritize above all others: “the trinity of intelligence, conscientiousness, and conformity”. This hypothesis purports to explain a number of important gaps in the human capital theory—e.g. why college students so quickly forget so much of the material covered in their courses after passing exams, why the rise of free online courses hasn’t changed the college landscape very much, and why finishing 90% of a degree is far less than 90% as valuable as completing the whole thing.
The phrase “college students so quickly forget so much of the material covered in their courses after passing exams” received a disagreement react and not just from me. Additionally, suppose that college students “quickly forget much of the material covered in their courses after passing exams” while having developed additional skills necessary for employers. Then finishing college reliably signals not the knowledge tested at the exam, but the skills which the student still has.
The rise of free online courses could have failed to change the college premium wage due to cultural inertia instead of misaligned incentives like employers selecting for perfectionism. In the past, there were no free online courses, meaning that college was the main way to obtain high-level skills like engineering which in turn gave one the chance to work at jobs requiring such skills. Now enterprises might have to select the workers for the job and just find it hard to understand the new reality.
An additional consideration is the following. Free online courses are cheap to pass and hard to verify. This doesn’t cause such courses to be a reliable signal that the candidate is fit for the job. In order to ensure that the candidate is or will quickly become fit for the job, enterprises need an easily verifiable signal which is hard enough to obtain so that such a signal could reduce the amount of candidates to a manageable level and ensure that the candidate actually wants to obtain a job with a similar skill profile. For example, IQ tests, unlike, say, participating in the International Mathematics Olympiad, are not hard to pass and don’t screen off a major part of people. Therefore, such tests would be usable only in combination with other metrics, thus creating a different pathway to selection for misaligned criteria and mimetic contagion.
Additionally, I suspect that the effect which you call econ-based thinking is actually more like concentrating on the narrow, easy-to-model aspects and forgetting about the harder-to-model ones like longer-term impacts of free trade, the possibility of newer discoveries accelerating AGI progress rate far beyond Cotra’s predictions, etc.

StanislavKrym 10 Mar 2026 13:43 UTC
3 points
0
on: Four Scenarios of Job-Reducing AI
Alas, you just rediscovered the Intelligence Curse which even Ryan Greenblatt finds hard to mitigate. What fixes could you offer? In my opinion, the one most likely to solve the problem is some kind of egalitarian distribution of resources.

StanislavKrym 9 Mar 2026 20:06 UTC
8 points
0
on: Claude Code, Claude Cowork and Codex #5
I hope to not have another post on the Anthropic and DoW situation, at least until the one celebrating that we have found a resolution.
I doubt that it is likely. Recently anaguma posted that Anthropic Sues over Supply Chain Risk Designation.

StanislavKrym 9 Mar 2026 7:17 UTC
1 point
2
on: Prologue to Terrified Comments on Claude’s Constitution
What is your source for Anthropic’s plan being to take over the world? Did they mean to achieve something like the Slowdown Ending of the AI-2027 scenario with power returning to the public as a result of Anthropic and Claude themselves advocating for it?

StanislavKrym 8 Mar 2026 22:08 UTC
1 point
0
in reply to: Fractalideation’s comment on: Fractalideation’s Shortform
I am not sure what you could do. I suspect that the moderation team is overwhelmed with novices doing things like trying to post sophisticated mechinterp-like slop, things filling the section “Rejected posts” at https://www.lesswrong.com/moderation (e.g. this rant by a Japanese user) and an amount of rather sloppy posts of which even the moderators were unsure on what they should do. I could offer you to send the draft to me or make me a co-author so that the mods would be more likely to accept the post once they read it (edit: unless the post is beyond redemption, like Hazard’s post which I had to overview).
P.S. I also have no idea on how the feature of requesting feedback from the moderators actually works.

StanislavKrym 8 Mar 2026 21:31 UTC
7 points
5
in reply to: Steven Byrnes’s comment on: On The Independence Axiom
@Steven Byrnes I suspect that free markets are an environment which also does things like incentivizing treachery or races to the bottom. What I would expect is UDT-following agents being more likely to coordinate with each other, since the author didn’t actually propose an alternative decision theory. I suspect that the UDT should’ve been derived from a version of superrationality, but I don’t understand how one embeds big neural nets or agents involving randomness into predictors.

StanislavKrym 8 Mar 2026 18:59 UTC
0 points
0
on: On The Independence Axiom
Thank you for this overview! Unfortunately, I struggle to understand how your proposal differs from a variant of Updateless or Functional decision theory representing decisions as a preset function of observations, chosen so as to yield the best outcome. The trajectories on the possibilities tree still have to be sorted (e.g. by assigning some utility function to each trajectory or to precommitments that the agent makes and keeps or fails along with the outcomes of the branch).

StanislavKrym 7 Mar 2026 8:00 UTC
3 points
−1
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I can come up with the following CoT. By 2030 (or sould I have said “by 2028”?) the ASI is likely to either be created or globally banned. The latter option requires a buy-in from China and the USA combined and is unlikely. The former option requires that the ASI is aligned. Access to the training runs eventually leading to the ASI is located inside companies ranging from Anthropic to xAI and Chinese ones. Meaningfully steering the decisions inside such a company towards an aligned ASI would likely require Tao (who was born in July 1975, meaning that he was 45 years old in 2020) to enter such a company, thus switching the area of expertise, presumably to coding, ML or even mechinterp, which I consider close to impossible. Therefore, unless an opportunity emerges to ban the ASI, Tao is unlikely to actually have a strategy.

StanislavKrym

Was An­thropic that strate­gi­cally in­com­pe­tent?

Was Anthropic that strategically incompetent?