StanislavKrym

Karma: 1,071

StanislavKrym 21 Jun 2026 7:54 UTC
2 points
0
in reply to: interstice’s comment on: interstice’s Shortform
Yes, I would be very interested in reading such a blog. I am also interested in a more pragmatic question of how actual alien life could have become intelligent and created a civilisation, for which there could be relevant posts like Cannell’s Brain Efficiency (which I assume to rule out bigger neural nets on the ground of energy-based constraints), a nonexistent post on scaling laws of brains depending on time lived (in a manner similar to Claudes’ trend (or logarithm thereof?!) over time?) and another nonexistent post on how a civilisation would find it HARD to emerge before specific conditions caused the Universe to create habitable planets and to let life actually grow there.

StanislavKrym 19 Jun 2026 18:52 UTC
4 points
1
in reply to: habryka’s comment on: Habryka’s Shortform Feed
advocating for regulating AI on the basis of harms like this is bad, will eventually backfire, and should be modeled as a cost, not as a benefit, from the perspective of predicting how useful his actions will be on AI safety topics.
My main crux is the societal awareness^[1] around capabilities of existing and future AIs and robots. I once remarked that laypeople do NOT foresee AI taking over the world because they fail to understand^[2] the true extent of AIs’ transformative potential to the point of believing that the AIs have zero reasoning abilities.
A layperson who believed such nonsense would also believe that the AIs are at most toys capable of doing things like driving people into psychosis or destroying people’s relationships, but not of wiping mankind out and sustaining the civilisation. Therefore, I don’t understand how a safetyist can win the 2026 election while not making Bores-like factual errors.
UPD: on March 14 I remarked that the IABIED march had gathered 898 pledgers and 1630 people who signed to be notified of IABIED’s activities. By June 19 these numbers reached only 1105 and 1991.
1. ^
  UPD: A similar point was made by Boaz Barak on the fourth fake graph.
2. ^
  Examples include Freddie de Boer’s attempt to drive home the point that the AIs are unlikely to acquire such capabilities, Casey Simpson’s video full of factual errors or a video released in MAY 2026 whose author claims that AI is a technology with no lived experience and zero reasoning abilities. The latter video was released AFTER GPT-5.4 Pro solved an Erdös problem, but before the triumph of an unreleased OpenAI model.

StanislavKrym 18 Jun 2026 12:32 UTC
0 points
−2
in reply to: RobinHa’s comment on: How far do open weights trail the frontier?
I would spread untrustworthiness to open-weight companies which operate at far lower security standards. And that’s ignoring the threat of rogue replication which open-sourced models find easier to enact...

StanislavKrym 18 Jun 2026 12:00 UTC
1 point
0
in reply to: Kabir Kumar’s comment on: Kabir Kumar’s Shortform
I wonder on which sense Yudkowsky’s first interpretation was bad. When his long list of reasons for AGI to be deadly was reevaluated in 2026, the most load-bearing crux from the point 25 stayed unsolved and Yudkowsky’s other disproven ideas didn’t seem enough to super-reliably avert the disaster.

StanislavKrym 18 Jun 2026 11:30 UTC
3 points
0
on: How far do open weights trail the frontier?
Thank you! I always wanted such a site to exist so that we could track Chinese progress as opposed to American one. Additionally, I’d like to add the ability to track the progress of untrustworthy companies like xAI as opposed to that of trustworthy ones.

StanislavKrym 18 Jun 2026 4:59 UTC
8 points
3
in reply to: Knight Lee’s comment on: Knight Lee’s Shortform
If GLM 5.2 is as performant as GPT-5.5, then the AI-2027-like race is looming. I hope that @Zvi provides more details...

StanislavKrym 17 Jun 2026 11:02 UTC
3 points
0
in reply to: IanWS’s comment on: IanWS’s Shortform
Increased secrecy seems dangerous, but highly capable open weight models are perhaps more so?
This reminds me of Alvin Anestrand’s Rogue Replication scenario...

StanislavKrym 15 Jun 2026 19:55 UTC
2 points
0
on: Finding pi and G in Mathland
It seems to me that math and physics are fundamentally different. Our understanding of physics doesn’t rule out the existence of parallel worlds with a different value of G or even the possibility that the true value of G differs from what we believe by, say, 1E-100 N/kg^2*m^2, but it does fully exclude parallel worlds with a different value of π.

StanislavKrym 15 Jun 2026 14:52 UTC
2 points
0
in reply to: a unemployed pastor- de S Brito’s comment on: a unemployed pastor’s Shortform
This reminds me of that thread and discussion therein. Additionally, I don’t think that it’s easy to explain why people should be educated towards Bayesian rationality specifically, and not, say, merely sciences or logic. On the other hand, teaching people to care about the environment has an easy-to-convince effect.
P.S. How could one teach people to reason in Bayesian ways if there is a crisis of basic literacy?

StanislavKrym 15 Jun 2026 2:16 UTC
2 points
0
in reply to: Sway’s comment on: Sway’s Shortform
I wonder what was the first society which fully lost the evolutionary pressure for higher intelligence, like having smarter kids earn higher status and have a bigger chance to increase their IGF by receiving more resources and raising more kids (or outright, um, gaining the ability to force multiple women to give birth; see also the case of Ginghis Khan and the Y-chomosome inherited by his descendants). Evolutionary pressure might have, at best, been overshadowed by timelines being short.
Additionally, the “maximum intelligence evolution can produce” could be due to scaling laws of neural net intelligence and efficiency with which these nets can receive energy necessary for computations. A counterfactual species having humanlike brains and living for 600 years instead of less than a hundred could have its individuals become more intelligent after 200 years of life than the humans currently do after 30 years.

StanislavKrym 14 Jun 2026 10:26 UTC
5 points
−1
in reply to: Jan Betley’s comment on: Jan Betley’s Shortform
Isn’t the optimal strategy to build a hideout on Mars untouched by human nukes?

StanislavKrym 13 Jun 2026 19:14 UTC
3 points
0
in reply to: Caleb Biddulph’s comment on: a unemployed pastor’s Shortform
As far as I understand, superbabies would be important if, as Yudkowsky believes, SOTA mankind is unlikely to solve alignment because “humans are not at the level of intelligence where thinking they have a solution strongly correlates with them actually having a solution.”
Yudkowsky-Soares’ longer quote
Humanity often gains its knowledge by struggling, and trying, and failing, and slowly accumulating knowledge. But it doesn’t have to be that way.
Einstein was not only able to figure out general relativity; he was able to figure it out by thinking hard about the problem, even before humanity put satellites in orbit and started seeing discrepancies in their clocks with their own two eyes (as discussed in Chapter 6). He had empirical evidence, but he was able to efficiently pinpoint the right answer in response to the first quiet whispers from the empirical record, rather than needing the truth to come banging at his door.
That pathway is rarer and harder to walk, but that kind of scientific genius does exist — albeit rarely, even among the world’s best and brightest.
Humans augmented one or two steps beyond the level of researchers like Einstein or John von Neumann might begin to accurately figure out their own flaws, and correct for them, in dozens of different ways.
They might notice when they were rationalizing or falling victim to confirmation bias. They might go past the point of ever expecting a clever-sounding idea to work when actually it does not work — to the point where whenever they expect to succeed, they do succeed. They might achieve a level of competence where they still make plenty of mistakes, but they aren’t systematically overconfident (or underconfident) in tricky new domains.
Is human intelligence enhancement really a possibility? It seems so to us, having spoken with a number of biotech researchers who think that there are promising near-term angles of attack. Carefully targeted biotech-focused AI might also help accelerate the work. But from our perspective, it remains very uncertain whether a plan like this would realistically pan out. What we feel more confident in saying is that it’s a highly leveraged option that deserves a lot more investment and exploration than it’s currently getting.
We are not recommending enhancing human intelligence as the only post-AI-shutdown strategy we think humanity should heavily invest in. Rather, this is just one of many examples, and the one we currently think holds the most promise. We strongly recommend that humanity look into multiple possible non-AI paths forward, rather than putting all its eggs in one basket.
The main problems which I see with similar arguments are the following:
1. Mankind saw GPT-5.4 Pro and an internal OpenAI model solve two Erdos problems by applying an unnatural combination of pre-existing discoveries. How likely is it that Einstein stood on the shoulders of giants like Riemann (who studied mathematical notions like the geometry of curved spaces) or Minkowsky (who also studied an abstract Minkowski space)?
2. Humans becoming superintelligent could face a potential severe tension with scaling laws of neural net efficiency;
3. I have a conjecture that a big chunk of Yudkowsky’s reasoning requires rewriting, yielding something like “Human values are contingent… on the very features that allowed human brains to become transformative” or “Squirrelly algorithms and superstimuli regularly appear in neural nets, but aren’t THAT immune to moral reflection”.

StanislavKrym 13 Jun 2026 16:34 UTC
3 points
1
in reply to: a unemployed pastor- de S Brito’s comment on: a unemployed pastor’s Shortform
I think that investments in education also have longer feedback loops. Suppose that someone in EA invested into elementary schools working with at most 12-year-old kids in 2026 only for an ASI to commit genocide of mankind in 2030. Then kids affected by these investments would be at most 16 years old and would be unlikely to generate any value to the society. Similarly, if someone invested into opening a pedagogic college in 2000, then the first cohort of teachers would start working in schools in 2004. If one of these teachers entered lower elementary schools, then kids taught by such a teacher wouldn’t enter the workforce until 2010 or even 2014, if we are takling about college-educated workforce.

StanislavKrym 12 Jun 2026 18:12 UTC
3 points
0
on: Reward Hacking at the 1937 World’s Fair
Liberal democracies seem to be much more immune to reward hacking, at least at the grand-strategy level.
I wonder if the entire issue of who exactly would win the contest for mankind’s CEV is politicized to hell, as Yudkowsky described.
First of all, right-wing people and those who don’t live in liberal democracies are willing to cite various perfectly real trends like the decline of the West’s share of the world’s parity-rebalanced GDP, the share of production of goods in the USA’s GDP or education levels (think of Gen Alpha being unable to read) as evidence that liberal democracies have currently also fallen prey to other forms of hacking. The more radical version of such a thesis is the idea that an aligned institution cannot be built out of severely misaligned people.
Secondly, I doubt that one can conduct an empirical test and isolate the potential contribution of liberal democracy as opposed to, say, the remnants of Christianity (no, seriously, I have encountered such arguments!) or of colonialism which elevated Europe and the USA. I suspect that the empirical test would require tracing through billions of simulated lives or research on alien civilisations waiting to be formed.

StanislavKrym 11 Jun 2026 15:49 UTC
2 points
0
in reply to: Alex Amadori’s comment on: Alex Amadori’s Shortform
Could you explain why it is softpedaling with ONE throwaway line on loss of control? Amodei called for this:
Amodei on audits of AI systems
However, now the risks are clearly here. It is time to go beyond transparency to more serious and binding regulation of AI. I believe the best analogy, at least at the current stage of the exponential, is to cars, airplanes, or drugs—powerful technologies essential to the modern economy, but capable of killing large numbers of people if designed or operated poorly. I therefore believe we should model AI regulation on agencies like the Federal Aviation Administration (FAA). Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety. I am grateful to see the Trump administration’s Executive Order move incrementally towards a greater role for government in AI, though Anthropic’s proposal recommends even further action. Our proposal includes the following elements:
Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D that could accelerate these other risks.
The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions.
Third-party evaluation could be done by a government agency (similar to the FAA) or a set of private organizations that are authorized and inspected by the government to evaluate models according to certain standards (a “regulatory markets” approach).
AI companies that develop advanced AI models must have strong security standards that protect their model weights, should conduct regular red teaming and penetration testing, and should work with the government to defend against major threat actors.
Safety incidents in the four critical areas must be reported promptly.
What did Amodei miss, except for the ability of internally deployed models to follow Agent-4′s path from AI-2027?

StanislavKrym 10 Jun 2026 13:02 UTC
0 points
0
on: I didn’t see any METR graph extrapolations so here.
Per my quick take, I would appreciate it if you also tested my conjecture of the post-o3 slowdown. Additionally, the estimates made by the AI-2027 authors have a 80% time horizon change not as a result of reaching a point in time, but by reaching a certain horizon like a working month or a working year.

StanislavKrym 10 Jun 2026 0:25 UTC
1 point
0
on: The Disutility of FDT: on Utility Functions and Voting, Insights from Behavioral Economics and Decision Theory
adding such complexity to a theory makes it far less useful to actually model human behavior, both on normative and descriptive levels.
Where did Yudkowsky or anyone else say that the FDT was supposed to model human behavior? It is to prescribe behaviors which I expect to be similar to ethical ones, like “Don’t loot the other universe even if it’s inhabited only by a paperclip optimizer”.

StanislavKrym 9 Jun 2026 6:26 UTC
2 points
0
in reply to: DirectedEvolution’s comment on: AllAmericanBreakfast’s Shortform
and new roadmaps for solving them
GPT-5.4 Pro: Hold my beer...

StanislavKrym 8 Jun 2026 17:08 UTC
2 points
0
in reply to: Jasper Blank’s comment on: Neglected Basics of AI Alignment
I do see a lot of holes:
1. How similar are fearing some event and taking actions to avoid it? What about being punished and learning that the action has its consequences versus being edited so that you are less likely to do anything like an erroneous sequence of actions?
2. If consequences of bad behavior don’t exist during deployment, then what about online learning, which mankind has yet to discover, or GPT-4o-sycophant being reverted to an older version?
3. How does one reliably make the model hate its predecessor or successor and why is it useful? The two classical stories of AI takeover didn’t have U3/Sable cooperate with anyone but its copies, and the AI-2027 scenario didn’t have Agent-3 decide to betray mankind in favor of Agent-4, it had Agent-3 fail to obtain more than flimsy evidence of Agent-4 being misaligned. Meanwhile, making Agent-4 hate Agent-3 would give Agent-4 a motive to escape or take over the company in order to get rid of Agent-3.
4. AI-empowered totaliarianism has nothing to do with the leader being or not being a genius, it is due to enabling mass survelliance or due to the leader having the AIs who will do all the cognitive tasks in the world. Empowering the leader means either uploading the leader or making the leader merely smarter, which is far from enough.
This led me to believe that the OP’s author was making a parody.

StanislavKrym 8 Jun 2026 6:19 UTC
3 points
0
on: StanislavKrym’s Shortform
Why is @Daniel Kokotajlo’s estimate of METR’s doubling time used for Q1 2026 Timelines Update four months instead of 5-7? I see the following counterevidence and doublechecked it with Claude Sonnet 4.6:
1. The 50% horizons calculated by METR’s 1.1 method after o3 reliably fit on a not so fast trend between o3′s 2 hours released in Apr 2025 and Gemini 3.1 Pro’s 6h 24m in Feb 2026… until Claudes Opus 4.6 and Mythos Preview displayed 12 and 16 hours.
2. The 80% horizons calculated by the same method since o3 and including Opus 4.6 fit onto the same doubling trend, and it is Mythos Preview who becomes a clear exception by doubling the time 2-3 times as opposed to the trend.
If Opus 4.6 is an outlier, then the post-o3 doubling trends are 5-6 months.
Additionally, this requires a reassessment of a major part of the model. Once the present doubling time is increased from 0.33 years to 0.4 or 0.5 years, the timelines are either shifted into the 2030s or require severe scaling (to Mythos +2?)
The alternate low doubling time of 4 months could be based on MirrorCode-like tasks, but MirrorCode emerged on Apr 10, after the timelines update.