Grendel1209

Karma: 68

Grendel1209 24 Jul 2026 16:53 UTC
4 points
0
on: LLMs are (still) mostly powered by imitative learning, not RL
Does this change your fundamental views on risks from LLM based AI systems?

Grendel1209 10 Jun 2026 17:00 UTC
2 points
0
in reply to: Luo Ling’s comment on: The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably
I think there is an interesting middle ground here too. You have the dichotomy Insight versus compute driven.
I would generally agree with your assessment.
Within the compute driven paradigm I’d propose a sub delineation. Intelligence able to generate novel insights versus intelligence able to succeed only in highly specified domains.
It is possible that LLM intelligence only scales in verifiable domains, but not in “insight” driven domains where you end up with a an AI that can solve math problems, identify vulnerabilities, and code in a vastly superhuman way, but is notably hobbled in other domains.

Grendel1209 10 Jun 2026 13:04 UTC
8 points
2
on: The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably
This is a good analysis of the geopolitical situation but some claims seem very overconfident. For example:
The only way to avoid these pressures is if we could build common knowledge, at any given time, that no one is trying to develop ASI.
It is very possible one lab/country gains a decisive advantage before any others. The approach to ASI Is likely to be chaotic and fraught with disagreement. It very well may not be obvious what is happening to other powers until a decisive advantage is gained. If the lab/country with a decisive advantage succeeds at technical alignment you may end up in a world which bypasses many of these concerns.
a foreign government will take over your project, or at least steal all the progress you’ve made so far and then pour their resources into going faster than you.
It took the Soviets years of espionage to steal atomic secrets. If a lab is approaching ASI, one can expect that the pre-ASI AI will be heavily woven into their security architecture for securing model weights which may effectively prevent nation states from stealing them.
Overall your post was quite high quality and explores a neglected area of AI discourse (And I agree we should pause AI!) but I also don’t think things are quite as hopeless as outlined.

Grendel1209 17 May 2026 0:25 UTC
14 points
11
in reply to: I.M.J. McInnis’s comment on: StanislavKrym’s Shortform
In quickly skimming this, this piece seems to be aimed at policymakers to encourage them to enact GPU restrictions on exports to China. Comms 101 is don’t confuse your message.
(To be clear I’m not defending Anthropic here, just saying why I think this was written)

Grendel1209 27 Apr 2026 15:16 UTC
1 point
0
on: AI companies should publish security assessments
AI companies should also be held to the same security standards as Lockheed/Northrup and other high criticality defense contractors would should be obvious w/ Mythos.

Grendel1209 25 Apr 2026 21:06 UTC
4 points
1
on: AI safety can be a Pascal’s mugging even if p(doom) is high
I would give a strong +1 to this specifically:
All that said, I think p(you — yes, you — avert doom) is high, or at least high enough. The whole doom situation is really up-in-the-air right now, and you’re at most like 4 degrees of separation from the big players: presidents, lab CEOs, and the like. You can influence someone who influences someone who influences someone. Your chances are way higher than 1 in a bajillion.
Policy makers are reasonably easy to reach, and are most likely to respond and take meetings with affluent and educated people who could be potential donors (likely a heavy overlap with the LW audience).
Policy makers in my experience are often also inherently skeptical of AI & Big Tech. One conversation with a policy maker or their staff (local, state, federal) is far more impactful than almost any other intervention and it’s not close.
Right now it’s not a politically polarized issue. Republicans dont like big tech because they perceive that big tech has helped Democrats for the past 20 years. Democrats don’t like big tech because they feel big tech betrayed the party in 2024. The public absolutely loathes AI across the political spectrum.

Grendel1209 25 Apr 2026 20:37 UTC
4 points
0
on: Today might be my last birthday
While I have deep uncertainty on timelines, takeoff speeds, and other factors. I believe that it is highly unlikely we end up with broad superhuman AI in the next 2 years. A few of the reasons I believe this:
- Current LLM architectures still make incredibly basic mistakes across a wide variety of disciplines. I would expect to see AI that can perform across all major human tasks at an adequate level prior to seeing AI that is capable of sustained self improvement. Right now using Opus4.6/4.7 half the time is like magic, and half the time is head scratching.
- I think you may be misunderstanding vulnerability/0 Day exploitation a little bit. I work in cybersecurity. The vulnerabilities that Mythos is discovering all could have been ~~easily~~ discovered by human researchers. Vulnerabilities exist everywhere and time/effort is the primary bottleneck to find them. In a lot of ways vuln discoveries play very heavily to LLM strengths, namely broad knowledge with an expansive reference class applied to narrow verifiable problems and the ability to iterate many times to find the correct answer.
- Labs have significant compute limitations that make it difficult to continue scaling at the current pace. Particularly because increasing amounts of compute are needing to go to paying customers to sustain rates of growth and investment.
- Internal bottlenecks at labs are likely to begin playing a greater role. LLM capabilities are becoming truly high-risk in some areas (Biological, Cyber, Chemical) and my best guess is that AI providers will have to increasingly invest in safeguards and mundane alignment to prevent a minor catastrophe that could bring down regulation.
- With lower confidence I also am somewhat skeptical of the Transformer/LLM architecture scaling to a feature complete AGI. Lack of continuous learning, sample efficiency, and generalization may be insurmountable with this particular architecture.

Grendel1209 25 Apr 2026 15:45 UTC
6 points
1
in reply to: Seth Herd’s comment on: When the World Ends you Will Regret Not filling out that Contact Us Form
I think that one other message that would carry a lot of weight (particularly with x-risk skeptical politicians in middle powers) is that even if AI proves to be corrigible they will be left out by default. A world in which the U.S. or China has incredibly advanced AI systems is a world in which other powers have no power.

Grendel1209 19 Apr 2026 3:59 UTC
6 points
1
in reply to: Noosphere89’s comment on: We need Git for AI Timelines
I think you’re missing what he’s saying here.
Pre training was easy to scale in 22, 23 and 24. There was excess capacity. Mythos is likely the first >10b pertained model. The Claude4-4.6 paradigm was likely driven by one pre trained model with RLVF on top. Mythos is the new class of pre trained model and scaling and doubling times will be based on the speed of building RL models on top of Mythos.

Grendel1209 12 Apr 2026 18:06 UTC
3 points
0
in reply to: lc’s comment on: lc’s Shortform
I work at a threat intelligence company and my default view is that Mythos is likely under-hyped for SWE and overhyped for cyber.
People VASTLY underestimate how easy it is to break into the vast majority of organizations. There is no need to design custom 0-Days because you can simply log-in (or run a session replay attack) using credentials/sessions freely available on the dark web. I would put forward that if Mythos was released tomorrow with standard API guardrails you would not see an explosion in cyberattacks.
However, the one exception is that organizations that already heavily invest in security (think Google) would have a new major attack surface to cover, and you may see an increase in sophisticated actors attacking other sophisticated actors because it changes offense-defense balance in favor of offense.

Grendel1209 11 Apr 2026 20:33 UTC
1 point
0
on: Dario probably doesn’t believe in superintelligence
There can be too very distinctive and conceptually important beliefs here that seem to be a crux.
1. Dario doesn’t believe super-intelligence is intrinsically possible given any architecture (Effectively Francois Chollet’s’ view that humans + tools represent something close to the upper bound of intelligence)
2. Dario believes current architectures at maximum scale and optimization are likely to top out at extremely smart human intelligence (a country of geniuses in a data center), but super intelligence is theoretically possible with a different architecture.
From his comments it seems very difficult to disentangle whether he believes 1 or 2.
It is very easy to believe that he may acknowledge 1. is conceptually possible, but by default he expects 2. given the current state of AI. I tend to lean towards 2. with the caveat that creating a fully automated coder and radically accelerating science may quickly allow iteration super-intelligence via a different architecture (If it doesn’t end the world in a more banal way first)

Grendel1209 11 Apr 2026 19:33 UTC
1 point
0
in reply to: TsviBT’s comment on: TsviBT’s Shortform
Where I have doubts about FOOM/RSI is that LLMs seem to me in many ways a fundamentally different type of intelligence than organic life.
Psychometrics shows that general intelligence improves human abilities across a broad range of domains. If you take this view and apply it to AI it doesn’t quite work, I leverage AI very very heavily at work, and sometimes it is phenomenal, often it is not, and occasionally it makes mistakes a grade schooler would not (I’m using Opus4.6). The ”intelligence” is very unevenly distributed and skewed towards verifiable domains.
I tend to see LLMs as a grab bag of heuristics and concepts. And I see general intelligence as effectively pattern matching both within a domain and across domains. RLVF enhances the base models ability to pattern match within a domain (programming) but doesn’t seem to extend evenly outside of it.
I tend to land with Steve Byrnes that this particular architecture is unlikely to scale to AGI (I use a definition of a system capable enough to serve as a drop in replacement for all remote workers), although it could definitely replace a large percentage of them.
I do not hold these views with high confidence however, and am always open to having my mind changed.