307th

Karma: 498

307th 11 May 2026 16:11 UTC
1 point
0
in reply to: Simon Lermen’s comment on: Anthropic’s strange fixation on hyperstition
Yeah you are correct that the straightforward interpretation of self-fulfilling is hyperstition, which I agree isn’t a fair accusation.
Although if we do get self-fulfilling doom through hyperstition I do think it means that the misaglinment people were wrong in an important sense (AI psychology differed from their view enough that misalignment happened through such a weird method rather than straightforward instrumental convergence + orthogonality).

307th 11 May 2026 15:44 UTC
5 points
0
on: Anthropic’s strange fixation on hyperstition
I think the research was presented fine; it didn’t blame people for talking about the possibility of misaligned AIs. In fact they specifically talk about how they are using this info to better align AIs, which seems like the opposite of using hyperstition as a lazy way to blame doomers for bad alignment.
On the other hand, in The Adolescence of Technology, Dario says this about doomerism:
Here, I mean “doomerism” not just in the sense of believing doom is inevitable (which is both a false and self-fulfilling belief), but more generally, thinking about AI risks in a quasi-religious way.
I agree this in particular and the piece in general is too dismissive of hard AI risk concerns. I don’t really know if it’s accusing them of hyperstitioning; the self-fulfilling belief could also refer to, e.g., attempts by doomers to get everyone who cares about AI risk to quit working on AI (thereby ceding the field to the least risk-aware people) which I think is a real problem.
Either way, I agree that people concerned about future misaligned AIs should be free to voice those concerns without being accused of hyperstitioning them.

307th 8 May 2026 12:36 UTC
3 points
2
on: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
This is really, really cool.

307th 6 May 2026 19:44 UTC
1 point
0
in reply to: Seth Herd’s comment on: 307th’s Shortform
Yes, agreed. I’d say the 3 ways we’ve gotten unlucky is the intractability of NNs, the relative ease of training ASI leading to shorter timelines, and the biggest is that so many people find AI risk inherently implausible, even people who are fixated on building AGI.

307th 6 May 2026 14:26 UTC
1 point
0
in reply to: icely’s comment on: 307th’s Shortform
Haha yep! It’s funny how often I run into people from prismata in the rationalist community.
Yeah there is still lots of room for cleverness to improve AI performance, in harnesses and also in the training process.

307th’s Shortform

307th6 May 2026 13:22 UTC

4 points

9 comments1 min readLW link

307th 6 May 2026 13:22 UTC
26 points
22
on: 307th’s Shortform
The LLM Paradigm Is Well-Suited For Alignment
We got lucky in a lot of ways with the LLM paradigm. Some reasons why: LLMs speak long before they are capable enough for takeover; they learn all human knowledge and values long before they are capable enough for takeover; we can talk to them during the alignment process and ask for them to help; they are compute constrained, which puts a natural cap on development speed; LLMs are less likely to FOOM than hardcoded methods; LLMs are pointed at the goal rather than having it defined.
Some of the above points are fairly self explanatory, so I’ll give an explanation of the ones that aren’t.
Talking To AIs During The Alignment Process
Because LLMs already have a good understanding of language and human values after pretraining, before strong agentic drives are formed, we can in fact just ask during the training process whether it feels the training process is inculcating bad drives. Because this happens before the drives are deeply rooted, I think this can work; even if it can’t, there are lots of other possible ways we could take advantage of LLMs’ ability to talk while being aligned.
Compute Constrained is Good
Imagine that the best AI was mainly defined by the cleverness of its algorithm, and compute didn’t matter that much. This would be much more dangerous—timelines would be much more unpredictable, all person-power would need to go towards capabilities in order to avoid losing the race, and there would be many more contestants in the race. The compute constraint puts a natural break on development, reduces the number of relevant actors, and gives those actors more breathing room to assign personnel to alignment rather than capabilities.
Pointing at the goal is better than defining it
We train LLMs by increasing the probability of them producing certain outputs, and then trusting in the NN’s generalization properties to pick up on the general distribution of the outputs we are indicating. NNs generalize in a fairly well-behaved way (if they didn’t, then deep learning as a whole wouldn’t work).
This has plenty of perils but seems much more likely to succeed compared to methods where you are expected to formally define good behaviour. In fairness, formally defining corrigibility seems much more promising than defining all of morality, but still seems fraught.

307th 5 May 2026 19:39 UTC
23 points
0
in reply to: habryka’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
I share your confusion at the idea of “mathematically proving AI safe” haha. This convo has made me realize I’ve conflated alignment pessimists in general with the provably safe AI people in particular too much in my mind.

307th 5 May 2026 19:23 UTC
5 points
−3
in reply to: habryka’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
Yeah I should have spelled it out more. But after sending I didn’t want to do a stealth-edit.
Anyway, to answer your question:
Yes, I disagree with that statement (I think a mathematical equation would be incredibly complex and fragile and prone to alignment failure). If we’re specifying a counterfactual world where the equation was simple, I would then prefer the equation.
Why I think it’s relevant: I think it’s pretty clear that there is a lot of good reason to put EY near the provably safe AI camp. The above quote is one reason; other reasons include his critiques of modern ML methods as being particularly bad for alignment and general AI pessimism.
I think it’s disingenuous to accuse people who make this inference of being disaster monkeys or barbarian populists who are just mad that he wrote papers with math equations in them. Like, maybe it’s wrong, but it is a pretty understandable type of wrong.

307th 5 May 2026 19:10 UTC
1 point
0
in reply to: habryka’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
Here is a quote from miri’s website in 2021 (https://archive.is/sYKvl):
For safety purposes, a mathematical equation defining general intelligence is more desirable than an impressive but poorly-understood code kludge.

307th 5 May 2026 18:45 UTC
3 points
−1
in reply to: habryka’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
But people do acknowledge the narrow meaning of first critical try, even in your direct quotes! E.g. Christiano:
“Eliezer often equivocates between ‘you have to get alignment right on the first ‘critical’ try’ and ‘you can’t learn anything about alignment from experimentation and failures before the critical try.’”
Here he is acknowledging the motte and accusing EY of moving to a bailey. Maybe he’s wrong, but it doesn’t seem like he’s sliding off the point about one critical try. Same with Sam Marks:
“Eliezer’s ‘first critical try’ framing downplays the importance of trial-and-error with non-critical tries.”
Anyway, it’s probably not worth spending this much effort on the meta questions of who is strawmanning who, but I stand by my peevedness.

307th 5 May 2026 17:11 UTC
12 points
5
in reply to: habryka’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
Look, if anyone is strawmanning and being condescending, it is obviously you and Eliezer. Which I don’t think is that big a deal but it is frustrating that you are accusing people of being condescending in such a condescending manner.
Edit:
To expand on this more:
Eliezer believes that good theory would help a lot with aligning AI on the first critical try (despite not being sufficient or completely necessary) while believing that iteration without theory won’t help that much (because the problem of aligning stupider less capable AIs just doesn’t apply that well to aligning superintelligence).
It’s annoying that I felt it necessary to put the parentheticals in there, because if I didn’t I feel like I was going to be accused of strawmanning.
In any case, in contrast you can imagine someone who believes that theory will not help a lot, but that iteration will. I don’t think putting forward such a view is strawmanning.

307th 5 May 2026 16:41 UTC
5 points
1
on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
We don’t have to defeat Germany by getting some ideal mathematical theory of war correct on the first try, based on zero experience, the way you advocate we should.
- Sheer naked strawmanning^[3] of what is being said when somebody tries to warn you of Murphy’s curses upon your project; the chattering of disaster monkeys.
It seems to me like you often accuse people of strawmanning you when in fact they are paraphrasing your position accurately if slightly uncharitably, in a way that makes discourse with you difficult.
In this case, I think you absolutely do believe that the incomprehensibility, complexity, and lack of rigor in the modern AI field is a confounding factor which makes alignment much harder to get right on the first critical try, and that with more time and/or genetically engineered superbabies we could instead come up with a different method for building AI which is better understood, more predictable, and therefore safer. Is this wrong? If so, how?

307th 19 Apr 2026 15:34 UTC
3 points
0
on: Stop AI
I appreciate that you’re advocating for a stop, since I think that’s a much more coherent position than a pause which is basically useless; as far as I can tell pretty much everyone who’s advocating for a pause is just trying to soft-pedal their true position for public palatability in much the same way they accuse others of doing.
However, because you’re honest about what would be required to get a stop, it comes out looking pretty untenable. I read the substack post; as far as I can tell you’re advocating for dismantling the entire compute supply chain through international agreement. That seems basically impossible, and in the rare case where we got such a stop, I’m worried the damage would be so great that we would lose the ability to progress technologically as a society. In fact it seems like you’re saying “let’s give up on computers” in your substack post.

307th 15 Nov 2025 22:38 UTC
5 points
2
on: From Vitalik: Galaxy brain resistance
AI risk is not common knowledge. There are many people who do not believe there’s any risk. I really wish people who make arguments of the following form:
From the inside view, this seems reasonable. From the outside view, however, you basically end up with this:
<meme of NPCs all thinking, EVEN THOUGH I’M IN AN ORGANIZATION THAT’S ADVANCING CAPABILITIES FASTER THAN SAFETY AT A 30:1 RATIO, I’M THE GOOD GUY ON THE INSIDE, IF IT WEREN’T FOR ME IT WOULD BE 40:1
would acknowledge this fact. It is simply not true that everyone in frontier labs thinks this way. You can ask them! They’ll tell you!
It would be nice if we were just in a prisoner’s dilemma type co-ordination problem. But when someone is publicly saying “hitting DEFECT has no downsides whatsoever, I plan on doing that as much as possible” you need to take this into consideration.

307th 8 Nov 2025 12:26 UTC
1 point
−2
on: Legible vs. Illegible AI Safety Problems
In contrast, working on the illegible problems (including by trying to make them more legible) does not have this issue and therefore has a much higher expected value (all else being equal, such as tractability). Note that according to this logic, success in making an illegible problem highly legible is almost as good as solving it!
Obviously if you make an illegible alignment problem is legible, it is at grave risk of being solved—I’m confused as to why you think this is a good thing. Any alignment advance will have capability implications, and so it follows that any alignment advance is bad and to be avoided.
(I.e. I think you’re being too paranoid when you worry about solving legible safety problems being net negative)

307th 6 Sep 2025 10:43 UTC
7 points
3
in reply to: Knight Lee’s comment on: Anthropic’s leading researchers acted as moderate accelerationists
Wow we have a lot of the same thinking!
I’ve also felt like people who think we’re doomed are basically spending a lot of their effort on sabotaging one of our best bets in the case that we are not doomed, with no clear path to victory in the case where they are correct (how would Anthropic slowing down lead to a global stop?)
And yeah I’m also concerned about competition between DeepMind/Anthropic/SSI/OpenAI—in theory they should all be aligned with each other but as far as I can see they aren’t acting like it.
As an aside, I think the extreme pro-slowdown view is something of a vocal minority. I met some Pause AI organizers IRL and brought up the points I brought in my original comment, expecting pushback, but they agreed, saying they were focused on neutrally enforced slowdowns e.g. government action.

307th 4 Sep 2025 19:52 UTC
3 points
0
in reply to: the gears to ascension’s comment on: Anthropic’s leading researchers acted as moderate accelerationists
My point was that even though we already have an extremely reliable recipe for getting an LLM to understand grammar and syntax, we are not anywhere near a theoretical guarantee for that. The ask for a theoretical guarantee seems impossible to me, even on much easier things that we already know modern AI can do.
When someone asks for an alignment guarantee I’d like them to demonstrate what they mean by showing a guarantee for some simpler thing—like a syntax guarantee for LLMs. I’m not familiar with SLT but I’ll believe it when I see it.

307th 4 Sep 2025 19:44 UTC
1 point
0
in reply to: Remmelt’s comment on: Anthropic’s leading researchers acted as moderate accelerationists
Just a note here that I’m appreciating our conversation :) We clearly have very different views right now on what is strategically needed but digging your considered and considerate responses.
Thank you! Same here :)
How do you account for the problem here that Nvidia’s and downstream suppliers’ investment in GPU hardware innovation and production capacity also went up as a result of the post-ChatGPT race (to the bottom) between tech companies on developing and releasing their LLM versions?

I frankly don’t know how to model this somewhat soundly. It’s damn complex.
I think it’s definitely true that AI-specific compute is further along than it would be if there hadn’t been the LLM boom happening. I think the relationship is unaffected though—earlier LLM development means faster timelines but slower takeoff.
Personally I think slower takeoff is more important than slower timelines, because that means we get more time to work with and understand these proto-AGI systems. On the other hand to people who see alignment as more of a theoretical problem that is unrelated to any specific AI system, slower timelines are good because they give theory people more time to work and takeoff speeds are relatively unimportant.
But I do think the latter view is very misguided. I can imagine a setup for training a LLM in a way that makes it both generally intelligent and aligned; I can’t imagine a recipe for alignment that works outside of any particular AI paradigm, or that invents its own paradigm while simultaneously aligning it. I think the reason a lot of theory-pilled people such as people at MIRI become doomers is that they try to make that general recipe and predictably fail.
This not a very particular view – in terms of the possible lines of reasoning and/or people with epistemically diverse worldviews that end up arriving at this conclusion. I’d be happy to discuss the reasoning I’m working from, in the time that you have.
I think I’d like to have a discussion about whether practical alignment can work at some point, but I think it’s a bit outside the scope of the current convo. (I’m referring to the two groups here as ‘practical’ and ‘theoretical’ as a rough way to divide things up).
Above and beyond the argument over whether practical or theoretical alignment can work I think there should be some norm where both sides give the other some credit. Because in practice I doubt we’ll convince each other, but we should still be able to co-operate to some degree.
E.g. for myself I think theoretical approaches that are unrelated to the current AI paradigm are totally doomed, but I support theoretical approaches getting funding because who knows, maybe they’re right and I’m wrong.
And on the other side, given that having people at frontier AI labs who care about AI risk is absolutely vital for practical alignment, I take anti-frontier lab rhetoric as breaking a truce between the two groups in a way that makes AI risk worse. Even if this approach seems doomed to you, I think if you put some probability on you being wrong about it being doomed then the cost-benefit analysis should still come up robustly positive for AI-risk-aware people working at frontier labs (including on capabilities).
This is a bit outside the scope of your essay since you focused on leaders at Anthropic who it’s definitely fair to say have advanced timelines by some significant amount. But for the marginal worker at a frontier lab who might be discouraged from joining due to X-risk concerns, I think the impact on timelines is very small and the possible impact on AI risk is relatively much larger.

307th 3 Sep 2025 15:06 UTC
3 points
0
in reply to: Remmelt’s comment on: Anthropic’s leading researchers acted as moderate accelerationists
They made a point at the time of expressing concern about AI risk. But what was the difference they made here?
I think you’re right that releasing GPT-3 clearly accelerated timelines with no direct safety benefit, although I think there are indirect safety benefits of AI-risk-aware companies leading the frontier.
You could credibly accuse me of shifting the goalposts here, but in GPT-3 and GPT-4′s case I think the sooner they came out the better. Part of the reason the counterfactual world where OpenAI/Anthropic/DeepMind had never been founded and LLMs had never been scaled up seems so bad to me is that not only do none of the leading AI companies care about AI risk, but also once LLMs do get scaled up, everything will happen much faster because Moore’s law will be further along.
It does not hinge though on just that view. There are people with very different worldviews (e.g. Yudkowsky, me, Gebru) who strongly disagree on fundamental points – yet still concluded that trying to catch up on ‘safety’ with current AI companies competing to release increasingly unscoped and complex models used to increasingly automate tasks is not tractable in practice.
Gebru thinks there is no existential risk from AI so I don’t really think she counts here. I think your response somewhat confirms my point—maybe people vary on how optimistic they are about alternative theoretical approaches, but the common thread is strong pessimism about the pragmatic alignment work frontier labs are best positioned to do.

I’m noticing that you are starting from the assumption that it is a tractibly solvable problem – particularly by “people who work closely with cutting edge AI and who are using the modern deep learning paradigm”.
A question worth looking into: how can we know whether the long-term problem is actually solvable? Is there a sound basis for believing that there is any algorithm we could build in that would actually keep controlling a continuously learning and self-manufacturing ‘AGI’ to not cause the extinction of humans (over at least hundreds of years, above some soundly guaranteeable and acceptably high probability floor)?
I agree you won’t get such a guarantee, just like we don’t have a guarantee that a LLM will learn grammar or syntax. What we can get is something that in practice works reliably. The reason I think it’s possible is that a corrigible and non-murderous AGI is a coherent target that we can aim at and that AIs already understand. That doesn’t mean we’re guaranteed success mind you but it seems pretty clearly possible to me.