307th

Karma: 654

307th 15 Sep 2026 11:54 UTC
5 points
−1
in reply to: quetzal_rainbow’s comment on: 307th’s Shortform
This is an example of the uncharitable framing I mentioned in the OP. EY’s claims are so sweeping (effective altruists + Dario Amodei) that it’s hard to know where to start. But to take Anthropic, right now it sucks that Donald Trump hates them. Was this avoidable? I don’t know; other people have agency and a lot of the polarization was simply a result of people not liking AI safety.
But Anthropic has obviously, on net, improved the credibility and salience of AI safety a ton. Really seems to me that people just like using Anthropic and EAs as punching bags and aren’t really thinking things through.
Also: it’s weird that the typical line for hating EAs was that they were too willing to hide their views for political pallatability. Now that Anthropic got in trouble with the current admin for being too principled, and they’re getting hate for that too. Again, really just seems like people have decided to ignore the good that comes from Anthropic and harp on anything bad that happens that seems tangentially related to them.

307th 15 Sep 2026 11:10 UTC
1 point
2
in reply to: Ben Pace’s comment on: 307th’s Shortform
You’re right that I was unaware of that, do you have a link?

But EY makes a ton of sweeping claims with no evidence and I’m a bit frustrated that if you contest them it seems like people will say “well, one of those sweeping claims was actually based on something” and I guess that’s enough for people to feel comfortable giving him a pass.
I think it’s pretty obvious that he’s being absurdly uncharitable here and that if you allow that much license for rewriting history, picking individual issues to criticize on, interpreting and attributing outcomes in an unrealistic way, then you could criticize anyone who had been active for the last 10 years in similar terms, certainly including EY himself.

307th 15 Sep 2026 1:21 UTC
19 points
−11
on: 307th’s Shortform
EY’s lately taken to punching “EAs” in a way that feels pretty ahistorical and unfair. Latest example: https://x.com/allTheYud/status/2099583457093734495
I note, because I think that even now it still matters: On my history, EAs generally and Amodei specifically, indulged in politically polarizing “AI safety” in a leftist direction, because that got them short-term power; stymieing others’ attempts to stay right-left neutral.
...
Now that the cost TO YOU of what got THEM some short-term goodies has become apparent… well, it’s frankly too late to take away what Amodei gained by fucking you over...
He adopts an extremely uncharitable frame, both for their motivations and outcomes. When EAs gain influence, this is evil and selfish; when EAs lose influence, this is also evil and selfish. Somehow they perfectly socialize the losses and privatize the gains; EAs gaining influence is for their benefit, EAs losing influence hurts all of us.
In reality, I don’t think EAs or Anthropic ever went particularly hard on leftism; I don’t think they gained that much from being vaguely left wing; and I would guess Anthropic has bore a fair bit of the brunt of the administration polarizing against it.

307th 8 Sep 2026 11:22 UTC
15 points
3
in reply to: johnswentworth’s comment on: Contra Piper on When Conversation Is Possible
As someone who falls into the latter camp, I would say the reason I’m not a fan of arguing about someone’s true reason for believing X is more pragmatic, which is that when it’s done it’s mostly done to bulverize rather than for any truth seeking reason. Talking about the object level issue is hard and fiddly, but making up a picture of why your opponent is wrong is fun and can be persuasive to bystanders.
As for why I think it’s bad:
- Accusations of bias are easy to dress up rhetorically and often can’t be disproven
- Even in the case where the speculation is accurate, it doesn’t have much bearing on the actual issue; you can successfully identify someone is biased towards believing X for silly emotional reason Y but they probably also have concrete reasons A, B, and C, and X may in fact be true despite the silliness of Y.
- Conversely, even in the case where X is false, convincing them of not-X seems more likely if you address their concrete reasons A, B, and C rather than spending time trying to figure out which bias is leading them to buy A, B, and C and trying to convince them that they are suffering from that bias.
As a result of all this, I just find it uninteresting to argue about people’s motives. It seems highly unproductive and is usually done by people playing games rather than truth seeking. If we’re talking about X, and you say I only believe that because Y despite me saying I believe it because of A, B, and C, I become less interested in the conversation, because now the convo is shifting to be about Y but regardless of whether Y is true or false it doesn’t have much to do with X.
Obviously people can in fact be wrong about their own motives and so there is a case where it can actually be productive, but even among rationalists I feel like I’ve much more often seen the unproductive failure mode described above.

307th 4 Sep 2026 17:23 UTC
1 point
−5
in reply to: habryka’s comment on: PauseAI Has ‘officially disendorsed’ PauseAI-US
I think there’s clearly a pretty huge difference between trying to stop the world from ending in a way that risks ending it, vs. actually doing the holocaust on purpose.
Should we try a surgeon for doing a dangerous invasive surgery that has a 20% chance of killing the patient, even considering the fact that the patient had an 80% chance to die left alone?
I think we should be consequentialist about AI risk. I’m not big on moral condemnation in general because I believe everyone deserves good things, but to the extent we’re doing it, we should try to do it in a way that incentivizes good behaviour, and that includes incentivizing decisions like the surgeon’s.

307th 4 Sep 2026 16:19 UTC
5 points
−4
in reply to: habryka’s comment on: PauseAI Has ‘officially disendorsed’ PauseAI-US
I’m much more interested in the moral dimension rather than the legal dimension here; Nuremberg trials means equating what lab employees are doing to the holocaust, I don’t think it’s equal to that, this is where my anger is coming from.
I also think it’s disingenuous to say “well, I’m just saying there should be a trial, and although I could be tried I’m protected under first amendment” when you are also likening what they’re doing to the holocaust. There’s a missing mood—if they were wrong and made things worse they’re like the Nazis and should be held to account, if you were wrong and made things worse well aw shucks, what can you do, I was just exercising free speech. I’m frustrated by the asymmetry.

307th 4 Sep 2026 14:46 UTC
3 points
−4
in reply to: habryka’s comment on: PauseAI Has ‘officially disendorsed’ PauseAI-US
Suppose we do these trials and they conclude that the current frontier labs lowered danger by being more responsible stewards of the technology than the counterfactuals that would have existed in their place. What do you believe should happen in that case? Should you yourself get put on trial for your efforts against these companies, for putting the world in danger? Obviously not!
Maybe my single biggest frustration with the anti-lab camp is the complete obliviousness to the idea they could be wrong, the idea that there could ever be negative consequences to their actions, the complete security that they feel while making these heavy, heavy accusations against others.
Many things are surprising about the present, and one of them is that frontier labs care about AI risk as much as they do, and take various costly actions in order to try to reduce it.^[1] Every action frontier labs take in mitigating AI risk is taken in spite of the best efforts of people who try to make them pay a huge PR price for any admission of AI risk, who try to deprive them of employees who care about AI risk, and who just generally act as if there’s no way their actions could have consequences. Take 10% of the self doubt and fear of negative consequences that you think lab employees should have, and apply it to yourself please!
In this latest wave of alignment failures by frontier AIs, have you never thought “man, I really wish I hadn’t discouraged so many safety people from working at frontier labs”? Has that thought really never crossed your mind? How!?!?
I get that you can make an argument that on net it’s worth it, I’m just saying, where is the acknowledgement of a cost to your own actions, regardless of whether the tradeoff is worth it?
1. ^
  It’s also surprising how hard they’re racing and how ambitious they’ve been in pursuing capabilities with huge capital expenditure, so I’m not saying that having AGI-aware labs is all upside.

307th 25 Aug 2026 19:35 UTC
13 points
7
in reply to: Richard_Ngo’s comment on: What just happened? Pragmatism and Pessimization
I agree there’s a danger of people mouthing the words “I care about x-risk” and not really caring, having some psychological disconnect where they agree with it in abstract but in practice see AI as a cool toy they can work on, and this is very dangerous if the people building AI end up in this category.
So in that sense, as someone whose main contribution to the discourse is to occasionally cheerlead for people to work at frontier labs and not worry about advancing timelines, I worry I could end up contributing, in a tiny way, to a fatal lack of care.
But my main frustration with the anti-capabilities ideology, that I interpreted you as belonging to, is that they seem to want to take this “I should be careful, if we screw this up it’s over” instinct, this bit of fear in people working at the frontier, and weaponize it to say “you’re terrible, quit immediately, get out of there, if you don’t you’re morally compromised” which is a really bad idea. We need the people building AI to feel the fear; if everyone who feels the fear quits, then...
When I imagine people following the advice to quit, I imagine basically what you said, people on the sidelines throwing pebbles; not because you have to be at a frontier lab to be relevant, but because pretty much everything beneficial that you can do will put you up on stage (i.e. contribute to capabilities) and get some pebbles thrown your way.
I’d say I’m demotivated on stopping AI, and optimistic on aligning it. When I engage with someone on the other side of this their theory of change often seems to be AI never getting built, which seems completely implausible to me, and also seems to neglect how much the AI pause movement owes to the frontier labs being vocal about x-risk.
If I had to guess at the cruxes which drive my disagreement with the anti-capabilities camp they’d be:
1) Alignment success is quite sensitive to how much care is taken by the people who build it.
2) Stopping AI from being built is very unlikely to succeed.
I’m not sure how much they apply to you—I’d guess your view would be something like agreeing with 1 and 2 as stated, but believing that the care taken needs to be far-sighted and free from the various incentives which pull people away from real alignment research, which means people need to halt and catch fire and leave the source of the distracting incentives before we can actually think with care; only after we’ve done that can we think about building it with care.

307th 24 Aug 2026 11:04 UTC
35 points
19
in reply to: 307th’s comment on: What just happened? Pragmatism and Pessimization
In hindsight, I’d describe my reaction as “flinching away from the possibility of no longer taking his claims at face value”. I only viscerally internalized that Sam had been lying about his motivations when I later heard about his plans to raise enormous amounts of money to build new chip fabs. This was shocking to me not just because it directly contradicted the arguments he’d been giving, but because it contradicted them to a greater extent than I’d even been able to consider as a plausible hypothesis.
As an aside I found this part quite moving. And as someone who’s found the compute overhang argument extremely convincing in the past, I should acknowledge that I think recent events have been reasonable evidence against it; it does seem like if the LLM craze and then Agent-LLM craze had been successfully delayed 4 years, things wouldn’t move that much faster due to the extra 4 years of Moore’s Law.

307th 24 Aug 2026 10:48 UTC
48 points
−23
on: What just happened? Pragmatism and Pessimization
All of this psychoanalysis misses the mark with me because it seems like the people advancing capabilities have been straightforwardly beneficial to X-risk, which therefore makes their stated motivation (helping with X-risk) seem very plausible.
There is no getting around the fact that AI will be built by people who build AI. Therefore, the people who build AI should care about X-risk. You seem to identify the messiness of the capability/alignment divide and want to abandon alignment in order to get away from capabilities; success depends upon the people who embrace capabilities in order to embrace alignment.
People could falsely seize on this line of reasoning to justify empowering themselves, but as someone who is not currently seizing on this line of reasoning to empower himself, it also seems plainly correct to me. Sometimes the best thing to do is also dunkable.
I don’t think this is particularly unusual in history either. It seems to me like the people who write history are often power-seeking and the success of our species depends upon these people also doing the right thing; even if these power-seeking people are in some ways annoying or unpleasant or straight-up bad people.
If you want a pause, yet again, the power-seeking AI risk people have done you a tremendous favour by being vocal about X-risk. Is there anything they can’t do, any way they haven’t helped?
Speaking of psychoanalysis, I would be tempted to do it to understand why people are so obsessed with getting more time, even at the expense of being able to use that time effectively; because there doesn’t seem to be any rational underpinning to it. Say that you got what you wanted 15 years ago, and everyone concerned about X-risk abandoned ship on anything that might advance timelines. There’s still Google Brain and Facebook AI Research and NVIDIA and many other labs working in AI; there’s still tons of academic researchers scattered around universities happily tinkering. Progress would still happen, slower and in a less focused way, but it would happen. What would we be doing with the extra time we bought? When things get critical, there would be no helpful quotes from lab CEOs floating around about X-risk; you wouldn’t be able to dunk on the CEOs (“If it’s so dangerous, why are you building it, huh???” 1B likes, best tweet of all time); when you went to senators to convince them about AI risk, you look like a total outsider. If you make any headway the labs actively campaign against you. And what alignment research is supposed to be happening with this extra time? Everyone is scared of doing anything too impactful! You just have to hope that AI alignment turns out to be the rare kind of problem you can one-shot theoretically without touching the details of how the AI is actually thinking or learning, since dwelling too much on those details is capability work.
And say you get your theoretical one shot, all thanks to the X years we bought. Then what? Who cares? Why should labs listen to you? Will you even trust them enough to tell them?

307th 21 Aug 2026 13:07 UTC
1 point
−7
in reply to: cousin_it’s comment on: leogao’s Shortform
The other stuff is fine, I’m making the narrow point that discouraging the subset of researchers who care about AI risk from working in the field is bad.

307th 21 Aug 2026 11:10 UTC
1 point
−7
in reply to: cousin_it’s comment on: leogao’s Shortform
I’m not sure I have a most-preferred future—I guess it would be everyone becoming aware of AI risk and the U.N. with US and China at the helm co-ordinating a joint pause and eventual AGI project. But people who point out this is super dangerous itself due to power centralization are correct, and on top of that I think it’s possible that instead of co-ordination, everyone becoming aware of AI risk leads to racing, sabotage, and other drastic actions.
The current situation where the leading labs are moderately risk aware and the rest of the world doesn’t seem that freaked out, while still being freaked out enough to potentially apply some brakes and regulations, is pretty good I think.
Your preferred future seems both nearly-impossible, not that good if it happens, and if we fall short of total victory on your goal then it comes out strongly negative—if we shame AI researchers and only some people worried about AI risk quit, and most people not worried about it stay, then all we’ve done is slow things down 1% and decreased the awareness of the field by 10%. I agree that more time to adjust and develop things like algorithmic liability would be very helpful though, I just think the price of buying that extra time through researcher-exit is absurdly high.

307th 18 Aug 2026 11:33 UTC
1 point
−7
in reply to: cousin_it’s comment on: leogao’s Shortform
My linked comment gave the example of RLHF, a safety contribution (or at least billed as such) which was quite legible and ended up speeding up the race a lot. The same can be said about the “helpful honest harmless assistant” idea
I think one crux we have is that your view is that we should have put this race off for as long as we could, so that we could solve or at least make significant progress on the alignment problem before we ever get to this point; whereas that has always seemed like a nonstarter to me, we needed to know what AGI systems actually look like and how they’re actually trained in order to make progress on the real alignment problem. Because alignment will be heavily dependent on the particulars of the systems we’re building and real experience, safety will largely be decided by people who are unafraid to roll up their sleeves and do capability work. Anyone who is avoiding contributing to capabilities will, for a sufficiently paranoid definition of “contribute to capabilities”, be a nonfactor.
RLHF is a good example. Like you said, RLHF is both safety and capabilities; the same will be true of future alignment techniques. Trying to avoid capability contributions also means avoiding alignment contributions.

307th 17 Aug 2026 11:20 UTC
7 points
−16
in reply to: cousin_it’s comment on: leogao’s Shortform
I think you’re making a mistake whereby any capabilities contribution is legible, while contribution to safety is illegible, and so you’re encouraging people to make massive sacrifices on safety in order to avoid a little contribution to capabilities.
To try to illustrate why this tradeoff is so bad:
- At the level of the marginal worker quitting, nothing really changes except that the lab becomes less risk-aware. People influence the culture where they work. The average person who replaces them will be a tiny bit less competent and a lot less risk-aware.
- Imagining this scaled up, in a world where large numbers of people took your advice sooner and quit the field, we get all this rogue agent stuff happening ~1 year later with labs that have been filtered hard to not care about safety, and so we get less disclosure, less acknowledgement of the issue, and more papering over the problem, with less possibility of a pause, regulation, etc. because the labs aren’t pushing for it.

307th 11 May 2026 16:11 UTC
1 point
0
in reply to: Simon Lermen’s comment on: Anthropic’s strange fixation on hyperstition
Yeah you are correct that the straightforward interpretation of self-fulfilling is hyperstition, which I agree isn’t a fair accusation.
Although if we do get self-fulfilling doom through hyperstition I do think it means that the misaglinment people were wrong in an important sense (AI psychology differed from their view enough that misalignment happened through such a weird method rather than straightforward instrumental convergence + orthogonality).

307th 11 May 2026 15:44 UTC
5 points
0
on: Anthropic’s strange fixation on hyperstition
I think the research was presented fine; it didn’t blame people for talking about the possibility of misaligned AIs. In fact they specifically talk about how they are using this info to better align AIs, which seems like the opposite of using hyperstition as a lazy way to blame doomers for bad alignment.
On the other hand, in The Adolescence of Technology, Dario says this about doomerism:
Here, I mean “doomerism” not just in the sense of believing doom is inevitable (which is both a false and self-fulfilling belief), but more generally, thinking about AI risks in a quasi-religious way.
I agree this in particular and the piece in general is too dismissive of hard AI risk concerns. I don’t really know if it’s accusing them of hyperstitioning; the self-fulfilling belief could also refer to, e.g., attempts by doomers to get everyone who cares about AI risk to quit working on AI (thereby ceding the field to the least risk-aware people) which I think is a real problem.
Either way, I agree that people concerned about future misaligned AIs should be free to voice those concerns without being accused of hyperstitioning them.

307th 8 May 2026 12:36 UTC
3 points
2
on: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
This is really, really cool.

307th 6 May 2026 19:44 UTC
1 point
0
in reply to: Seth Herd’s comment on: 307th’s Shortform
Yes, agreed. I’d say the 3 ways we’ve gotten unlucky is the intractability of NNs, the relative ease of training ASI leading to shorter timelines, and the biggest is that so many people find AI risk inherently implausible, even people who are fixated on building AGI.

307th 6 May 2026 14:26 UTC
1 point
0
in reply to: icely’s comment on: 307th’s Shortform
Haha yep! It’s funny how often I run into people from prismata in the rationalist community.
Yeah there is still lots of room for cleverness to improve AI performance, in harnesses and also in the training process.

307th’s Shortform

307th6 May 2026 13:22 UTC

4 points

30 comments1 min readLW link