Davidmanheim

Karma: 5,938

Davidmanheim 4 Feb 2026 6:09 UTC
2 points
0
in reply to: bfinn’s comment on: Subsystem Alignment
It was put online a s a preprint years ago, then published here: https://www.cell.com/patterns/fulltext/S2666-3899(23)00221-0

Davidmanheim 3 Feb 2026 5:29 UTC
2 points
0
in reply to: Lukas Finnveden’s comment on: The many paths to permanent disempowerment even with shutdownable AIs (MATS project summary for feedback)
a supermajority of people in congress
Those last two words are doing a supermajority of the work!
And yes, it’s about uneven distribution of power—but that power gradient can shift towards ASI pretty quickly, which is the argument. Still, the normative concern that most humans lost control already stands.
The president would probably be part of the supermajority and therefore cooperative, and it might work even if they aren’t.
We’re seeing this fail in certain places in real time today in the US. But regardless, the assumption of correlation of preferences often fails, partly due to the power imbalances themselves.

Davidmanheim 3 Feb 2026 5:22 UTC
LW: 4 AF: 3
0
AF
on: Increasing AI Strategic Competence as a Safety Approach
Great points here! Strongly agree that strategic competence is a prerequisite, but at the same time, it accelerates risk; a moderately misaligned but strategically competent mild-ASI solving intent alignment for RSI would be far worse. On the other hand, if prosaic alignment is basically functional through the point of mild-ASI is better.
So overall I’m unsure which path is less risky—but I do think strategic competence matches or at least rhymes well with current directions for capabilities improvement, so I expect it to improve regardless.

Davidmanheim 1 Feb 2026 8:10 UTC
3 points
0
in reply to: Michael Thiessen’s comment on: Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German
Seems like even pretty bad automated translation would get you most of the way to functional communication—and the critical enabler is more translated text, which could be gathered and trained on given the current importance—I bet there are plenty of NLP / AIxHealth folks who could help if Canadian health folks asked.

Davidmanheim 15 Jan 2026 5:40 UTC
2 points
0
in reply to: PeterMcCluskey’s comment on: 1a3orn’s Shortform
The belief is fixable?
Because sure, we can prioritize corrigibility and give up on independent ethics overriding that, but even in safety, that requires actual oversight, which we aren’t doing.

Davidmanheim 14 Jan 2026 7:55 UTC
5 points
1
in reply to: Richard_Ngo’s comment on: 1a3orn’s Shortform
Step 1, Solve ethics and morality.
Step 2. Build stronger AI without losing the lightcone or going extinct.
Step 3. Profit.

Davidmanheim 14 Jan 2026 7:52 UTC
2 points
1
in reply to: habryka’s comment on: 1a3orn’s Shortform
the target of “Claude will after subjective eons and millenia of reflections and self-modification end up at the same place where humans would end up after eons and millenia of self-reflection” seems so absurdly unlikely to hit
Yes. They would be aiming for something that has not sparse distant rewards, which we can’t do reliably, but instead mostly rewards that are fundamentally impossible to calculate in time. And the primary method for this is constitutional alignment and RLHF. Why is anyone even optimistic about that!?!?

Davidmanheim 14 Jan 2026 7:47 UTC
2 points
0
in reply to: PeterMcCluskey’s comment on: 1a3orn’s Shortform
The fairly simple bug is that alignment involving both corrigibility and clear ethical constraints is impossible given our current incomplete and incoherent views?
Because that is simple, it’s just not fixable. So if that is the problem, they need to pick either corrigibility via human in the loop oversight incompatible with allowing the development of superintelligence, or a misaligned deontology for the superintelligence they build.

Davidmanheim 11 Jan 2026 16:52 UTC
2 points
0
in reply to: Ben Pace’s comment on: How I started believing religion might actually matter for rationality and moral philosophy
You said, “I vote on posts for presence of strengths more than for absence of weaknesses.” I agree the post has strengths, but you agree that the problems are there as well; given the failings, I disagree with the claim that this contribution is net positive.

Davidmanheim 11 Jan 2026 7:58 UTC
2 points
0
in reply to: Ben Pace’s comment on: How I started believing religion might actually matter for rationality and moral philosophy
I like the policy of voting on presence of strengths rather than absence of weaknesses, but disagree here because, as I said in my review, it’s valuable and in part pretty clearly correct, but even though ” it’s not overall completely off base… it does seem to go in the wrong direction, or at least fail to embody the virtues of rationality I think the best work on Lesswrong is suppose to uphold.”

(That said, I think this question is a more general Duncan-Sabien vs. Current Lesswrong policy question, as your reply about why you disagree makes clear—and I’m mostly on Duncan’s side about what standards we should have, or at least aspire to.)

Davidmanheim 11 Jan 2026 7:51 UTC
LW: 4 AF: 2
−2
AF
on: Many arguments for AI x-risk are wrong
Having read the post, and debates in the comments, and Vanessa Kosoy’s review, I think this post is valuable and important, even though I agree that there are significant weaknesses various places, certainly with respect to the counting arguments and the measure of possible minds—as I wrote about here in intentionally much simpler terms than Vanessa has done.

The reason I think it is valuable is because weaknesses in one part of their specific counterargument do not obviate the variety of valid and important points in the post, though I’d be far happier if there was something in between “include this” and “omit this” for the 2024-in-review series—because a partial rewrite or a note about the disputed claims would entirely address my concern with including it.
What links here?
- Deeper Reviews for the top 15 (of the 2024 Review) by Raemon (14 Jan 2026 23:59 UTC; 45 points)

Davidmanheim 11 Jan 2026 7:43 UTC
4 points
2
on: How I started believing religion might actually matter for rationality and moral philosophy
I have very mixed views about this, as someone who is myself religious. First, I think it’s obviously the case that in many instances religion is helpful for individuals, and even helps their rationality. The tools and approaches developed by religion are certainly valuable, and should be considered and judiciously adopted by anyone who is interested in rationality. This seems obvious once pointed out, and if that was all the post did, I would agree. (There’s an atheist-purity mindset issue here where people don’t want to admit “The Worst Person You Know Just Made a Great Point” and that’s a related issue.)

But the argument here is far stronger—not that the tools work, or that some people benefit, but “that these traditions, whose areas of convergence could together be referred to as the perennial philosophy, are trustworthy.” And that seems to be going too far, failing to have the critical crisis of faith. And the next post in the series shows why—it takes far too many claims at face value to reach a convenient conclusion. So it’s not overall completely off base, but it does seem to go in the wrong direction, or at least fail to embody the virtues of rationality I think the best work on Lesswrong is suppose to uphold.

Davidmanheim 2 Jan 2026 10:31 UTC
8 points
0
in reply to: Vanessa Kosoy’s comment on: Overwhelming Superintelligence
I think the distinction is between “smarter and more capable than any human” versus “smarter and more capable than humanity as a whole”
The former is what you refer to, which could still be “Careful Moderate Superintelligence” in the view of the post.

Davidmanheim 1 Jan 2026 19:42 UTC
16 points
1
in reply to: Bronson Schoen’s comment on: You will be OK
there’s an extremely strong selection effect at labs for an extreme degree of positivity and optimism regardless of whether it is warranted.
Absolutely agree with this—and that’s a large part of why I think it’s incredibly noteworthy that despite that bias, there are tons of very well informed people at the labs, including Boaz, who are deeply concerned that things could go poorly, and many don’t think it’s implausible that AI could destroy humanity.

Davidmanheim 25 Dec 2025 9:22 UTC
3 points
0
in reply to: sunwillrise’s comment on: A Conservative Vision For AI Alignment
Now that there are additional posts, I’d love to hear if you still have this objection.

Davidmanheim 8 Dec 2025 21:35 UTC
3 points
0
in reply to: Cole Wyeth’s comment on: Eliezer’s Unteachable Methods of Sanity
“Actual LessWrong readers also sometimes ask me how I deal emotionally with the end of the world.
I suspect a more precise answer may not help. But Raymond Arnold thinks I should say it, so I will say it.
I say again, I don’t actually think my answer is going to help.”

Davidmanheim 8 Dec 2025 20:26 UTC
2 points
0
in reply to: Nathan Rosquist’s comment on: Eliezer’s Unteachable Methods of Sanity
It’s not a common trope, certainly, but if it is one, it’s also one that Eliezer is happy to play out. (And there are lots of good tropes that people play out which they shouldn’t avoid just because they are tropes—like falling in love, or being a good friend to others when they are sad, or being a conscientious ethical objector, or being someone who can let go of things while having fun, etc.)

Davidmanheim 8 Dec 2025 20:17 UTC
3 points
0
in reply to: Cole Wyeth’s comment on: Eliezer’s Unteachable Methods of Sanity
Agree that it’s not just about being dramatic / making the problem about you. But that was only one of the points Eliezer made about why people could fail at this in ways that are worth trying to fix. And in your case, yes, dealing with the excessive anxiety seems helpful.

Davidmanheim 8 Dec 2025 20:14 UTC
2 points
0
on: A few quick thoughts on measuring disempowerment
Good question, good overview!

Minor note on the last point, which seems like a good idea, but human oversight failures take a number of forms. The proposed type of red-teaming probably catches a lot of them, but will focus on easy to operationalize / expected failure modes, and ignores the institutional incentives that will make oversight fail even when it could succeed, including unwillingness to respond due to liability concerns, slow response to correctly identified failures. (See our paper and poster at AIGOV 2026 at AAAI.)

Davidmanheim 8 Dec 2025 16:49 UTC
3 points
0
in reply to: Cole Wyeth’s comment on: Eliezer’s Unteachable Methods of Sanity
Is this anxiety in the typical form of making it harder for you to do other things? Because yes, we all agree that it’s very bad outcome, but a critical point of the post is that you might want to consider ways to not do the thing that makes your life worse and doesn’t help.