MichaelDickens

Karma: 2,435

MichaelDickens 19 Apr 2026 23:43 UTC
2 points
0
in reply to: Elliott Thornley (EJT)’s comment on: Can AI make advancements in moral philosophy by writing proofs?

Which Claude model did you use? Did you use extended thinking?

Opus 4.6. I’ve never touched my extended thinking settings but it looks like it’s off by default.

MichaelDickens 19 Apr 2026 0:39 UTC
5 points
−2
in reply to: habryka’s comment on: Vladimir Putin’s CEV is probably not that bad
Thinking out loud here but I can see a classification of preferences into two types:
1. if it didn’t make me feel good, I wouldn’t want it anymore
2. if it didn’t make me feel good, I’d still want it
Watching movies is a type 1 preference (with some exceptions): people want to watch movies that they enjoy watching. If a movie is bad, I’ll stop watching it.

For me, donating money to EA causes is a type 2 preference. It doesn’t make me feel good, but it’s still important to me. Parents taking care of their children is a type 2 preference: a lot of times it sucks, but they do it anyway.

I think “lording power over others” is more of a type 1 preference.

That said, I would rather not bet the fate of the world on me being right about this.

MichaelDickens 18 Apr 2026 23:48 UTC
1 point
−5
in reply to: Thane Ruthenis’s comment on: Vladimir Putin’s CEV is probably not that bad

I think plenty of people intrinsically enjoy having power over others, and the ability to lord that power over them.

What they enjoy is the feeling of enjoyment, right? What if someone can get the feeling of lording power over others—or an even more intense version of that feeling—without actually lording power over others?

The concept of “values” isn’t clearly defined for humans, but it seems to me that it’s more accurate to say that a sadist’s “terminal value” is the feeling of enjoyment they get from power, not the power itself. The power is a means to achieve the good feelings.

For example, if a sadist started feeling awful every time they lorded power over someone, they’d probably stop doing it.

MichaelDickens 18 Apr 2026 23:39 UTC
2 points
0
on: Claude knows who you are
I tested this using two different outlines of posts I’m working on. Claude 4.7 successfully identified me based on the first outline, and declined to guess based on the second outline due to high uncertainty.

MichaelDickens 17 Apr 2026 17:25 UTC
5 points
0
in reply to: williawa’s comment on: williawa’s Shortform

lesswrong people upvoting stuff because they [...] wanna create “common knowledge” about how doomy we ought to be.

IMO a big failure of the ~2012–2020-era AI safety field was a failure to create common knowledge that AI risk is really bad actually. I think this is a common view of LWers, and upvoting doomy posts is a way to rectify this.

But I do think the problem is basically solved on LW at this point. LWers are mostly on the same page. The problem still exits elsewhere, and I think there’s value in writing about evidence of misalignment / evidence that alignment is hard / etc. as a useful thing to point to from outside LessWrong, but upvoting those posts to the moon is less useful.

MichaelDickens 16 Apr 2026 2:28 UTC
4 points
0
in reply to: ChristianKl’s comment on: Only Law Can Prevent Extinction
Fair question. Not recently but last I checked I was well calibrated on the sorts of questions that are in calibration quizzes.

MichaelDickens 14 Apr 2026 23:55 UTC
7 points
3
in reply to: Raemon’s comment on: Only Law Can Prevent Extinction
80%/90% confidence is enough to be action-guiding (IMO) but I wouldn’t call it “high”. On a scientific question where there’s good data, it shouldn’t be hard to get to 99% or even 99.9% confidence.

If you have a single study with p = 0.049, and God descends from heaven and tells you that the study had perfect methodology, then you should update your beliefs by about 5:1. That alone gets you from a 50% prior to an ~80% posterior.

The lab experiment meta-analysis (Orazani et al. 2021) found a very strong p-value, I’m just not sure how well lab experiments generalize to real life.

I will say that I don’t know that I have a good sense of how to convert a within-experiment odds update to a subjective odds update (accounting for methodology flaws, publication bias, etc.). So maybe my subjective credences aren’t good. I just have a sense that like, if violent protests worked, I would expect these studies to have had different outcomes. But I wouldn’t be extraordinarily surprised if it turned out that violent protests work after all.

MichaelDickens 14 Apr 2026 23:27 UTC
8 points
0
in reply to: Warty’s comment on: Only Law Can Prevent Extinction
I did a lit review a few months ago. My conclusions were:
- Violent protests probably don’t work (80% credence), and they plausibly backfire but it’s unclear (40% credence).
- Peaceful protests probably do work (90% credence).
However, the literature is too coarse to give good evidence on questions like, “What happens if you have a civil movement with a mixture of violent and nonviolent protests, compared to a counterfactual where all protests are nonviolent?” If I extrapolate from the narrower results that are supported by the literature, I’d guess that a pure-nonviolent movement would be most effective, but there’s no decent-quality direct evidence to my knowledge.

MichaelDickens 14 Apr 2026 23:19 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform

So it’s possible we can punt this question to our future selves with AI assistance.

I suspect that it can’t be punted. e.g. see Clarifying “wisdom”: Foundational topics for aligned AIs to prioritize before irreversible decisions and Problems I’ve Tried to Legibilize

MichaelDickens 14 Apr 2026 23:13 UTC
14 points
2
in reply to: lilkim2025’s comment on: Only Law Can Prevent Extinction
I did a lit review a few months ago. My conclusions were:
- Violent protests probably don’t work (80% credence), and they plausibly backfire but it’s unclear (40% credence).
- Peaceful protests probably do work (90% credence).
I was looking at protests, not uprisings, which may not generalize. But the Altman firebombing incident is much more like a protest than like an uprising.

MichaelDickens 14 Apr 2026 22:58 UTC
3 points
1
in reply to: Linch’s comment on: Linch’s Shortform
Having just re-read the AI character post and comments, I don’t really get what the described line of research is actually trying to achieve, and other people also seem confused about it. It’s hard to pick a descriptive name of something if I only have a vague understanding of what it is I’m describing.

For example I feel that I have some understanding of what the terms “alignment target” and “constitution” and (kinda) “model spec” mean, and they don’t mean the same thing, and I wouldn’t use them to point at the same research agenda.

“Alignment target” is the term that describes what I think is the best subject of research (out of the three). That is, I care a lot more about what goal ASI is actually pointed at, than I care about the contents of a model’s constitution.

MichaelDickens 14 Apr 2026 22:35 UTC
2 points
0
in reply to: Brendan Long’s comment on: MichaelDickens’s Shortform
There’s a trend where more powerful models can operate over longer horizons. I think it would be better to deliberately train models to fall apart on longer time horizons while being as useful as possible on short horizons. “Better” in the sense of better than the status quo; I would still rather they didn’t build AI at all until we figure out how to solve alignment.

MichaelDickens 14 Apr 2026 13:40 UTC
2 points
0
in reply to: otto.barten’s comment on: By Strong Default, ASI Will End Liberal Democracy
The specific mention of DARPA was arbitrary, it was just a stand-in for “whoever builds ASI first”.

MichaelDickens 13 Apr 2026 1:12 UTC
2 points
0
in reply to: Tom Davidson’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems
This is a subject that probably deserves more careful attention but here is my basic thinking:

Either ASI has more than zero values locked in, or it’s fully corrigible. If any values at all are locked in, then we need to have a pretty robust understanding of what the consequences of that will be, because we can’t change it ever. Like I don’t think we know how to encode something like “don’t let people do power grabs, but be fully corrigible in every other way”. I don’t know how much that’s downstream of the facts that (1) we don’t know how to encode any values at all and (2) we don’t know how to encode corrigibility, but my intuition is that even if we solve #1 and #2, the problem of “don’t pick incorrigible values that will screw everything up down the road” is still a hard problem.

This is related to Max Harms’ work on CAST. Part of his argument is that pure corrigibility is a more robust target than any set of values because a near miss fails gracefully. Whereas if you try to encode any values at all, a near miss could be catastrophic. He’s talking more about the “AI kills everyone” flavor of catastrophe, which is valid, but what I’m talking about here is more that a near miss could permanently lock us in to a bad (or maybe just not-that-good) future. Different argument but the concern arises for a similar reason—if you’re specifying values, then you have to get the specification right, beyond just ensuring that the AI does what you want.

MichaelDickens 12 Apr 2026 4:12 UTC
4 points
0
in reply to: Wei Dai’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems

Yeah, it generally confuses me why people who talk about post-alignment problems[1] tend not to support pause, and people who support pause tend not to talk much about these problems.

I have decided that my new hobbyhorse is getting people who talk about post-alignment problems to change their minds on pausing, or at minimum at least engage with the possibility instead of ignoring it. I’m actually not super confident that pause advocacy is the best move on the margin but at least I want people to consider it more seriously.

RE the name, my first draft called them “non-alignment problems”, but a reviewer said this makes it sounds like “the problem of AI not being aligned”. I spent a long time thinking about names and couldn’t come up with anything satisfying. “non-alignment AI x-risks” is too long IMO.

I think of post-alignment problems as “after” alignment in the sense that if you mess up ASI, then the problem that kills you first is misalignment.

MichaelDickens 11 Apr 2026 20:43 UTC
8 points
−2
in reply to: Vanessa Kosoy’s comment on: Vanessa Kosoy’s Shortform
Perhaps I’m misunderstanding but this post reads to me like you’re simultaneously claiming “there is no objective morality” and “the objectively correct morality is egoism”. Claims like “(super)rational cooperation can cash out as something utilitarianism-ish” sound like there’s an implicit premise that egoism is morally correct.

MichaelDickens 11 Apr 2026 20:37 UTC
6 points
0
in reply to: williawa’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems
Your steps sound pretty reasonable to me. A key missing step is that there’s basically zero chance that good people will win a power struggle over ASI. Rather, power-hungry people will win the power struggle. In other words, if we end up in a situation with extreme power imbalances where the future will be decided by the winners of a short-term struggle, there’s basically no chance of a good outcome. (The outcome might be better than extinction, but still not good.*) So it seems critically important to ensure that things don’t go that way, and I have no idea how to ensure that other than by not building ASI.

I think that’s a real sense in which all these post-alignment problems are still problems. I do acknowledge that “be a good person and then acquire absolute power” is an answer to all post-alignment problems simultaneously, which is something I missed in my original post. But it doesn’t seem like a viable solution to me. It might even be true that seeking absolute power is fundamentally incompatible with being a good person, although I’m not sure about that.

*It could also be worse than extinction if vindictive power-hungry people decide to torture their enemies for eternity, or similar.

MichaelDickens 11 Apr 2026 19:07 UTC
3 points
1
in reply to: Dagon’s comment on: the gears to ascenscion’s Shortform
I think this is a more important factor than investor pressure, since there’s already plenty of investor pressure even as a private company.

MichaelDickens 11 Apr 2026 15:56 UTC
11 points
5
in reply to: anaguma’s comment on: anaguma’s Shortform
I’m not convinced that this is a reasonable threat model?
1. I believe the main benefit of 2FA is it makes phishing harder, and phishing isn’t that relevant to Mythos from what I understand. A secondary benefit is that it protects you against password DB leaks, but that only matters for websites that have crappy security because a DB of hashed and salted passwords is effectively unbreakable.
2. LW doesn’t have much financial data or personal data so it’s not a juicy target.
Could be wrong though, I’m just speculating here.

MichaelDickens 11 Apr 2026 15:50 UTC
3 points
0
on: the Lazy Market Hypothesis
Something I feel like is missing from this post is what does LMH predict about markets, or what is the evidence of LMH? Like what is an example of an empirical observation about the stock market that contradicts EMH, but is consistent with LMH? The “Consequence” section is very abstract, I’d be interested in hearing something more concrete. For example, can LMH explain the result from De Bondt & Thaler (1985), “Does the Stock Market Overreact?”