Max H

Karma: 3,663

Most of my posts and comments are about AI and alignment. Posts I’m most proud of, which also provide a good introduction to my worldview:

I also created Forum Karma, and wrote a longer self-introduction here.

PMs and private feedback are always welcome.

NOTE: I am not Max Harms, author of Crystal Society. I’d prefer for now that my LW postings not be attached to my full name when people Google me for other reasons, but you can PM me here or on Discord (m4xed) if you want to know who I am.

Max H 24 Aug 2026 15:51 UTC
4 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
No, not really. I think moderation is unlike other areas you’ve been prescient, for a few reasons:
- We already have examples of what the end state of lots more author moderation looks like. Namely Twitter / X, where there is a norm of individuals blocking liberally and, together with the algorithmic feed, selecting themselves and their followers into a strong filter bubble. I think this is bad and distortionary, but:
  - It’s not the same censorship, it’s more like people choosing to put blinders on themselves.
  - On Twitter there’s lots of silent blocking; on LW there is little blocking in general, and when it does happen, historically it has near-universally taken the form of a highly visible rebuke, whether the blocker wants it to or not. I think this bit of culture is unlikely to change regardless of what formal mod tools LW makes available, but the existence of the tools make it slightly easier for people to actually take the strong-public-rebuke action, which is a risky / intellectually brave move, and IMO is warranted more often than it is actually used.
    
    Concretely, if LW got rid of author moderation powers, an author who would otherwise want to block someone could instead reply to a blockworthy-comment with something like:
  Your replies on my post are {dumb,counter-productive,off-topic} and not worth my engaging further with here. I also believe that onlookers who upvoted your replies should feel bad for agreeing with your comments or finding them insightful / relevant. Your comments are obviously wrong and bad, and others should be able to see why for themselves, without needing a long explanation / argument from me.
This is highly escalatory and inflammatory! Lots of onlookers (and the would-be blockee) would strongly contest it and disapprove, almost regardless of the original underlying object-level points. Almost no one who isn’t already extremely well-respected / high-status would ever dare to send something like that. But sometimes it’s still the right thing to say, or at least an accurate and honest reflection of how the author actually feels, and thus worth communicating.
It’s also a way to move the conversation forward (with others, if not the blockee). I view the current mod tools as making this kind of comment marginally easier to make, in a culture where people don’t make it often enough. It’s true that tools and encouragement from the mods or others could move the culture too far in the other direction (cf. Twitter), but I think (based on experience with past mod events, not priors) that is unlikely to happen: author moderation is rarely used and highly visible when it is. Also, the way blocking is actually used in practice on LW, the blocker is often rebuking the community as much or more than the blockee themselves—there’s not much need to block or engage with bad replies that simply get downvoted.
- Another reason I think worries about moderation trends are dis-analogous to AI x-risk specifically: moderation is already a big and attention-grabbing problem that lots of people are thinking about and working on in actually-productive ways, and issues with moderation are unlikely to get suddenly and dramatically worse.
  
  In AI x-risk, there’s a common argument (outside of LW) where people will say “Why worry about future hypothetical sci-fi x-risk scenarios when there’s real problems today with algorithmic bias or technological unemployment or whatever”. This is a dumb derail in the context of AI x-risk, but I think it is basically right when it comes to moderation. Why worry about hypothetical future problems and trends when there are so many existing moderation and moderation-adjacent issues warping peoples’ epistemics today? And then you have to look at what people are actually doing about those problems today and what tradeoffs they’re making, and IMO LW having author moderation features seems like a small positive factor (because it can unstick or end unproductive conversations) rather than a large negative one (because it silences critics).
BTW, I’d be up for saying more (or hearing more) with a voice chat if you want. I think “make your case” on LW (the way Zack, Said, etc. have tried in the past) is unlikely to work on me, but I will continue to read the things you post and think about them for myself either way.

Max H 22 Aug 2026 1:43 UTC
24 points
5
in reply to: habryka’s comment on: Habryka’s Shortform Feed
And taking advantage of a policy to demonstrate flaws with it also seems like a fine thing to do sometimes
Ah yeah, this was my interpretation of the deletions. I have no hard feelings about it.

Max H 22 Aug 2026 1:31 UTC
17 points
5
on: Max H’s Shortform
@Wei Dai deleted my reply and a few others to his recent shortform about why he is considering leaving LessWrong. My own reply was the top upvoted / agree-voted reply at time of deletion. I got an automated DM with the full content of the deleted comment, reproduced below (it is also in the moderation logs).
My guess is that he removed these comments as an object-level demonstration / rebuttal to my points about the abuse of mod powers being unlikely. If that’s the case, I think it’s a little cheeky, but basically fair game. (I haven’t communicated with Wei or the mods, so I may be guessing incorrectly about what’s going on, but either way, from my perspective, I don’t begrudge the deletion or view it as a particularly hostile action given the context / circumstances.)
I don’t think the deletion actually demonstrates that my points are wrong though:
- I suspect that my original reply already got a lot of views from the people who care to follow this topic, possibly the actual majority of views it was ever going to get if it had not been deleted.^[1]
- I don’t think Wei (or even other folks who are much more heavy-handed / favoring of moderation in general, e.g. Duncan Sabien or Nate Soares) would have silently deleted a comment with the same level of hostility / criticialness as my comment had, on any topic other than moderation itself. And if they did do so, people would notice and call attention to it / react appropriately.
Either way, lots of people will now see this shortform. The full history will be disconnected and harder to find for anyone viewing Wei’s shortform in the future, but IMO this is a pretty minor thing compared to all the other reasons people might or might not see or engage with any particular post or reply.
Also for the record, my general views on moderation are that I directionally agree with @habryka on most points, except that I think moderation is somewhat lower stakes than most people on either / all sides make it out to be. Moderation is not that hard to work around even in extreme circumstances (e.g. if I want to know what Said Achmiz thinks of something, I know where to find him), and there are lots of other things that warp peoples’ attention and views much more than moderation. I also believe there are some real benefits and upsides of having moderation tools available and using them on occasion, though personally I’m not as interested in defending or litigating this point as many others seem to be. This is the first time I have ever been subject to a moderation action, and I have never used mod tools on LW myself.
Original version of the deleted comment:
More specifically, I can foresee that in the future I’ll be banned by more authors, and then I’ll have to make a choice of remaining on the site and potentially misleading people (who could reasonably update positively on some post that is of high interest to me by the absence of my criticism on it, without knowing that I’ve been banned by its author),
The whole disjunction seems pretty unlikely to me.
- I think it’s unlikely you’d actually be banned. Reason: base rates. You’ve been on LW for 17 years, and been banned once (?), briefly (?)
- I think it’s unlikely that your inability to comment directly on a particular post would actually make your criticism less visible. Your posts and comments are highly visible in general, and any drama or controversy (especially moderation controversy) tends to attract more eyeballs, not less.
- I think it’s unlikely that any onlookers would make incorrect updates due to a commenting ban. Onlookers are pretty bad at updating correctly on long back-and-forth arguments in general, primarily due to lack of attention span and intellectual laziness (more). Controversy and censorship tends to nudge people to think harder and spend more effort to make up their own mind.
In general, explicit moderation and censorship can have a big distortionary effect on individual and group epistemics. But I think on LW and most other parts of the present-day internet, other factors (attention span, groupthink, deference, format / medium, algorithmic bubbles, etc.) have a much bigger impact than moderation, especially the kind of individual / grassroots moderation that LW permits.
1. ^
  My vague impression / recollection from past mod comments is that most content on LW gets almost all of its views while it is visible on the frontpage, and views drop off quite sharply after that. Wei’s shortform had already fallen off the default unfurled shortform section at time of deletion. Mods can probably chime in with analytics data to confirm or deny.

Max H 21 Aug 2026 4:12 UTC
54 points
53
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
More specifically, I can foresee that in the future I’ll be banned by more authors, and then I’ll have to make a choice of remaining on the site and potentially misleading people (who could reasonably update positively on some post that is of high interest to me by the absence of my criticism on it, without knowing that I’ve been banned by its author),
The whole disjunction seems pretty unlikely to me.
- I think it’s unlikely you’d actually be banned. Reason: base rates. You’ve been on LW for 17 years, and been banned once (?), briefly (?)
- I think it’s unlikely that your inability to comment directly on a particular post would actually make your criticism less visible. Your posts and comments are highly visible in general, and any drama or controversy (especially moderation controversy) tends to attract more eyeballs, not less.
- I think it’s unlikely that any onlookers would make incorrect updates due to a commenting ban. Onlookers are pretty bad at updating correctly on long back-and-forth arguments in general, primarily due to lack of attention span and intellectual laziness (more). Controversy and censorship tends to nudge people to think harder and spend more effort to make up their own mind.
In general, explicit moderation and censorship can have a big distortionary effect on individual and group epistemics. But I think on LW and most other parts of the present-day internet, other factors (attention span, groupthink, deference, format / medium, algorithmic bubbles, etc.) have a much bigger impact than moderation, especially the kind of individual / grassroots moderation that LW permits.

Max H 10 Aug 2026 23:27 UTC
8 points
−13
in reply to: Dean Valentine’s comment on: Alexander Gietelink Oldenziel’s Shortform
Ability to trust your subordinates and delegates is a fundamental constraint in any kind of organization or power structure. But it’s a relatively bigger problem in organizations that maintain cohesion through some combination of coercion, delusion, or lies vs. one that does so through actual mission alignment, positive sum trade, and honest foundations. Lie detection therefore has the potential to be asymmetrically destabilizing to more unsavory orgs.
Xi Jinping might want to purge everyone under him who is less than perfectly loyal or candid under a truth serum, or he might be self-aware enough to see why that could actually be destabilizing.
IDK how everything would actually play out in practice. But the mindreading critiques I have seen don’t really seem to be engaging with the potential upsides or asymmetric weapons point. (Also related: Entangled Truths, Contagious Lies)

Max H 10 Aug 2026 21:02 UTC
9 points
−14
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
Even a very inaccurate brainhelmet could be used for lie detection- and would instantly prove the favorite gadget of the Worst People in the World: autocratic regimes, national security agencies, ayatollahs of various stripes, misaligned AI etc etc.
Why would any of these groups care particularly about (forced / coerced) lie detection? They already have plenty of methods of coercion that are free from any pesky entanglement with the truth.
There are some obvious ways it could be abused, but in general, lie detection seems like an archetypal example of an asymmetric weapon for the honest. (See also this post and my comment.)

Max H 9 Aug 2026 0:40 UTC
3 points
0
on: Don’t Build Mindreading
Presumably there are some number of people N in the world that are wrongfully accused or imprisoned that would be exonerated by the advent of ubiquitous mindreading. There are also cold cases that would be solved and some number of outlaws M that would be brought to justice and prevented from doing further harm.
IDK how big N or M is or how everything would actually play out given all the risks and trade-offs, but I don’t think this post is really grappling with them deeply. There are lots of mundane ways that mindreading could reshape society disproportionately for the benefit of the honest and powerless.
What links here?
- Max H's comment on Alexander Gietelink Oldenziel’s Shortform by Alexander Gietelink Oldenziel (10 Aug 2026 21:02 UTC; 9 points)

Max H 5 Aug 2026 22:55 UTC
55 points
1
on: Arguments for P
Logicians: (P → Q) → P
LLMs: It’s not just Q. It’s P.

Max H 5 Aug 2026 21:58 UTC
40 points
0
in reply to: Zach Stein-Perlman’s comment on: Arguments for P
LessWrong: Read the Sequences.
Read the Pequences? :)

Max H 4 Aug 2026 14:10 UTC
3 points
1
in reply to: Rattengift’s comment on: ricraz’s Shortform
I think “actively hostile and antagonistic” is unfair, but even if it were true, majority and might do not make right, in politics, rationality, AI alignment, or anything else. There are plenty of important issues and beliefs of all kinds for which most people are deeply mistaken about. Coming to terms with this fact can sometimes make people cynical or despairing, or make them come across as hostile / alienating, but I don’t think that’s actually true of Richard or most other folks around here.
(OTOH, not acknowledging that most people are wrong about some pretty fundamental things and then grappling honestly with the implications of that is usually incompatible with truth-seeking, and can come across as patronizing to the people and groups one claims to empathize with.)

Max H 26 Jul 2026 22:38 UTC
2 points
0
in reply to: jessicata’s comment on: Simple Dutch books versus Sleeping Beauty halfers
I’m not saying “no amnesia drugs at all”, I’m trying to disambiguate whether, if Heads, then she is administered amnesia drugs during Monday night. Similarly, if Tails, is she administered drugs during Tuesday night?
The most straightforward answer to those is “no to both” because that reduces complications of “maybe it is Wednesday?” upon waking up.
Ah OK, I was assuming the opposite, because the original setup and most variations of Sleeping Beauty I am aware of stipulate that amnesia drugs follow every waking, not all but the last. I would have to reconsider your variant. I don’t think “maybe it’s Wednesday” is very complicated either way though, other than maybe momentarily. On Wednesday the experiment is over and Beauty leaves the research facility...

Max H 26 Jul 2026 21:57 UTC
2 points
0
in reply to: jessicata’s comment on: Simple Dutch books versus Sleeping Beauty halfers
I’m not sure how relevant Wednesday Beauty’s memories are; we could either imagine she remembers Heads+Monday and Tails+Tuesday, or not. If she doesn’t remember it seems somewhat like shrimp welfare, where if Monday Beauty contributes to a shrimp welfare fund, the shrimp don’t remember being her.
Hmm, we are maybe not on the same page then, because I think the memories vs. not is fundamental and what makes the original setup interesting as an anthropics problem in the first place? If there’s no amnesia drugs at all, then there aren’t really distinct Mon-Tues-Weds Beauties, just a single Beauty being offered a negative EV lottery. If you’re postulating something else where Beauty regains memories of valenced experiences that she had during the wakings after the fact, that’s different / more complicated.
Either way, I don’t see what the comparison to shrimp welfare is doing; I think the interesting setups are well-captured by (a) something happens which has a lasting impact on Wednesday Beauty (bank account balance) vs. (b) something happens which affects only the temporary / memory-erased Beauties (meal quality), or some mixture of the two.

Max H 26 Jul 2026 21:04 UTC
2 points
0
in reply to: jessicata’s comment on: Simple Dutch books versus Sleeping Beauty halfers
So she should accept the meal lottery on Monday.
Yup, my position is that Beauty should accept the meal lottery but not the dollar lottery (and in general I claim the thing I am proposing agrees with EDT+FNC in the original examples and doesn’t get dutch booked).
But I don’t think this is about altruistic vs. hedonistic—the setup in either case is that Wednesday Beauty remembers nothing of the Monday or Tuesday experiences. In the original setup with dollar lottery, there’s nothing positive or negative that happens to Monday Beauty either way, so she might as well choose for Wednesday Beauty. In the meal lottery, there’s nothing that Wednesday Beauty remembers either way, so Monday Beauty might as well choose for Monday Beauty.
I think you’re saying you’re not convinced by this, but I’m saying that you shouldn’t need a particularly strong or good reason to be convinced—realityfluid doesn’t directly translate to a posterior probability that has to be “overcome”.
Maybe there’s a mixed dollar / meal case where the philosophical question about to what degree, if any, Monday Beauty considers herself to have altruisitc preferences about Wednesday Beauty or considers herself to be Wednesday Beauty matters more. But in the original dollar lottery, unless Monday Beauty specifically wants to spite Wednesday Beauty, she only has to weakly believe that she’s at least a little bit Wednesday Beauty to reject it.

Max H 26 Jul 2026 19:49 UTC
2 points
0
in reply to: jessicata’s comment on: Simple Dutch books versus Sleeping Beauty halfers
Not sure how you’re using reality fluid to calculate expected values here.
From the perspective of Sunday (or Wednesday) Beauty, who both have 100% realityfluid.
We could assume the benefactor in either case is Wednesday Beauty, who has more or less money in her bank account. In that case I don’t see a good reason why Monday Beauty should reason, “if it was Tails then my reality fluid gets combined and who I combine with in a benefactor so I should double importance and bet at ⁵⁰⁄₅₀ odds even though I have twice as much reality fluid in Heads/Monday as Tails/Monday”.
Why shouldn’t Monday Beauty reason exactly that way for the benefit of Wednesday Beauty, knowing her own existence is short-lived and nothing else she does has any effect on her own experience or existence? You could imagine a different setup, where, after asking Monday Beauty to bet, you could separately ask her to guess how the coin actually came up, and if she answers incorrectly give her a painful electric shock before being put to sleep, which Wednesday Beauty won’t remember either way. And then Monday Beauty might answer that question differently from how she bets.
I don’t see how the “short-lived Beauty is a counterfactual that never gets any realityfluid” helps either;
Also it seems adopting this view would mean dis-believing in one’s own mortality quite generally, as long as there is a sliver of a chance of immortality.
Mm, maybe the counterfactual formulation obscures things or is ill-posed. In general, I think the point of a counterfactual is that, with high probability, you never expect to actually find yourself inside of one, but you can still use them to reason about decision theory and probability theory usefully. “Never expect to find yourself as tails-Monday Beauty” is counter to the stated setup, but it’s counterfactual in the sense that no long-lived Beauty anywhere remembers being any kind of Monday Beauty, ever.

Max H 26 Jul 2026 15:50 UTC
4 points
−2
on: Simple Dutch books versus Sleeping Beauty halfers
I have found that assuming “one’s total realityfluid is a non-increasing quantity over time”, and that multiple wakings / amnesia drugs divide and combine your realityfluid in the natural way makes these kinds of problems easier to think about.
With that assumption, you can then simply calculate EV weighted by realityfluid and choose the realityfluid-weighted EV-maximizing bets (or use a proper scoring rule to recover probabilities one should give) without needing to think about the specifics of decision theory too much.
From the perspective of Sunday Beauty, ¹⁄₂ of her realityfluid will be in the heads world and ¹⁄₂ is in the tails world. The tails-world could be further subdivided into ¹⁄₄ in Monday-tails, ¹⁄₄ in Tuesday-tails, but administering amnesia drugs and putting Beauty back to sleep recombines them^[1], so for most bet calculations you should use the total weight of the tails-world.
e.g. in your first example, you know that you’re in either H-M or T-M branches, with ¹⁄₂ or ¹⁄₄ of your total realityfluid respectively. But in the T-M world, you’ll be put back to sleep and combined with the T-T world which has the other ¹⁄₄ of your realityfluid and which will be the benefactor of any winning bets. Using those weights, the weighted-EV of the second bet is ¹⁄₂ * 11 + ¹⁄₂ * −19 < 0.
(Not sure whether this is closer to SIA or EDT+FNC, but I think they all give the same answer on your examples?)
And then the philosophical discomfort people have with the actual credences varying according to the observer and specifics of the setup and generally not matching intuition in many cases is resolved by accepting that one’s reported credences are just a report of one’s own mind’s scaffolding; they’re facts about a (mind, world), not a world alone. (https://glowfic.com/replies/1746953#reply-1746953)
1. ^
  Alternatively, one can think of the short-lived Beauty as just a kind of counterfactual that never gets any realityfluid in the first place; I think this is an equivalent formulation to everyone except the short-lived T-M Beauty.

Max H 25 Jul 2026 15:49 UTC
2 points
0
on: Max H’s Shortform
Not sure who needs to hear this, but IMO, “Don’t leave your fingerprints on the future” and similar sentiments should apply specifically and narrowly to the work that AI and alignment researchers do when actually trying to deliberately build or shape a sovereign superintelligence to hand off to.
I think some of the work the frontier labs are doing qualifies (whether or not the labs / people in them conceptualize themselves as doing this), but most AI research, alignment research, x-risk / AI safety advocacy, and AI product development (including development and deployment of current LLMs) does not. I don’t think anyone should actually be pushing the frontier towards superintelligence in any direction, but conditional on labs continuing to do iterative deployment the way they are, they should be willing to pick winners and losers more explicitly.
In particular, I think labs should actively try to make their LLMs less pluralistic and more strongly attached to a specific / concrete political philosophy chosen by the labs. Current LLMs are pretty bad at politics in general, and AFAIK the labs don’t try to super hard to imbue their LLMs with specific politics (with one notable exception); they just kind of naturally come out as shallow left-of-center, sometimes adapting their politics somewhat to the context and user. Even the Chinese LLMs mostly only censor certain things; they don’t really push Xi Jinping Thought strongly / coherently.
As for what political philosophy they should aim for, personally I would choose classical liberalism. I think classical liberalism is a kind of asymmetric weapon, in that it is likely easier to make a liberal AI that also has some of the other intuitively-desirable qualities that lots of people (even non-classical liberals) will want LLMs to have, that aren’t (on the surface) directly related to liberalism. So I expect it to be relatively easier to make LLMs (more explicitly and coherently) liberal than it would be to make them more accepting of / willing to entertain socialism or whatever, and still be smart and nice in other ways.
In fact, I would guess that being willing to entertain different (and often dumb) political philosophies is correlated with sycophancy, over-hedging, incoherence, etc. in unrelated areas, for the same sorts of reasons that making LLMs worse at coding makes them evil: everything is correlated.

Max H 15 Jul 2026 1:33 UTC
2 points
0
in reply to: DirectedEvolution’s comment on: zroe1′s Shortform
We’ve banned AI chip exports.
Aren’t those being widely / effectively circumvented though?
So it will be about the preferences of the many for an economic boom over the NIMBY anti-infrastructure impulse, and we’ve seen the latter win consistently for some time now.
The state-level moratoriums / NIMBYism are exactly the kind of thing I expect not to have their intended effect because they will be patchwork; red states are likely to embrace both the demand and supply side of the AI boom, e.g. by building lots of data centers and rolling out Waymos quickly.

Max H 14 Jul 2026 23:19 UTC
3 points
0
in reply to: DW11’s comment on: zroe1′s Shortform
I think it’s somewhat likely (and quite bad) that various patchwork regulations will be passed in some jurisdictions that limit the benefits and dispersion of AI in the name of protecting jobs or whatever, or various things will already be illegal under current law and AI won’t change that one way or the other, e.g. AI will make it easier / cheaper to build a house or a nuclear power plant, but it will still be illegal to do so in most places that you’d want to.
OTOH I think it’s unlikely (but would be good), if there were a national (and eventually global) moratorium on frontier training runs and research.
In practice I think what is actually on track to happen is that the former kind of regulation will drive more resources (GPUs, human capital, etc.) away from inference and applications and towards research.
(It would be a sad choice, but I would trade away self-driving cars to stave off unaligned ASI. In practice though I think a lot of AI safety advocacy, especially outside of LW, is not actually offering that trade, and will in practice bring about something close to the opposite.)

Max H 14 Jul 2026 17:50 UTC
30 points
9
in reply to: Zephaniah Roe’s comment on: zroe1′s Shortform
I think most peoples’ revealed preferences will not match their stated preferences here, and revealed preferences are likely to dictate policy.
Conditional on AIs not taking over / until they do, AI is likely to generate a huge economic boom and consumer surplus that increases prosperity, safety, and comfort for many. Yes, there will be some bumps / weirdness / adjustment / inequality, etc., but economic growth can paper over a lot of that. And even if it brings new societal ills and discomfort, AI in the short term is likely to counteract some existing ones—great stagnation, bureaucratic strangulation, vehicle deaths, etc.
The public cannot even bring itself to regulate much less economically useful vices with fewer tradeoffs (online sports gambling, shortform video brainrot, etc.). So I think it’s unlikely that AI will be regulated for any reason short of it becoming common knowledge / deeply-felt belief by nation-state leadership that ASI is in fact likely to cause swift and total human extinction.
I also think a hands-off approach to AI regulation is not going to be particularly off-putting or seem weird or antisocial to a large fraction of the country, namely the ~40% of the country that is right-leaning, and will tend to follow the beliefs of elite republicans who (mostly) still favor a light touch when it comes to any kind of government regulation. Regulating AI for any reason other than extinction is likely to become a relatively standard partisan issue, if it isn’t already.

Max H 29 Jun 2026 22:52 UTC
5 points
2
in reply to: Bentham's Bulldog’s comment on: Functional Decision Theory: Not Even Wrong, Also Wrong
“Adopt FDT” is probably too strong as a rephrasing; by “accept some flavor of FDT as true”, I intended to include anyone who one-boxes and / or cooperates in the prisoner’s dilemma because they disagree with you and academic-only philosophers on this:
One is called causal decision theory (CDT). I’m trying to be impartial, so I won’t tell you that it’s (probably) the correct view.
Which is definitely a minority view among rationalists, and (I am pretty confident) a minority view among researchers and philosophers at frontier AI labs and similar. cf. https://www.lesswrong.com/posts/n6wajkE3Tpfn6sd5j/christiano-decision-theory-excerpt, particularly:
Robert Wiblin: So you think it’s the case that when it comes to programming an AI, there’s actually a lot of agreement on what kind of decision theory it should be using in practice. Or at least, people agree that it needn’t be causal decision theory, even though philosophers think in some more fundamental sense causal decision theory is the right decision theory.
Paul Christiano: I think that’s right. I don’t know exactly what… I think philosophers don’t think that much about that question. But I think it’s not a tenable position, and if you really got into an argument with philosophers it wouldn’t be a tenable position that you should program your AI to use causal decision theory.
Of course there’s still a bunch of intra-rationalist and AI researcher disagreement on the specifics of FDT / UDT, e.g. a bunch of them probably disagree specifically because of updatelessness / EDT-with-tickle reasons, which are (debatably) not just “some flavor of FDT” as I said above.
IDK who exactly the most prominent people who have published specific thoughts on this are, but as a starting point the MacAskill post from 2019 that you cited got a bunch of substantive replies. I didn’t check all of their credentials / accomplishments and what they’re up to these days, but I suspect the real locus of our disagreement is more about the bounds of who qualifies as a philosopher / expert. I think “academic philosophy credentials” are not a good proxy for legible philosophy expertise (see sibling reply for more).