_will_

• _will_ 28 Mar 2026 1:39 UTC

  3 points

  0

  in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform

  Strongly agree, especially with your latter point 3. Fwiw, I used to work at Metaculus. Copy-pasting something I wrote after I left (which was in response to a Slack post about this study):

  Forgive my cynicism/​jadedness, but IMO this is just another example of superforecasters not understanding alignment or the intelligence explosion, and putting forward forecasts that make the future look kinda normal. My favourite explanation for what is going on, here, is Nuño Sempere’s ‘Excellent forecasters and Superforecasters™ have an imperfect fit for long-term questions’.

  (~This is essentially why I lost faith in forecasting as a cause area, and now work on other things.)

  [person replied to me saying that “superforecasters not understanding alignment or the intelligence explosion” are overly specific hypotheses—superforecasters may simply disagree with, e.g., alignment being a serious problem]

  I agree that they’re specific hypotheses, but they are borne out by the evidence I’ve observed. Like, in my previous role, I spent a lot of time talking to superforecasters/​pro forecasters, and when I’d probe them on their p(doom) or their timelines, I’d often find ~zero awareness of the arguments for why AI might be transformative or an x-risk, and also—kinda bafflingly—zero interest in grappling with these arguments.

  Disagreement, for me, means ‘they understood what I was talking about, but had models/​(counter)arguments for why they weren’t expecting AI to be transformative, etc.’ That’s not what I’ve observed.

Long-term risks from ide­olog­i­cal fanaticism

• _will_ 20 Dec 2025 1:18 UTC

  5 points

  0

  in reply to: habryka’s comment on: Toss a bit­coin to your Light­cone – LW + Lighthaven’s 2026 fundraiser

  Tangent:

  If the Deepmind alignment team wanted to host something here I would probably give them a substantial discount! If the Anthropic alignment team wanted to host something here I would probably charge them a small but not enormous tax.

  Are you saying that you believe Google Deepmind’s alignment team is doing better work than Anthropic’s alignment team? If so, that’s a noteworthy update for me. (My vague impression was that Anthropic is the least bad AI lab, but perhaps I’ve not paid enough attention.)

• _will_ 28 Jan 2025 13:43 UTC

  8 points

  0

  on: Six Plau­si­ble Meta-Eth­i­cal Alternatives

  Great post! I find myself coming back to it—especially possibility 5—as I sit here in 2025 thinking/​worrying about AI philosophical competence and the long reflection.

  On 6,^[1] I’m curious if you’ve seen this paper by Joar Skalse? It begins:

  I present an argument and a general schema which can be used to construct a problem case for any decision theory, in a way that could be taken to show that one cannot formulate a decision theory that is never outperformed by any other decision theory.

  1. ^

     Pasting here for easy reference (emphasis my own):

     6. There aren’t any normative facts at all, including facts about what is rational. For example, it turns out there is no one decision theory that does better than every other decision theory in every situation, and there is no obvious or widely-agreed-upon way to determine which one “wins” overall.

• _will_ 11 Jan 2025 10:44 UTC

  12 points

  2

  in reply to: avturchin’s comment on: john­swent­worth’s Shortform

  See also ‘The Main Sources of AI Risk?’ by Wei Dai and Daniel Kokotajlo, which puts forward 35 routes to catastrophe (most of which are disjunctive). (Note that many of the routes involve something other than intent alignment going wrong.)

• _will_ 4 Jan 2025 16:50 UTC

  8 points

  7

  in reply to: Quinn’s comment on: The Field of AI Align­ment: A Post­mortem, and What To Do About It

  Any chance you have a link to this tweet? (I just tried control+f’ing through @Richard’s tweets over the past 5 months, but couldn’t find it.)

• _will_ 6 Dec 2024 11:36 UTC

  5 points

  0

  in reply to: RobertM’s comment on: RobertM’s Shortform

  On your second point, I think that MacAskill and Ord were more saying “It would be worth it to spend thousands of years figuring out moral philosophy /​ figuring out what to do with the cosmos, if that’s how long it takes to be ~sure we’ve reached the ‘correct’ answer before locking things in, on account of the astronomical waste argument” than “I literally predict it will take today-humans thousands of years to figure out moral philosophy, even if we make a serious and coordinated effort to do so.” Somewhat relatedly, quoting from the ‘Long Reflection Reading List’ I wrote earlier this year (fn. 4):

  Original discussion of the long reflection indicated that it could be a lengthy process of 10,000 years or more. More recent discussion I’m aware of, which is nonpublic, hence no corresponding reading, i) takes seriously the possibility that the long reflection could last just weeks rather than years or millenia, and ii) notes that wall clock time is probably not the most useful way to think about the length of reflection, given that the reflection process, if it happens at all, will likely involve many superfast AIs doing the bulk of the cognitive labor.

  On your first point, I continue to be curious about your perspective. I basically agree with the following (written by Zach Stein-Perlman), but, based on what you said in your parentheses, it sounds like you view it as a bad plan?

  The outline of the best [post-AGI] plan I’ve heard is build human-obsoleting AIs which are sufficiently aligned/​trustworthy that we can safely defer^[1] to them (before building wildly superintelligent AI). Assume it will take 5-10 years after AGI to build such systems and give them sufficient time. To buy time (or: avoid being rushed by other AI projects^[2]), inform the US government and convince it to enforce nobody builds wildly superintelligent AI for a while (and likely limit AGI weights to allied projects with excellent security and control).

  (I could be off, but it sounds like either you expect solving AI philosophical competence to come pretty much hand in hand with solving intent alignment (because you see them as similar technical problems?), or you expect not solving AI philosophical competence (while having solved intent alignment) to lead to catastrophe (thus putting us outside the worlds in which x-risks are reliably ‘solved’ for), perhaps in the way Wei Dai has talked about?)

  1. ^

     We don’t need these human-obsoleting AIs to be able to implement CEV. We want to be able to defer to them on tricky wisdom-loaded questions like what should we do about the overall AI situation? They can ask us questions as needed.

  2. ^

     To avoid being rushed by your own AI project, you also have to ensure that your AI can’t be stolen and can’t escape, so you have to implement excellent security and control.

• _will_ 30 Oct 2024 4:40 UTC

  3 points

  2

  in reply to: habryka’s comment on: MIRI 2024 Com­mu­ni­ca­tions Strategy

  Thanks, that’s helpful!

  (Fwiw, I don’t find the ‘caring a tiny bit’ story very reassuring, for the same reasons as Wei Dai, although I do find the acausal trade story for why humans might be left with Earth somewhat heartening. (I’m assuming that by ‘game-theoretic reasons’ you mean acausal trade.))

• _will_ 30 Oct 2024 4:08 UTC

  1 point

  0

  in reply to: Daniel Kokotajlo’s comment on: MIRI 2024 Com­mu­ni­ca­tions Strategy

  I don’t think [AGI/​ASI] literally killing everyone is the most likely outcome

  Huh, I was surprised to read this. I’ve imbibed a non-trivial fraction of your posts and comments here on LessWrong, and, before reading the above, my shoulder Daniel definitely saw extinction as the most likely existential catastrophe.

  If you have the time, I’d be very interested to hear what you do think is the most likely outcome. (It’s very possible that you have written about this before and I missed it—my bad, if so.)

• _will_ 28 Oct 2024 14:37 UTC

  4 points

  2

  in reply to: Ben Pace’s comment on: john­swent­worth’s Shortform

  Hmm, the ‘making friends’ part seems the most important (since there are ways to share new information you’ve learned, or solve problems, beyond conversation), but it also seems a bit circular. Like, if the reason for making friends is to hang out and have good conversations(?), but one has little interest in having conversations, then doesn’t one have little reason to make friends in the first place, and therefore little reason to ‘git gud’ at the conversation game?

Ev­i­den­tial Co­op­er­a­tion in Large Wor­lds: Po­ten­tial Ob­jec­tions & FAQ

Everett branches, in­ter-light cone trade and other alien mat­ters: Ap­pendix to “An ECL ex­plainer”

Co­op­er­at­ing with aliens and AGIs: An ECL explainer

• _will_ 31 Aug 2023 19:53 UTC

  3 points

  −5

  in reply to: Linch’s comment on: LTFF and EAIF are un­usu­ally fund­ing-con­strained right now

  So basically I don’t think it’s possible to do robustly positive actions in longtermism with high (>70%? >60%?) probability of being net positive for the long-term future

  This seems like an important point, and it’s one I’ve not heard before. (At least, not outside of cluelessness or specific concerns around AI safety speeding up capabilities; I’m pretty sure that most EAs I know have ~100% confidence that what they’re doing is net positive for the long-term future.)

  I’m super interested in how you might have arrived at this belief: would you be able to elaborate a little? For instance, is there a theoretical argument going on here, like a weak form of cluelessness? Or is it more empirical, for example, did you get here through evaluating a bunch of grants and noticing that even the best seem to carry 30-ish percent downside risk? Something else?

• _will_ 3 Jun 2023 22:49 UTC

  6 points

  7

  on: How to have Poly­geni­cally Screened Children

  “GeneSmith”… the pun just landed with me. nice.

• _will_ 25 May 2023 15:40 UTC

  3 points

  1

  on: Open Thread With Ex­per­i­men­tal Fea­ture: Reactions

  Very nitpicky (sorry): it’d be nice if the capitalization to the epistemic status reactions was consistent. Currently, some are in title case, for example “Too Harsh” and “Hits the Mark”, while others are in sentence case, like “Key insight” and “Missed the point”. The autistic part of me finds this upsetting.

• _will_ 17 May 2023 12:53 UTC

  4 points

  0

  in reply to: Nathan Helm-Burger’s comment on: AI Risk & Policy Forecasts

  Thanks for this comment. I don’t have much to add, other than: have you considered fleshing out and writing up this scenario in a style similar to “What 2026 looks like”?

• _will_ 17 May 2023 12:42 UTC

  4 points

  0

  in reply to: SteveZ’s comment on: AI Risk & Policy Forecasts

  Thanks for this question.

  Firstly, I agree with you that firmware-based monitoring and compute capacity restrictions would require similar amounts of political will to happen. Then, in terms of technical challenges, I remember one of the forecasters saying they believe that “usage-tracking firmware updates being rolled out to 95% of all chips covered by the 2022 US export controls before 2028” is 90% likely to be physically possible, and 70% likely to be logistically possible. (I was surprised at how high these stated percentages were, but I didn’t have time then to probe them on why exactly they were at these percentages—I may do so at the next workshop.)

  Assuming the technical challenges of compute capacity restrictions aren’t significant, fixing compute capacity restrictions at 15% likely, and applying the following crude calculation:

  P(firmware) = P(compute) x P(firmware technical challenges are met)

  = 0.15 x (0.9 x 0.7) = 0.15 x 0.63 = 0.0945 ~ 9%

  9% is a little above the reported 7%, which I take as meaning that the other forecasters on this question believe the firmware technical challenges are a little, but not massively, harder than the 90%–70% breakdown given above.

AI Risk & Policy Fore­casts from Me­tac­u­lus & FLI’s AI Path­ways Workshop

• _will_ 7 May 2023 15:16 UTC

  6 points

  7

  in reply to: Raemon’s comment on: Why aren’t more of us work­ing to pre­vent AI hell?

  There is a vibe that I often get from suffering focused people, which is a combo of

  a) seeming to be actively stuck in some kind of anxiety loop, preoccupied with hell in a way that seems more pathological to me than well-reasoned.

  b) something about their writing and vibe feels generally off,

  ...

  I agree that this seems to be the case with LessWrong users who engage in suffering-related topics like quantum immortality and Roko’s basilisk. However, I don’t think any(?) of these users are/​have been professional s-risk researchers; the few (three, iirc) s-risk researchers I’ve talked to in real life did not give off this kind of vibe at all.