Ben Pace

• Ben Pace 22 Oct 2025 6:14 UTC

  4 points

  2

  in reply to: Daniel Kokotajlo’s comment on: leogao’s Shortform

  I am not really clear that I should be worried on the scale of decades? If we’re doing a calculation of expected future years of a flourishing technologically mature civilization, slowing down for 1,000 years here in order to increase the chance of success by like 1 percentage point is totally worth it in expectation.

  Given this, it seems plausible to me that one should rather spend 200 years trying to improve civilizational wisdom and decision-making rather than instead attempt to specifically just unlock regulation on AI (of course the specifics here are cruxy).

• Ben Pace 22 Oct 2025 1:56 UTC

  2 points

  0

  on: Why the Strug­gle for Safe AI Must Be Political

  Mod note. I believe this post fails to pass our LLM Writing Policy for LessWrong, so I have delisted the post, meaning while it is accessible via link, it does not naturally appear on the frontpage of LessWrong or on your user profile (though it’s still in the sequence). In future please make sure that you do not publish so much LLM-written content! Or else we will remove your accounts’ posting permissions.

Lighthaven Se­quences Read­ing Group #54 (Tues­day 10/​21)

• Ben Pace 19 Oct 2025 19:04 UTC

  13 points

  0

  in reply to: romeostevensit’s comment on: romeosteven­sit’s Shortform

  Pretty similar to the “write down next actions, not end state” from Getting Things Done.

• Ben Pace 17 Oct 2025 5:33 UTC

  12 points

  6

  on: The “Length” of “Hori­zons”

  Curated. This helped me a lot in thinking about what the paper really means. It’s also a paper that’s affecting a lot of people’s thinking about AI, so it’s worth highlighting disagreements.

Lighthaven Se­quences Read­ing Group #53 (Tues­day 10/​14)

• Ben Pace 12 Oct 2025 20:41 UTC

  38 points

  7

  in reply to: Katalina Hernandez’s comment on: Katal­ina Her­nan­dez’s Shortform

  I thought this meant she endorsed the title as her beliefs, so in case anyone else was confused, she has endorsed the book as worth reading, but not (to my knowledge) endorsed its thesis as “what I believe”.

  Here’s what she writes:

  Although, in general, I disagree with catastrophic framings of AI risk (which have been exploited by AI CEOs to increase interest in their products, as I recently wrote in my newsletter), the AI safety debate is an important one, and it concerns all of us.
  
  There are differing opinions on the current path of AI development and its possible futures. There are also various gray zones and unanswered questions on possible ways to mitigate risk and avoid harm.
  
  Yudkowsky has been researching AI alignment for over 20 years, and together with Soares, he has built a strong argument for why AI safety concerns are urgent and why action is needed now. Whether you agree with their tone or not, their book is worth reading.

  Seems that she finds Eliezer+Nate credible in their concerns because they are not AI company CEOs but have been working on the problem for 2 decades.

• Ben Pace 12 Oct 2025 5:30 UTC

  6 points

  2

  in reply to: Zack_M_Davis’s comment on: The Re­la­tion­ship Between So­cial Pu­n­ish­ment and Shared Maps

  I admit I am a bit confused about the thesis here… I get that accurate behavioral accounting is sometimes tightly related to social punishment such that the attempt to give or defend oneself from punishment provides incentive to lie about the behavior (and attempts to describe the behavior have direct implications for punishment).

  But are you further claiming that that all social punishment is identical^[1] to truth-claims about other things (i.e. “reasons for the punishment”)? This seems like an ideal that I aspire to, but not how most people relate to social punishment, where social ostracism can sometimes simply be a matter of fashion or personal preference.

  Personally I use phrases like “X is lame” or “X isn’t cool” to intentionally and explicitly set the status of things. I endeavor to always have good reasons for why and to provide them (or at least to have them ready if requested), but the move itself does not require justification in order to successfully communicate that something is having its status lowered or is something that I oppose. People would often happily just accept the status-claims without reasons, similar to learning what is currently ‘in fashion’.

  1. ^

     On reflection I don’t quite mean identical to, but something more like “Is a deterministic function of truth-claims about good/​bad behavior, taking that-and-only-that as input”.

• Ben Pace 10 Oct 2025 19:42 UTC

  6 points

  0

  in reply to: cubefox’s comment on: Buck’s Shortform

  I think of “It seems” as coming from the view from nowhere. I say “It seems to me” to own it, unless I mean to make the active claim that I believe it will seem this way to everyone.

  “Arguably” feels super weak to me. I would only use it if I actively wanted to distance myself from a view. Almost anything is ‘arguable’ in principle.

• Ben Pace 9 Oct 2025 21:49 UTC

  31 points

  20

  in reply to: Buck’s comment on: Buck’s Shortform

  Probably this isn’t the exclusive reason, but typically I use “I think” whenever I want to rule out the interpretation that I am implying we all agree on my claim. If I say “It was a mistake for you to paint this room yellow” this is more natural if you agree with me; if I say “I think it was a mistake for you to paint this room yellow” this is more natural if I’m informing you of my opinion but I expect you to disagree.

  This is not a universal rule, and fwiw I do think there’s something good about clear and simple writing that cuts out all the probably-unnecessary qualifiers, but I think this is a common case where I find it worth adding it in.

• Ben Pace 7 Oct 2025 19:12 UTC

  4 points

  0

  in reply to: Adam Scholl’s comment on: Cole Wyeth’s Shortform

  Datapoint: I’m currently setting up a recording studio at Lighthaven, and I am using them all the time to get guides for things like “how to change a setting on this camera” or “how to use this microphone” or “how to use this recording software”.

  Yes, they confabulate menus and things a lot, but as long as I keep uploading photos of what I actually see, they know the basics much better than me (e.g. what bit rate to set the video vs the audio, where to look to kill the random white noise input I’m getting, etc).

  I’d say they confabulate like 50% of the time but that they’re still a much more effective search engine for me than google, and can read the manual much faster than me. My guess is I simply couldn’t do some of the projects I’m doing without them.

• Ben Pace 7 Oct 2025 19:02 UTC

  2 points

  0

  in reply to: gustaf’s comment on: Ben Pace’s Short­form Feed

  Awesome, thanks so much gustaf!

  I’ve re-arranged the order a little bit to have some of the more common ones first.

  Yeah, I have been thinking about linking it from the reacts menu. Will think about that next time I ship some changes. (I’ll be more inclined to do that if there are more explanations and links and culture and stuff.)

Lighthaven Se­quences Read­ing Group #52 (Tues­day 10/​7)

• Ben Pace 6 Oct 2025 5:19 UTC

  4 points

  0

  on: Notes on fatal­ities from AI takeover

  Curated. This is a topic that is getting a ton of attention at the minute, and the post does a decent job at laying out relevant considerations and summarizing some prior debate.

  (I disagree with the conclusion but it seems positive to me to curate something with a different conclusion than the recent book on the issue.)

• Ben Pace 5 Oct 2025 23:07 UTC

  9 points

  0

  in reply to: ryan_greenblatt’s comment on: Notes on fatal­ities from AI takeover

  Link to Nate’s comments.

• Ben Pace 4 Oct 2025 6:19 UTC

  2 points

  0

  in reply to: Mikhail Samin’s comment on: Mikhail Samin’s Shortform

  Further follow-up: Guido Reichstadter wraps up after 30 days. Impressively long! And a bit longer than I’d guessed.

• Ben Pace 3 Oct 2025 22:11 UTC

  −3 points

  28

  in reply to: wdmacaskill’s comment on: A Re­ply to MacAskill on “If Any­one Builds It, Every­one Dies”

  I think I’m particularly triggered by all this because of a conversation I had last year with someone who takes AI takeover risk very seriously and could double AI safety philanthropy if they wanted to. I was arguing they should start funding AI safety, but the conversation was a total misfire because they conflated “AI safety” with “stop AI development”: their view was that that will never happen, and they were actively annoyed that they were hearing what they considered to be such a dumb idea. My guess was that EY’s TIME article was a big factor there.

  Mandatory check: was this billionaire a sociopath who made their money unethically or illegally (perhaps through crypto), like the last time you persuaded someone in this position to put tons of their philanthropy into AI safety?

  (Perhaps you can show that they weren’t, but given your atrocious track record these datapoints shouldn’t really be taken seriously without double-checking.)

  (As a suggestion, you could DM the name to me or anyone in this thread and have them report back their impression of whether the person is a crook or obviously unethical, without releasing the identity widely.)

• Ben Pace 3 Oct 2025 18:10 UTC

  4 points

  2

  in reply to: 1a3orn’s comment on: Why Cor­rigi­bil­ity is Hard and Im­por­tant (i.e. “Whence the high MIRI con­fi­dence in al­ign­ment difficulty?”)

  Yeah but “this theory sometimes correctly predicts the economy in a way no other theory has been capable of, and sometimes gets things totally wrong, and this theory says AI will cause extinction” is not unjustly privileging the hypothesis. It’s a mistake to say that theory “just isn’t very informative” when it’s been incredibly informative on lots of issues, even while mistaken on others.

• Ben Pace 3 Oct 2025 17:08 UTC

  6 points

  4

  in reply to: 1a3orn’s comment on: Why Cor­rigi­bil­ity is Hard and Im­por­tant (i.e. “Whence the high MIRI con­fi­dence in al­ign­ment difficulty?”)

  But it’s made tons of accurate predictions in game theory and microeconomics?

• Ben Pace 2 Oct 2025 18:59 UTC

  9 points

  2

  on: Eth­i­cal De­sign Patterns

  Curated! Thanks for the post.

  I am having a hard time explaining why I’m curating this. I think for me this post is helping me move ethics from some mystical thing, into an engineering/​design problem, which is how I think a lot of it should be thought of. I recently having been reading a book on the classic virtues, and it makes them sound so dreary and uncompelling; this account seems more true and healthier.

  I think the examples were great and really illustrated this well. But I agree with a comment (from Wei Dai) that this post is weak on proving that the presence or lack of good ethical design patterns was critical in the success or failure of different systems, rather than other factors.

  I hope to see more writing about concrete cases, and more work to help turn our consequentialist analyses into ethical design patterns /​ folk ethics.