PeterMcCluskey

• PeterMcCluskey 5 Dec 2025 17:50 UTC

  2 points

  0

  on: Power Over­whelming: dis­sect­ing the $1.5T AI rev­enue shortfall

  Business models will change significantly. I speculated here about one likely change. Robotics-related business models will probably become important by 2030.

• PeterMcCluskey 4 Dec 2025 17:15 UTC

  2 points

  −8

  in reply to: lilkim2025’s comment on: Beat­ing China to ASI

  That’s not much of a proxy. I’m relying on my subjective impressions from many reports. A more precise phrasing of my claim is that I’ve seen numerous reports of what I consider to be open contempt for the rule of law among elected officials, but judges in newsworthy cases have almost always looked like they’re trying to take the law seriously.

  Some of my impressions come from a private mailing list where conservative lawyers have been expressing dismay at the Trump administration’s lack of interest in whether their actions could plausibly be defended in a court.

• PeterMcCluskey 4 Dec 2025 17:14 UTC

  5 points

  1

  in reply to: Wei Dai’s comment on: Beat­ing China to ASI

  Did you know that Deng approved the 1989 crackdown on Tiananmen protesters

  Yes, I’m aware that he did a few things that I consider evil. Wanting to keep his party in power is common enough among politicians that it’s not much evidence of psychopathy. His overall attitude toward independent thought was a least no worse than average for a political leader.

  A lot of what I have in mind is that Deng allowed more freedom than can readily explained by his self-interest, and Xi seems more Maoist than Deng.

  But I wouldn’t be surprised if you have better information about their personalities than do I.

  A darker interpretation is that the (subconscious, but more real or substantial in some sense) goals of nearly all humans are to gain power and status, and utopian ideologies are merely a tool for achieving this.

  The ideologies are partly a tool for that, but they have more effects on the wielder than a mere tool does. My biggest piece of evidence for that is the mostly peaceful collapse of the Soviet Union. I was quite surprised that the leaders didn’t use more force to suppress dissent.

Beat­ing China to ASI

• PeterMcCluskey 1 Dec 2025 2:55 UTC

  5 points

  0

  on: Se­ri­ous Flaws in CAST

  I am also somewhat dissatisfied with the basin of attraction metaphor, but for a slightly different reason.

  I am concerned that an AI that functions as mostly corrigible in environments that resemble the training environment will be less corrigible when the environment changes significantly.

  I’m guessing that a better metaphor would be based on evolutionary pressures. That would emphasize both the uncertainties about any given change, and the sensitivity to out-of-distribution environments.

  Maybe a metaphor about how cats are sometimes selected for being friendly to humans? Or the forces that led to the peacock’s tail?

• PeterMcCluskey 12 Nov 2025 20:43 UTC

  8 points

  2

  on: I Read Red Heart and I Heart It

  Corrigibility would clearly be a nice property

  Thinking of it as “a property” will mislead you about how Max’s strategy works. It needs to become the AI’s only top-level goal in order to work as Max imagines.

  It sure looks like AI growers know how to instill some goals in AIs. I’m confused as to why you think they don’t. Maybe you’re missing the part where the shards that want corrigibility are working to overcome any conflicting shards?

  I find it quite realistic that the AI growers would believe at the end of Red Heart that they probably had succeeded (I’ll guess that they ended up 80% confident?). That doesn’t tell us what probability we should put on it. I’m sure that in that situation Eliezer would still believe that the AI is likely not corrigible.

  I don’t know what year the novel is actually set in,

  It’s an alternate timeline where AI capabilities have progressed faster than ours, likely by a couple of years.

  Note this Manifold market on when the audiobook is released.

Red Heart

• PeterMcCluskey 2 Nov 2025 3:17 UTC

  7 points

  2

  in reply to: Algon’s comment on: Al­gon’s Shortform

  SemiAnalysis has a report (partly paywalled) here about a potential competitor to ASML.

• PeterMcCluskey 28 Oct 2025 16:06 UTC

  4 points

  0

  on: Heuris­tics for as­sess­ing how much of a bub­ble AI is in/​will be

  Novice investor participation is nowhere near what it was at the 2000 dot com peak. Current conditions look more like 1998. A bubble is probably coming, but there’s lots of room still for increased novice enthusiasm.

• PeterMcCluskey 23 Oct 2025 17:33 UTC

  4 points

  2

  in reply to: williawa’s comment on: Any cor­rigi­bil­ity naysay­ers out­side of MIRI?

  you can’t just train your ASI for corrigibility because it will sit and do nothing

  I’m confused. That doesn’t sound like what Max means by corrigibility. A corrigible ASI would respond to requests from its principal(s) as a subgoal of being corrigible, rather than just sit and do nothing.

  Or did you mean that you need to do some next-token training in order to get it to be smart enough for corrigibility training to be feasible? And that next-token training conflicts with corrigibility?

• PeterMcCluskey 21 Oct 2025 2:26 UTC

  5 points

  0

  on: Bub­ble, Bub­ble, Toil and Trouble

  Nothing importantly bearish happened in that month other than bullish deals

  What happened that made a bunch of people more bearish is that AI stocks went up a good deal, especially some of the lesser known ones.

  I’m unsure what exact time period you’re talking about, but here are some of the more interesting changes between Aug 29 and Oct 15:

  IREN +157%

  CLSK +145%

  APLD +136%

  INOD +118%

  NBIS +75%

  MU +61%

  AMD +47%

  If I thought AI was mostly hype, those kinds of near-panic buying would have convinced me to change my mind from “I don’t know” to “some of those are almost certainly in a bubble”. (Given my actual beliefs, I’m still quite bullish on MU, and weakly bullish on half of the others).

• PeterMcCluskey 13 Oct 2025 15:44 UTC

  4 points

  0

  on: Sublin­ear Utility in Pop­u­la­tion and other Un­com­mon Utilitarianism

  See here for a similar argument.

• PeterMcCluskey 12 Oct 2025 0:05 UTC

  46 points

  −1

  on: The Most Com­mon Bad Ar­gu­ment In Th­ese Parts

  A bunch of superforecasters were asked what their probability of an AI killing everyone was. They listed out the main ways in which an AI could kill everyone (pandemic, nuclear war, chemical weapons) and decided none of those would be particularly likely to work, for everyone.

  As someone who participated in that XPT tournament, that doesn’t match what I encountered. Most superforecasters didn’t list those methods when they focused on AI killing people. Instead, they tried to imagine how AI could differ enough from normal technology that it could attempt to start a nuclear war, and mostly came up with zero ways in which AI could be powerful enough that they should analyze specific ways in which it might kill people.

  I think Proof by Failure of Imagination describes that process better than does EFA.

• PeterMcCluskey 11 Oct 2025 21:47 UTC

  2 points

  −3

  in reply to: Czynski’s comment on: IABIED: Paradigm Con­fu­sion and Overconfidence

  The progress that I’m referring to is Max Harms’ work, which I tried to summarize here.

• PeterMcCluskey 9 Oct 2025 16:44 UTC

  4 points

  3

  in reply to: Davidmanheim’s comment on: IABIED: Paradigm Con­fu­sion and Overconfidence

  I guess “steering abilities” wasn’t quite the right way to describe what I meant.

  I’ll edit it to “desire to do anything other than predict”.

  I’m referring to the very simple strategy of leaving out the “then do that thing”.

  Training an AI to predict X normally doesn’t cause an AI to develop a desire to cause X.

• PeterMcCluskey 8 Oct 2025 20:32 UTC

  2 points

  0

  in reply to: Davidmanheim’s comment on: IABIED: Paradigm Con­fu­sion and Overconfidence

  begging the question.

  It seems that you want me to answer a question that I didn’t plan to answer. I’m trying to describe some ways in which I expect solutions to look different from what MIRI is looking for.

IABIED: Paradigm Con­fu­sion and Overconfidence

• PeterMcCluskey 30 Sep 2025 19:05 UTC

  4 points

  −10

  in reply to: StanislavKrym’s comment on: Why Cor­rigi­bil­ity is Hard and Im­por­tant (i.e. “Whence the high MIRI con­fi­dence in al­ign­ment difficulty?”)

  I’m referring mainly to MIRI’s confidence that the desire to preserve goals will conflict with corrigibility. There’s no such conflict if we avoid giving the AI terminal goals other than corrigibility.

  I’m also referring somewhat to MIRI’s belief that it’s hard to clarify what we mean by corrigibility. Max has made enough progress at clarifying what he means that it now looks like an engineering problem rather than a problem that needs a major theoretical breakthrough.

• PeterMcCluskey 30 Sep 2025 15:29 UTC

  4 points

  −11

  on: Why Cor­rigi­bil­ity is Hard and Im­por­tant (i.e. “Whence the high MIRI con­fi­dence in al­ign­ment difficulty?”)

  Max Harms’ work seems to discredit most of MIRI’s confidence. Why is there so little reaction to it?

Yet Another IABIED Review