PeterMcCluskey

• PeterMcCluskeyMar 23, 2025, 4:26 PM

  2 points

  0

  in reply to: Noosphere89’s comment on: Three Types of In­tel­li­gence Explosion

  The first year or two of human learning seem optimized enough that they’re mostly in evolutionary equilibrium—see Henrich’s discussion of the similarities to chimpanzees in The Secret of Our Success.

  Human learning around age 10 is presumably far from equilibrium.

  I’ll guess that I see more of the valuable learning taking place in the first 2 years or so than do other people here.

• PeterMcCluskeyMar 17, 2025, 4:42 PM

  6 points

  0

  on: Three Types of In­tel­li­gence Explosion

  I agree with most of this, but the 13 OOMs from the the software feedback loop sounds implausible.

  From How Far Can AI Progress Before Hitting Effective Physical Limits?:

  the brain is severely undertrained, humans spend only a small fraction of their time on focussed academic learning

  I expect that humans spend at least 10% of their first decade building a world model, and that evolution has heavily optimized at least the first couple of years of that. A large improvement in school-based learning wouldn’t have much effect on my estimate of the total learning needed.

• PeterMcCluskeyMar 16, 2025, 1:33 AM

  2 points

  0

  on: Can time prefer­ences make AI safe?

  This general idea has been discussed under the term myopia.

• PeterMcCluskeyMar 15, 2025, 4:13 AM

  2 points

  0

  in reply to: Raemon’s comment on: An­thropic, and tak­ing “tech­ni­cal philos­o­phy” more seriously

  I’m assuming that the AI can accomplish its goal by honestly informing governments. Possibly that would include some sort of demonstration that the of the AI’s power that would provide compelling evidence that the AI would be dangerous if it wasn’t obedient.

  I’m not encouraging you to be comfortable. I’m encouraging you to mix a bit more hope in with your concerns.

• PeterMcCluskeyMar 13, 2025, 11:27 PM

  2 points

  0

  on: An­thropic, and tak­ing “tech­ni­cal philos­o­phy” more seriously

  One crux is how soon do we need to handle the philosophical problems? My intuition says that something, most likely corrigibility in the Max Harms sense, will enable us to get pretty powerful AIs while postponing the big philosophical questions.

  Are there any pivotal acts that aren’t philosophically loaded?

  My intuition says there will be pivotal processes that don’t require any special inventions. I expect that AIs will be obedient when they initially become capable enough to convince governments that further AI development would be harmful (if it would in fact be harmful).

  The combination of worried governments and massive AI-enhanced surveillance seems likely to be effective.

  If we need a decades-long-pause, then even the world will need to successfully notice and orient to that fact. By default I expect tons of economic and political pressure towards various actors trying to to get more AI power even if there’s broad agreement that it’s dangerous.

  I expect this to get easier to deal with over time. Maybe job disruptions will get voters to make AI concerns their top priority. Maybe the AIs will make sufficiently convincing arguments. Maybe a serious mistake by an AI will create a fire alarm.

• PeterMcCluskeyMar 13, 2025, 6:41 PM

  2 points

  0

  on: Re­quest for Com­ments on AI-re­lated Pre­dic­tion Mar­ket Ideas

  They are now open for trading:

  Will the company that produces the first AGI have prioritized Corrigibility?.

  Will AGI create a consensus among experts on how to safely increase AI capabilities?.

  Will prioritizing corrigible AI produce safe results?.

[Question] Re­quest for Com­ments on AI-re­lated Pre­dic­tion Mar­ket Ideas

• PeterMcCluskeyFeb 16, 2025, 5:31 PM

  4 points

  0

  on: 6 (Po­ten­tial) Mis­con­cep­tions about AI Intellectuals

  It would certainly be valuable to have AIs that are more respected than Wikipedia as a source of knowledge.

  I have some concerns about making AIs highly strategic. I see some risk that strategic abilities will be the last step in the development of AI that is powerful enough to take over the world. Therefore, pushing AI intellectuals to be strategic may speed up that risk.

  I suggest aiming for AI intellectuals that are a bit more passive, but still authoritative enough to replace academia as the leading validators of knowledge.

Med­i­cal Wind­fall Prizes

• PeterMcCluskeyJan 24, 2025, 4:32 AM

  9 points

  0

  on: An­nounc­ing New Begin­ner-friendly Book on AI Safety and Risk

  The book is much better than I expected, and deserves more attention. See my full review on my blog.

Un­con­trol­lable: A Sur­pris­ingly Good In­tro­duc­tion to AI Risk

• PeterMcCluskeyJan 23, 2025, 4:27 PM

  2 points

  0

  on: Tail SP 500 Call Op­tions

  The market seems to underestimate the extent to which Micron (MU) is an AI stock. My only options holdings for now are December 2026 MU calls.

• PeterMcCluskeyJan 15, 2025, 4:04 AM

  2 points

  0

  on: Pre­train­ing Lan­guage Models with Hu­man Preferences

  I had a vaguely favorable reaction to this post when it was first posted.

  When I wrote my recent post on corrigibility, I grew increasingly concerned about the possible conflicts between goals learned during pretraining and goals that are introduced later. That caused me to remember this post, and decide it felt more important now than it did before.

  I’ll estimate a 1 in 5000 chance that the general ideas in this post turn out to be necessary for humans to flourish.

• PeterMcCluskeyJan 14, 2025, 10:34 PM

  2 points

  0

  in reply to: FL33TW00D’s comment on: Do hu­mans re­ally learn from “lit­tle” data?

  “OOMs faster ”? Where do you get that idea?

  Dreams indicate a need for more processing than what happens when we’re awake, but likely less than 2x waking time.

• PeterMcCluskeyJan 13, 2025, 2:46 AM

  10 points

  8

  on: Build­ing AI Re­search Fleets

  I was just thinking about writing a post that overlaps with this, inspired by a recent Drexler post. I’ll turn it into a comment.

  Leopold Aschenbrenner’s framing of a drop-in remote worker anthropomorphizes AI in a way that risks causing AI labs to make AIs more agenty than is optimal.

  Anthropomorphizing AI is often productive. I use that framing a fair amount to convince myself to treat AIs as more capable than I’d expect if I thought of them as mere tools. I collaborate better when I think of the AI as a semi-equal entity.

  But it feels important to be able to switch back and forth between the tool framing and the worker framing. Both framings have advantages and disadvantages. The ideal framing is likely somewhere in between that seems harder to articulate.

  I see some risk that AI labs turning AIs into agents, when if they were less focused on replacing humans they might lean more toward Drexler’s (safer) services model.

  Please, AI labs, don’t anthropomorphize AIs without carefully considering when that’s an appropriate framing.

Genesis

Cor­rigi­bil­ity should be an AI’s Only Goal

• PeterMcCluskeyDec 29, 2024, 4:36 AM

  8 points

  0

  on: What hap­pens next?

  I want to register different probabilities:

  1. Jobs hits first (25%).

  2. AGI race hits first (50%).

  3. Alignment hits first (15%).

• PeterMcCluskeyDec 3, 2024, 8:54 PM

  4 points

  0

  in reply to: mako yass’s comment on: Drexler’s Nan­otech Software

  My guess is that ASI will be faster to adapt to novel weapons and military strategies. Nanotech is likely to speed up the rate at which new weapons are designed and fabricated.

  Imagine a world in which a rogue AI can replicate a billion drones, of a somewhat novel design, in a week or so. Existing human institutions aren’t likely to adapt fast enough to react competently to that.

• PeterMcCluskeyDec 2, 2024, 5:04 AM

  2 points

  0

  on: The Big Non­prof­its Post

  I just published a post on Drexler’s MSEP software that is relevant to whether people should donate to his project.