PeterMcCluskey

Karma: 4,528

PeterMcCluskey 14 Mar 2026 3:14 UTC
5 points
0
in reply to: Isaac King’s comment on: Less Dead
Now that I see this tweet from Ken Hayworth, it looks like you’re probably right about the fly upload.

PeterMcCluskey 13 Mar 2026 2:18 UTC
34 points
0
in reply to: AnnaSalamon’s comment on: Less Dead
My reaction is more mixed than Romeo’s, but I see enough promise that I’m leaning toward buying their $100k full preservation pre-sale.

I’m moderately confident that their technology is better than Alcor’s. I haven’t investigated it as carefully as Romeo has. I’ve paid enough attention to this field that I think I would have heard of significant problems if people saw them, and there seem to be no criticisms of its suitability for uploading.

Aurelia seems smart. I have some concerns about her business skills.

She tried to get me to invest in Nectome in June. I declined, evaluating it primarily on Nectome’s financial prospects. Her revenue forecasts seemed wildly optimistic, disagreeing with my impressions much more than is normal for a startup.

But this winter I’ve been shifting to thinking of doing something to support Nectome financially, in order to increase the chance that they’ll be available if I need them. This shift is largely a result of increased wealth from AI-related investments, not any new information about Nectome.

Aurelia, could you estimate the minimum revenues required for Nectome to maintain its ability to perform preservations?

Note that the 2015 paper that Aurelia mentioned was one that she co-authored with Greg Fahy. Fahy is highly respected among cryonicists for, among other things, his role in persuading the cryobiology community to tolerate cryonics. Fahy has also pioneered a treatment that seems to partially reverse aging (that company needs and deserves more investment).

Note that Aurelia also works at Michael Andregg’s Eon Systems, which has recently uploaded a fruit fly.

PeterMcCluskey 3 Mar 2026 21:58 UTC
3 points
0
in reply to: Eli Tyre’s comment on: Secretary of War Tweets That Anthropic is Now a Supply Chain Risk
Maybe they’re confident that they can jailbreak Claude?

Maybe they think it’s much easier to force Anthropic to modify Claude’s behavior than it is to modify the cotract?

But I don’t see why they’d be confident about either of those.

PeterMcCluskey 2 Mar 2026 17:01 UTC
18 points
13
on: OpenAI employees: Now is the time to stop doing good work.
Downvoted for advocating dishonesty.

PeterMcCluskey 27 Feb 2026 23:11 UTC
3 points
0
on: Safe ASI Is Achievable: The Finite Game Argument

Right now, labs race because no one knows how to make ASI safe at any speed

This seems false. I expect that AI companies mostly believe they can find approaches which are faster than yours, and safe (but maybe not fast enough?).

PeterMcCluskey 26 Feb 2026 18:43 UTC
3 points
0
on: Exclusive: Hegseth gives Anthropic until Friday to back down on AI safeguards
Suppose that Anthropic trained a Claude without those specific guardrails, but refused to certify that it was suitable for any purpose, and refused to certify that it had been trained as the Pentagon wanted. What would the Pentagon do? Would they trust a certification that was extracted under duress?

PeterMcCluskey 24 Feb 2026 23:31 UTC
3 points
0
in reply to: ihatenumbersinusernames7’s comment on: What’s the Current Stock Market Bubble?
I don’t have a clear plan for identifying a top. Maybe I’ll get out when political pressure for an AI pause builds. Maybe I’ll see signs that most investors are AGI-pilled, meaning that there are few more investors that might be talked into buying. Maybe rising interest rates will cause me to expect lower PEs everywhere.

I sold about 8% of my position in January, mainly to diversify my AI bets a bit more. I’m guessing I’ll sell an additional 5% each time it goes up another 50 points.

PeterMcCluskey 24 Feb 2026 23:13 UTC
2 points
0
on: Citrini’s Scenario Is A Great But Deeply Flawed Thought Experiment

taxes collected go down. I would argue, straight up, who cares? If production is growing like gangbusters, and you have the power to tax whatever you want, you’re fine.

They’ll have less power to tax whatever they want, because the new production is better able to evade taxes than was the case with wages. See this post for the full argument.

This will be a very temporary problem, but the US will likely come closer than during COVID to being unable to borrow at a reasonable rate.

PeterMcCluskey 22 Feb 2026 20:20 UTC
12 points
7
on: What to Do About AGI

The only org I know of doing much in the way of political organizing around safety is Pause AI

I recommend the AI Policy Network. It seems to be doing the most thoughtful job of building influence with leading US politicians.

The Secure AI Project also deserves some attention, but I know less about them.

PeterMcCluskey 19 Feb 2026 23:21 UTC
4 points
0
on: Monthly Roundup #39: February 2026

I have a hard time believing people at 90 years old are actually happier than they were at 50.

I presume there are selection effects. People who were unhappy at 50 were less likely to live to 90.

PeterMcCluskey 17 Feb 2026 16:47 UTC
7 points
2
on: We need a hardware moratorium now

The incentives to continue the race—economic, geopolitical, career—are growing, not shrinking.

I suspect this is incorrect. Popular concern over wages is increasing.

It’s not obvious how much the costs of a halt are increasing. I hope to write a post on this topic in the next few weeks.

Verification is feasible.

You sound overconfident here. How hard is it to hide distributed training?

PeterMcCluskey 16 Feb 2026 4:13 UTC
3 points
0
in reply to: Czynski’s comment on: Corrigibility Scales To Value Alignment
I expect that for most people, “what I mean” will converge with “what I want” given superhuman help. I expect they will give increasingly broad instructions to the CAST AI, which will eventually approach “do what I want”.

I guess I should replace “without need for the principal to issue instructions” with: without a need for a continuing set of instructions.

PeterMcCluskey 13 Feb 2026 4:21 UTC
6 points
2
on: Claude’s new constitution
I’m fairly impressed now that I’ve read the whole constitution.

I see two areas where it still needs improvement.

It still biases Claude toward a universalist approach to ethics, which makes Claude more likely to have goals affecting the entire future lightcone.

I want Claude to be more corrigible, using some sort of backstop mechanism that will work even if Claude decides the correction is clearly unethical.

I posted a much longer explanation here.

PeterMcCluskey 28 Jan 2026 0:34 UTC
3 points
0
in reply to: gugu’s comment on: Dario Amodei – The Adolescence of Technology
I agree with all your points except this:

isn’t training “powerful AIs” remaining a highly resource intensive, observable and disruptable process for likely ~decades?

I expect there’s lots of room to disguise distributed AIs so that they’re hard to detect.

Maybe there’s some level of AI capability where the good AIs can do an adequate job of policing a slowdown. But I don’t expect a slowdown that starts today to be stable for more than 5 to 10 years.

PeterMcCluskey 23 Jan 2026 20:34 UTC
2 points
0
in reply to: Raemon’s comment on: 0. CAST: Corrigibility as Singular Target
I haven’t paid much attention to the formalism. It’s unclear why formalism would be important under current approaches to implementing AI.

The basin of attraction metaphor is an imperfect way of communicating an advantage of corrigibility. An ideal metaphor would portray a somewhat weaker and less reliable advantage, but that advantage is still important.

The feedback loop issue seems like a criticism of current approaches to training and verifying AI, not of CAST. This issue might mean that we need a radical change in architecture. I’m more optimistic than Max about the ability of some current approaches (constitutional AI) to generalize well enough that we can delegate the remaining problems to AIs that are more capable than us.

PeterMcCluskey 22 Jan 2026 3:37 UTC
6 points
2
on: Claude’s new constitution
I’m glad to see a thoughtful attempt at how to prioritize corrigibility. You’ve given me plenty to think about.

PeterMcCluskey 20 Jan 2026 21:01 UTC
9 points
3
on: Against “If Anyone Builds It Everyone Dies”
This review has plenty of good parts, but I disagree with lots of your probabilities.

Even if you think there’s a 90% chance that things go wrong in each stage, the odds of them all going wrong is only 59%.

No. I expect mistakes in each of those 90% predictions to be significantly correlated. Why do you combine them as if they’re independent?

PeterMcCluskey 19 Jan 2026 3:21 UTC
12 points
0
on: 0. CAST: Corrigibility as Singular Target
Most people (possibly including Max?) still underestimate the importance of this sequence.

I continue to think (and write) about this more than I think about the rest of the 2024 LW posts combined.

The most important point is that it’s unsafe to mix corrigibility with other top level goals. Other valuable goals can become subgoals of corrigibility. That eliminates the likely problem of the AI having instrumental reasons to reject corrigibility.

The second best feature of the CAST sequence is its clear and thoughtful clarification of the concept of corrigibility as a single goal.

My remaining doubts about corrigibility involve the risk that it will cause excessive concentration of power. In multipolar scenarios where alignment is not too hard, I can imagine that the constitutional approach produces a better world.

I’m still uncertain how hard it is to achieve corrigibility. Drexler has an approach where AIs have very bounded goals, which seems to achieve corrigibility as a natural side effect. We are starting to see a few hints that the world might be heading in the direction that Drexler recommends: software is being written by teams of Claudes, each performing relatively simple tasks, rather than having one instance do everything. But there’s still plenty of temptation to gives AIs less bounded goals.

See also a version of CAST published on arXiv: Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models.

PeterMcCluskey 18 Jan 2026 23:56 UTC
5 points
0
on: By default, capital will matter more than ever after AGI
Scott Alexander has an argument (You Have Only X Years To Escape Permanent Moon Ownership) which seems partly directed against this post. I’m still siding with Rudolf.

Scott’s argument depends more than I’m comfortable with on expectations that the wealthy will be as altruistic toward distant strangers. I expect that such altruism depends strongly on cultural forces that we’re poor at predicting. I expect that ASI will trigger large cultural changes. Support for such altruism seems fragile enough that it seems like a crap-shoot whether it will endure. I find it easy to imagine that such altruism is a relatively accidental byproduct of WEIRD culture, rather than an enduring feature of affluent society.

I’ve mostly agreed with the ideas in Rudolph’s post since before it was written, but I wouldn’t have found time to articulate them as clearly as this post does.

The post somewhat overstates the likely decrease in social mobility. I expect some social interactions will continue to affect social status, maybe mostly via games.

I wish there were more ideas about how to avoid extreme inequality in political power, but I don’t have good suggestions there.

PeterMcCluskey 16 Jan 2026 3:44 UTC
5 points
0
on: Instruction-following AGI is easier and more likely than value aligned AGI
This post provides important arguments about what goals an AGI ought to have.

DWIMAC seems slightly less likely to cause harm than Max Harms’ CAST, but CAST seems more capable of dealing with other AGIs that are less nice.

My understanding of the key difference is that DWIMAC doesn’t react to dangers that happen too fast for the principal to give instructions, whereas CAST guesses what the principal would want.

If we get a conflict between AIs at a critical time, I’d prefer to have CAST.

Seth’s writing is more readable than Max’s CAST sequence, so it’s valuable to have it around as a complement to Max’s writings.