Greg C

Karma: 446

Greg C 22 Apr 2026 17:19 UTC
8 points
0
on: Preventing extinction from ASI on a $50M yearly budget
We’re currently not set up to receive smaller donations.
Do you plan on rectifying this soon? If you’re building a mass movement, you should be aiming for hundreds of thousands of people donating $10 - $20/month (say, the cost of an AI subscription..). That would give you a lot of funding, and also a lot of democratic legitimacy.

Greg C 22 Apr 2026 16:38 UTC
18 points
14
in reply to: cousin_it’s comment on: Preventing extinction from ASI on a $50M yearly budget
I think you have a point, but I still think the Control AI strategy is probably now the best investment out there in terms of reducing p(doom). The backfire effects are limited (given many powerful people are already aware of AI), and the upside is very high in expectation.
its power base needs to be at the bottom.
This might be true, but the fact remains that we need a well enforced international treaty. And only the people at the top can make that happen. The masses have a big role too though: enacting a global taboo on AGI/ASI research, similar to the ones that already exists for other things like human cloning, which will make enforcing the treaty so much easier.

Greg C 23 Mar 2026 17:31 UTC
1 point
0
in reply to: Greg C’s comment on: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Update: things have come along way in the last 3 years:

Greg C 15 Feb 2026 14:50 UTC
1 point
0
on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
it may be useful to briefly list some of the ways that an AI pause, or efforts to bring about such a pause, could have undesirable effects (aside from simply delaying the arrival of the benefits that successful AGI could bring)
I think this section is mostly just strawmen.
The pause occurs too early. People conclude that it was pointless, and become less willing to pause later when it would have been useful.
This might’ve been true of a Pause 2023 (in hindsight), but seems unlikely for a Pause in 2026.
The call for a pause results in poorly designed or incomplete regulation...Compliance and box-ticking crowd out substantive work on risk reduction… Work may be driven underground, or shift towards less scrupulous actors or less cooperative states....The pause has an exemption for national security, pushing AI activities away from the civilian into the military sector… An international agreement is reached on pausing, but this creates a prisoner’s dilemma in which some parties cheat
These are all just saying that a Pause might be unsuccessful. We will have failed then (by not actually getting a Pause). This doesn’t mean that accelerating isn’t strictly worse in terms of risk.
A pause is implemented, leading to economic recession and general pessimism and lowered hopes for the future. People see the world more as a zero-sum battle for a limited set of resources, increasing conflict and tribalism.
This is just a non-sequitur. Our whole global economy is very far from being dependent on (proto-)AGI. How did we possibly manage to have economic prosperity without AI before?
A pause prolongs the period during which the world is exposed to dangers from applications of already developed levels of AI (and to risks independent of AI), which more advanced AI could have helped mitigate.
Why is more advanced AI more likely to help than just cause more risk? This is just blind faith, when no one has the faintest idea how to solve superalignment or ASI control.
To enforce a pause, a strong control apparatus is created. The future shifts in a more totalitarian direction
We don’t live under global totalitarianism because nuclear material, and chemical and biological weapons, (and hell, even conventional weapons in most countries,) are controlled. Right now controlling large data centers isn’t any more difficult than that.
When the pause is eventually lifted, there is a massive compute and/or algorithm overhang that leads to explosive advances in AI that are riskier than if AI had advanced at a steadier pace throughout.
This again, is a failed Pause. A successful Pause would prevent (and taboo) further compute and algorithm progress. And any real Pause would not be lifted until there is a consensus on safety.
The world will also not have had the opportunity to learn from and adapt to living with weaker AI systems.
We are already doing that now. Those systems will not going away with a Pause in further development.
Attitudes towards AI become polarized to such an extent as to make constructive dialogue difficult and destroy the ability of institutions to pass nuanced adaptive safety policy.
Nick, you really aren’t helping with this paper.
The push for a pause galvanizes supporters of AI to push back. Leading AI firms and AI authorities close ranks to downplay risk, marginalizing AI safety researchers and policy experts concerned with AI risk, reducing their resourcing and influence.
This is already happening. The blame can’t be placed on the Pausers.
A pause, initially sold as a brief moratorium to allow social adjustments and safety work to catch up, calcifies into a de facto permaban that prevents the immense promise of superintelligence from ever being realized—or is indefinitely extended without ever being formally made permanent.
Any realistic global Pause is never going to be of a fixed, pre-determined length. It will be lifted when there is global consensus on lifting it (either through consensus on safety, or consensus on taking the risk).

Greg C 12 Jan 2026 17:35 UTC
2 points
0
in reply to: alexgieg’s comment on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
Ok, but Mutually Assured AI Malfunction (MAIM) is another possibility.

Greg C 6 Jan 2026 20:57 UTC
1 point
0
in reply to: alexgieg’s comment on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
Once they realise the risk of extinction isn’t “tiny” (and we can all help, here), then the rational move is to not play, and prevent anyone else from playing.

Greg C 6 Jan 2026 12:09 UTC
5 points
3
on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
One option, if you want to do a lot more about it than you currently are, is Pause House. Another is donating to PauseAI (US, Global). In my experience, being pro-active about the threat does help.

Greg C 6 Jan 2026 11:14 UTC
1 point
0
in reply to: StanislavKrym’s comment on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
I think your prompt to Claude is pretty leading^[1]. You are assuming the answer with “the AIs end up with motivations similar to those of the humans”. The point is that we don’t actually know what their underlying motivations are—we only see how they act when trained and system-prompted into mimicking humans. And no alignment techniques are even 3 9s reliable (and we need >13 9s in the limit of ASI).

Also “Can this crux be partially resolved by, say, studying the values of humans whose brain was developed abnormally” is not thinking at the right level of abstraction. Humans who’s brains developed abnormally are still very close to normal humans in the grand scheme of mindspace. AIs share zero evolutionary history and development (evo-devo), and close to zero brain architecture with humans. Sharing our corpus of media is a very shallow and brittle substitute (i.e. it can make a half-decent mask for the shoggoth, but it doesn’t do anything in the way of evolving the shoggoth into a digital human).
1. ^
  not to mention that the fact that you are using Claude as a trusted source of information on this in the first place is problematic.

Greg C 5 Jan 2026 20:31 UTC
6 points
4
in reply to: Bogdan Ionut Cirstea’s comment on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
By catastrophe, I was thinking of something much worse than Covid; or indeed, x-risky. Point-of-no-return is a good stand-in. So: what makes you confident that AI safety research will be automated before a point-of-no-return for humanity is crossed?
I’m pretty confident it’s feasible to at the very least 10x AI safety prosaic research through AI augmentation without increasing x-risk by more than 1% yearly
I’d agree that it’s feasible—but is it at all likely? Surely that would require us to Pause at ~the current level (as you say: “LLMs already seem like they might be human-level at (ML) research ideation.”). You aren’t getting only a 1% increase in x-risk yearly on the current trajectory.
I think Claudes are probably already more beneficial and more prosocial companions than the median human
I think you (like many in the LW/EA/AIS community) might be on a slippery slope here to having your mind altered by AI use to the point of losing sight of the fact that these things are fundamentally alien underneath. (See also.)

Greg C 5 Jan 2026 20:16 UTC
2 points
1
in reply to: StanislavKrym’s comment on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
Yes, but it wouldn’t be given the throne. It will take it (or rather, just obliterate it).

Greg C 5 Jan 2026 16:27 UTC
2 points
0
on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
If you want to share this, especially to people outside the LW community, I’ve also posted it on X and Substack.

Greg C 5 Jan 2026 13:32 UTC
11 points
6
in reply to: Bogdan Ionut Cirstea’s comment on: AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!
What makes you confident that AI safety research will be automated before catastrophe is automated?

AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!

Greg C5 Jan 2026 11:57 UTC

61 points

18 comments1 min readLW link

Greg C 1 Dec 2025 14:49 UTC
1 point
0
in reply to: Zac Hatfield-Dodds’s comment on: Leaving Open Philanthropy, going to Anthropic
Good to hear. Look forward to seeing the results!

Greg C 7 Nov 2025 19:02 UTC
6 points
5
in reply to: Greg C’s comment on: Leaving Open Philanthropy, going to Anthropic
(I say this as someone who has already put a lot of their money where their mouth is.)

Greg C 7 Nov 2025 18:47 UTC
1 point
0
in reply to: Drake Thomas’s comment on: Leaving Open Philanthropy, going to Anthropic
Interesting. I really hope that some of them do something, soon. Time is fast running out. There’s no point being a rich philanthropist (or rich, or a philanthropist) if the world gets destroyed before you deploy your resources.

Greg C 6 Nov 2025 13:18 UTC
3 points
0
in reply to: Drake Thomas’s comment on: Leaving Open Philanthropy, going to Anthropic
Thanks, that’s good to hear. What form does the pledge take? Do you have a DAF that contains half your shares? When do you think the next liquidation opportunity might be? (I guess you weren’t eligible for the one in May^[1]?)
1. ^
  I’m disappointed that no one (EA-ish or otherwise) seems do have done anything interesting with that liquidation opportunity.

Greg C 5 Nov 2025 9:21 UTC
3 points
0
in reply to: Drake Thomas’s comment on: Leaving Open Philanthropy, going to Anthropic
Have you donated any of your equity yet? If not, why not?

Greg C 23 Oct 2025 14:32 UTC
1 point
0
on: Which side of the AI safety community are you in?
Shared on the EA Forum, with some commentary on the state of the EA Community (I guess the LessWrong rationality community is somewhat similar?)

Greg C 23 Oct 2025 14:29 UTC
7 points
5
in reply to: mattmacdermott’s comment on: Which side of the AI safety community are you in?
In practice, bans can be lifted, so “never” is never going to become an unassailable law of the universe. And right now, it seems misguided to quibble over “Pause for 5, 10, 20 years”, and “Stop for good”, given the urgency of the extinction threat we are currently facing. If we’re going to survive the next decade with any degree of certainty, we need an alliance between B1 and B2, and I’m happy for one to exist.

Greg C

AI Risk timelines: 10% chance (by year X) should be the head­line (and dead­line), not 50%. And 10% is _this year_!

AI Risk timelines: 10% chance (by year X) should be the headline (and deadline), not 50%. And 10% is _this year_!