rvnnt

Karma: 292

rvnnt 18 Sep 2026 18:35 UTC
11 points
0
on: rvnnt’s Shortform
Possibly important and neglected: What is the plan for the following scenario?
- LLM hyperscalers (OpenAI, Anthropic, etc.) go bankrupt ^[1]
- The lesson decisionmakers and the public take away from that is something like “See? All that AI stuff was just hype, AI couldn’t even make a profit. Stories of existential risks were nonsense (or a marketing strategy) all along.”
- The political will to pass any kind of regulation (let alone do international coordination) effectively dies.
- AGI research continues. (Possibly: lots of researchers looking for a more compute-efficient (and thus dangerous) paradigm.)
Some first-pass thoughts:
- Probably a good idea to sometimes try to clearly communicate that LLMs are just one approach to building powerful AIs. Beware leading people to equate “AGI” with “what OpenAI/Anthropic/GDM do”.
- OTOH, don’t go fearmongering about possible more efficient/dangerous paradigms; the historical lesson seems to be that warning about world-endingly-dangerous thing X leads lots of Darwin-Award-aspirants going and working on building X.
1. ↩︎
  I haven’t looked into it much, but IIUC, they have hundreds of billions in purchase commitments (chips/infrastructure), and their continued existence depends on being able to scale up their profits fast enough; which seems somewhat precarious?

rvnnt 8 Sep 2026 15:17 UTC
4 points
1
on: The Scramble: getting in position to pace the frontier

The key decisions were limited to a group of roughly fifteen individuals — the Executive Committee of the National Security Council, or “ExComm” — with President John F. Kennedy himself spending a great deal of time shaping and steering discussions. There was incomplete information, large uncertainty [...] and a lot of chaos.

I expect the initial phase of AI superintelligence management to share these features.

Sounds like it would be a very bad idea to rely on “(democratic) checks and balances” to prevent catastrophic concentration of power, if grabbing such power is possible during a scramble. (E.g. due to the fledgling ASIs being corrigible, or something.)

rvnnt 19 Aug 2026 14:20 UTC
1 point
2
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform
Given that “one” means “some indefinite person in general”, that doesn’t really answer the question. What people do you mean, more concretely? Some executive at an AI lab? Random engineers at an AI lab? The director of the NSA? …? And how would those people restrict who has access to the model’s outputs, if the models are open-weights (and being run on some rando’s personal GPU cluster)?

(I get the sense that you might be thinking about this stuff in something like “far mode”, and it might be useful to try thinking more concretely.)

rvnnt 19 Aug 2026 14:15 UTC
3 points
2
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform
That response seems deeply unserious, in a way that updates me negatively on “is Nina even trying to reason sanely about this issue”.

But just in case you were asking that question earnestly:

The US military ^[1] has not nuked my home because
- they would gain nothing by doing so,
- and it would be actively harmful to their own interests ^[2] .
In contrast,
- Humans in control of an ASI would no longer be constrained to single bodies/brains, and thus would be able to better satisfy their preferences with more resources; and using all the resources leads to everyone else on Earth dying. ^[3]
- Most humans, especially the kinds of humans who tend to end up in positions of power, have desires for things like status and sex. An obvious way to satisfy such desires is by creating lots of slaves of various kinds, or by enslaving existing people. (And given ASI-grade power, they would be able to do so.)
In other words, doing horrible things would probably be strongly in line with the interests/preferences of (power-seeking) humans, and given control of ASI they would be able to act on those preferences.

So, if you’re imagining that a small group of humans in control of ASI would not do terrible things, for the same reasons that the US military would not nuke my home, then AFAICT your imaginings do not remotely track reality, and you might want to do some re-imagining.
1. ↩︎
  or, more precisely, the people in control of nukes within the US military
2. ↩︎
  it would cause immense collateral damage inside a NATO country,
  nukes are expensive,
  it would probably start a war with their own allies,
  it might lead to the people constituting the US military revolting against whatever leaders made the decision to nuke an allied city, and that would be the end of those leaders,
  etc.
3. ↩︎
  As the saying goes, “the AI/God-Emperor (probably) neither loves you nor hates you, but you depend on a whole lot of atoms being in a narrow range of possible configurations, and the AI’s/God-Emperor’s preferences are extremely unlikely to be maximized by any of those specific configurations”.

rvnnt 18 Aug 2026 13:16 UTC
10 points
10
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform

I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs

Who is “we”? And if this “we” has control over a corrigible AGI/ASI, what prevents them from doing terrible things?
What links here?
- CommentBench: Can Models Match Human Comments on AI Safety Posts? by OscarGilg (19 Sep 2026 14:27 UTC; 25 points)

rvnnt 18 Jul 2026 12:57 UTC
17 points
10
on: Announcing the Corrigibility Research Fund
human alignment is [...] amenable to known strategies, such as democratic oversight.

I think that’s probably true in some polities, at least under “normal” conditions that do not include “a small group of humans can gain a decisive strategic advantage over all of Earth”.

However, it does not appear to be true in most of the relevant places in the United States ^[1] . (See e.g. the failed firing of Sam Altman, or various actions of the Trump 2.0 admin.) And even if “democratic oversight” were working as intended, it operates via elections and other feedback mechanisms that were not designed (are too slow) to respond to threats like “the executives can suddenly order the entire automated military to depose their enemies and take control of all critical infra” ^[2] .

I.e., even if solutions to human alignment might exist in theory (and/or Iceland), they appear to currently be absent from the real world. And yet, in order for ASI-being-corrigible to end well, such solutions need to actually be implemented in reality in ways that will actually work under pressure.

I do not think humanity is remotely on track to invent and implement such solutions any time in the foreseeable future. Consequently I think (publicly) working on corrigibility is strongly negative-EV.

A thing that could change my mind: Could you describe, in concrete detail, a “strategy for solving the human alignment problem”, which would actually work under realistic conditions ^[3] , and which seems likely to actually be implemented (in the relevant places) before ASI is built?
1. ↩︎
  Let alone e.g. China or Russia.
2. ↩︎
  Not saying that specific class of scenario is super likely; intended only as an illustrative example.
3. ↩︎
  Humans following their myopic incentives, humans sucking at coordinating, humans being incompetent, humans being selfish sociopaths, etc.

rvnnt 18 Jul 2026 12:55 UTC
8 points
8
on: Announcing the Corrigibility Research Fund
I think that, in the absence of a robust solution to the “human alignment” problem, (public) solutions to corrigibility differentially increase S-risks. Under current conditions, “solving corrigibility” would modestly increase the probability of good futures, and strongly increase the probability of astronomical suffering.

rvnnt 9 Jul 2026 14:01 UTC
2 points
0
on: Modular Pretraining Enables Access Control
Does GRAM remain useful when models become capable of learning continually and re-deriving any missing knowledge?

rvnnt 12 Jun 2026 13:20 UTC
16 points
13
in reply to: Cleo Nardo’s comment on: ryan_greenblatt’s Shortform

You should be worried about corrigibility if: some actor can [...] hold a gun against your head.

Yep. And the government almost surely can and will effectively hold a gun to your head, if they think you’re close to building a superintelligence.

rvnnt 12 Jun 2026 13:18 UTC
2 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
What, in concrete practice, would you predict happens, if some lab succeeded at building a corrigible ASI, or was close to doing so?

I think that, under the current circumstances, the ASI would very likely end up controlled by some power-hungry sociopath. We would hit the third filter.

That being the case, I’m currently leaning towards thinking that (all current approaches to building ASI are terrible but) Anthropic’s approach is less bad on expectation.

(If anyone could lay out a concrete, realistic scenario in which a corrigible ASI gets built on Earth any time in the next 15 years, that does not end with a sociopath in power, I’d be curious to see it! (May or may not change my mind w.r.t. “corrigibility vs constitution” question, depending on how likely that class of scenario seems.))

rvnnt 11 Jun 2026 14:23 UTC
−1 points
−2
in reply to: Ulisse Mini’s comment on: Vladimir Putin’s CEV is probably not that bad
That comment would make sense in a world where everyone is more or less neurotypical and psychopaths ^[1] do not exist. But that’s not planet Earth. The listed things do not even begin to apply to most psychopaths. Most of the worst harm is committed by psychopaths. And positions of high power tend to be strongly disproportionately populated by psychopaths.

EDIT: I believe the above applies in reasonably civilized societies during peacetime. During wartime, or if non-psychopaths are given immense power, I’d expect most of the worst harm to be done by non-psychopaths (because they are more numerous), for entirely mundane reasons of sadism/rapaciousness/spite/etc.
1. ↩︎
  I’m using the term “psychopath” loosely here; hopefully my point still comes across.

rvnnt 11 Jun 2026 14:19 UTC
3 points
0
on: Vladimir Putin’s CEV is probably not that bad
IIUC, you (habryka) are saying that in order for the dictator to fill the universe with sentient slaves, the dictator would have to care about “the full depth and complexity” of others’ suffering/submission.

Yet you also say that the same dictator would probably fill the universe with nice things like happy sentient people, without making a strong argument for why we should expect the dictator to care about the full depth & complexity of others’ happiness.

It seems to me you’re applying the “would have to care about the full complexity of X” argument only to bad stuff, but not to good stuff.

In what way do you think people’s desire for others to submit/suffer/worship differs from people’s desire for others to be happy/free/etc, that it makes sense to apply that argument so asymmetrically?

rvnnt 10 Apr 2026 14:29 UTC
3 points
0
on: Claude Mythos: The System Card

One worry I have with ‘the [AI] whisperers’ and others who investigate these matters is that they may think the model they see is in important senses the true one far more than it is, as opposed to being one aspect or mask out of many.

Imagine telling someone from (e.g.) 2016 that the above sentence is a reasonable thing to say in 2026. (The frog, it boils.)

rvnnt 4 Apr 2026 12:31 UTC
12 points
5
on: Am I the baddie?
I think it might be more accurate to say you’re an efficient component in Moloch’s machine.

But if you care about how things like gradual disempowerment play out, then I think the “baddie / goodie / pawn-of-Moloch” framing is probably not very useful. It might be worth instead thinking more concretely, about things like
- How much are my actions contributing to speeding up human disempowerment? ^[1]
- How could I keep my job (or whatever) while contributing as little as possible to various bad things?
- Who are the relevant actors I would need to coordinate with, in order to slow things down? What, concretely, is stopping me from coordinating with them, and how could I fix that?
- What other important considerations are there, besides “speeding up adoption of AI / replacement of humans”?
- What could I do to offset harms I cause?
1. ↩︎
  accounting for the other actors in the Molochian race

rvnnt 13 Feb 2026 8:44 UTC
1 point
0
on: [Hiring] Principia Research Fellows
How would you rate Principia’s operational adequacy? In particular: how would you rate Principia on closure and opsec? ^[1]
1. ↩︎
  Alternatively, if you use a different framework for thinking about operational adequacy, I’d be very curious to read your assessment w.r.t. that framework!

rvnnt 9 Feb 2026 16:57 UTC
2 points
0
on: Design sketches for a more sensible world
I think something like what you’re sketching here, viz “harnessing technology to make people and civilization saner”, is probably highly valuable and possibly quite neglected. ^[1] Thank you for working on this.

A class of infrastructure/technologies that seem very important, but which I didn’t see mentioned in this post: infrastructure for creating common knowledge of better equilibria and coordinating transitions to them. ^[2] Do you (have plans to) address anything matching that (vague) description anywhere?
1. ↩︎
  Low-hanging dignity points!
2. ↩︎
  I.e., something that would solve problems of form “there exists a much better equilibrium, but getting there would require lots of people to have common knowledge of that better equilibrium, and also coordinate and sufficiently-credibly commit to near-simultaneously taking action that would be detrimental to them if they took it alone”. Some examples: move from frequentist stats to Bayesian stats; make it easier for AI labs to (conditionally) stop racing; US voters coordinate to vote for a less sociopathic party-independent candidate (or to replace first-past-the-post with a saner voting system entirely); abolish all JavaScript forever, refactor the Internet to use a non-insane language; kill Elsevier; almost everyone simultaneously leaves (at least the more toxic platforms of) social media (and move to a less toxic new platform); journals/researchers commit to preregistering studies and publishing negative results; etc.

rvnnt 6 Feb 2026 15:04 UTC
2 points
1
on: The Simplest Case for AI Catastrophe

There’s some chance that the patchwork AI safety strategy of the leading companies might just work well enough

I wonder if it’d be a good idea to mention that “a small number of humans have intent-aligned vastly superhuman AIs” does not yet imply “things will be nice (or tolerable) for most people”?

OTOH, then you maybe also need to point out that “superhuman AIs are open-weights and anyone can run them” also leads to death-or-worse (to pre-empt “solutions” like “so let’s give everyone AI!”)?

rvnnt 21 Jan 2026 16:06 UTC
1 point
0
on: What do you do when you find out you have inconsistent probabilities?
The four probabilities given as premises are inconsistent. The first three determine the fourth. (Also, there’s an arithmetic error in the p(G|M) calculation, as pointed out by Bucky.)

Given
- p(M) = 0.9
- p(G) = 0.05
- p(M|G) = 0.99
it must be that

p(M|-G) = (p(M) - p(M,G)) / (1 - p(M,G) - p(-M,G)) = 0.8505 / 0.95

which is approximately 0.895263. Not 0.02.

If this feels confusing, I suggest drawing a Venn diagram or something. If you have a box of area 1.0, containing a blob M of area 0.9 and another blob G of area 0.05, such that G is almost entirely inside M, then...

rvnnt 21 Jan 2026 13:36 UTC
3 points
0
in reply to: Mitchell_Porter’s comment on: rvnnt’s Shortform
Interesting. Thanks. How did you arrive at the above picture? Any sources of information you’d recommend in particular?

rvnnt 20 Jan 2026 16:35 UTC
11 points
7
on: rvnnt’s Shortform
After reading about Trump’s actions w.r.t. Greenland, I’m updating further away from
- Trump’s policies/actions are at least partially aimed at pursuing the national interests of the US (albeit possibly in a very misguided/incompetent way)
and further in favor of both
- Trump has gone insane,
- and/or Trump is intentionally acting on behalf of Putin.
I’d like to find more/better sources of evidence about “what is the US executive branch optimizing for?”; curious to hear suggestions.

(Also, to Americans: How high/low salience is the issue in the US? Also: curious to read your analysis of your chief executive’s behavior.)