rvnnt

Karma: 243

rvnnt 12 Jun 2026 13:20 UTC
16 points
13
in reply to: Cleo Nardo’s comment on: ryan_greenblatt’s Shortform

You should be worried about corrigibility if: some actor can [...] hold a gun against your head.

Yep. And the government almost surely can and will effectively hold a gun to your head, if they think you’re close to building a superintelligence.

rvnnt 12 Jun 2026 13:18 UTC
1 point
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
What, in concrete practice, would you predict happens, if some lab succeeded at building a corrigible ASI, or was close to doing so?

I think that, under the current circumstances, the ASI would very likely end up controlled by some power-hungry sociopath. We would hit the third filter.

That being the case, I’m currently leaning towards thinking that (all current approaches to building ASI are terrible but) Anthropic’s approach is less bad on expectation.

(If anyone could lay out a concrete, realistic scenario in which a corrigible ASI gets built on Earth any time in the next 15 years, that does not end with a sociopath in power, I’d be curious to see it! (May or may not change my mind w.r.t. “corrigibility vs constitution” question, depending on how likely that class of scenario seems.))

rvnnt 11 Jun 2026 14:23 UTC
−1 points
−2
in reply to: Ulisse Mini’s comment on: Vladimir Putin’s CEV is probably not that bad
That comment would make sense in a world where everyone is more or less neurotypical and psychopaths ^[1] do not exist. But that’s not planet Earth. The listed things do not even begin to apply to most psychopaths. Most of the worst harm is committed by psychopaths. And positions of high power tend to be strongly disproportionately populated by psychopaths.
1. ↩︎
  I’m using the term “psychopath” loosely here; hopefully my point still comes across.

rvnnt 11 Jun 2026 14:19 UTC
3 points
0
on: Vladimir Putin’s CEV is probably not that bad
IIUC, you (habryka) are saying that in order for the dictator to fill the universe with sentient slaves, the dictator would have to care about “the full depth and complexity” of others’ suffering/submission.

Yet you also say that the same dictator would probably fill the universe with nice things like happy sentient people, without making a strong argument for why we should expect the dictator to care about the full depth & complexity of others’ happiness.

It seems to me you’re applying the “would have to care about the full complexity of X” argument only to bad stuff, but not to good stuff.

In what way do you think people’s desire for others to submit/suffer/worship differs from people’s desire for others to be happy/free/etc, that it makes sense to apply that argument so asymmetrically?

rvnnt 10 Apr 2026 14:29 UTC
3 points
0
on: Claude Mythos: The System Card

One worry I have with ‘the [AI] whisperers’ and others who investigate these matters is that they may think the model they see is in important senses the true one far more than it is, as opposed to being one aspect or mask out of many.

Imagine telling someone from (e.g.) 2016 that the above sentence is a reasonable thing to say in 2026. (The frog, it boils.)

rvnnt 4 Apr 2026 12:31 UTC
12 points
5
on: Am I the baddie?
I think it might be more accurate to say you’re an efficient component in Moloch’s machine.

But if you care about how things like gradual disempowerment play out, then I think the “baddie / goodie / pawn-of-Moloch” framing is probably not very useful. It might be worth instead thinking more concretely, about things like
- How much are my actions contributing to speeding up human disempowerment? ^[1]
- How could I keep my job (or whatever) while contributing as little as possible to various bad things?
- Who are the relevant actors I would need to coordinate with, in order to slow things down? What, concretely, is stopping me from coordinating with them, and how could I fix that?
- What other important considerations are there, besides “speeding up adoption of AI / replacement of humans”?
- What could I do to offset harms I cause?
1. ↩︎
  accounting for the other actors in the Molochian race

rvnnt 13 Feb 2026 8:44 UTC
1 point
0
on: [Hiring] Principia Research Fellows
How would you rate Principia’s operational adequacy? In particular: how would you rate Principia on closure and opsec? ^[1]
1. ↩︎
  Alternatively, if you use a different framework for thinking about operational adequacy, I’d be very curious to read your assessment w.r.t. that framework!

rvnnt 9 Feb 2026 16:57 UTC
2 points
0
on: Design sketches for a more sensible world
I think something like what you’re sketching here, viz “harnessing technology to make people and civilization saner”, is probably highly valuable and possibly quite neglected. ^[1] Thank you for working on this.

A class of infrastructure/technologies that seem very important, but which I didn’t see mentioned in this post: infrastructure for creating common knowledge of better equilibria and coordinating transitions to them. ^[2] Do you (have plans to) address anything matching that (vague) description anywhere?
1. ↩︎
  Low-hanging dignity points!
2. ↩︎
  I.e., something that would solve problems of form “there exists a much better equilibrium, but getting there would require lots of people to have common knowledge of that better equilibrium, and also coordinate and sufficiently-credibly commit to near-simultaneously taking action that would be detrimental to them if they took it alone”. Some examples: move from frequentist stats to Bayesian stats; make it easier for AI labs to (conditionally) stop racing; US voters coordinate to vote for a less sociopathic party-independent candidate (or to replace first-past-the-post with a saner voting system entirely); abolish all JavaScript forever, refactor the Internet to use a non-insane language; kill Elsevier; almost everyone simultaneously leaves (at least the more toxic platforms of) social media (and move to a less toxic new platform); journals/researchers commit to preregistering studies and publishing negative results; etc.

rvnnt 6 Feb 2026 15:04 UTC
2 points
1
on: The Simplest Case for AI Catastrophe

There’s some chance that the patchwork AI safety strategy of the leading companies might just work well enough

I wonder if it’d be a good idea to mention that “a small number of humans have intent-aligned vastly superhuman AIs” does not yet imply “things will be nice (or tolerable) for most people”?

OTOH, then you maybe also need to point out that “superhuman AIs are open-weights and anyone can run them” also leads to death-or-worse (to pre-empt “solutions” like “so let’s give everyone AI!”)?

rvnnt 21 Jan 2026 16:06 UTC
1 point
0
on: What do you do when you find out you have inconsistent probabilities?
The four probabilities given as premises are inconsistent. The first three determine the fourth. (Also, there’s an arithmetic error in the p(G|M) calculation, as pointed out by Bucky.)

Given
- p(M) = 0.9
- p(G) = 0.05
- p(M|G) = 0.99
it must be that

p(M|-G) = (p(M) - p(M,G)) / (1 - p(M,G) - p(-M,G)) = 0.8505 / 0.95

which is approximately 0.895263. Not 0.02.

If this feels confusing, I suggest drawing a Venn diagram or something. If you have a box of area 1.0, containing a blob M of area 0.9 and another blob G of area 0.05, such that G is almost entirely inside M, then...

rvnnt 21 Jan 2026 13:36 UTC
3 points
0
in reply to: Mitchell_Porter’s comment on: rvnnt’s Shortform
Interesting. Thanks. How did you arrive at the above picture? Any sources of information you’d recommend in particular?

rvnnt 20 Jan 2026 16:35 UTC
11 points
7
on: rvnnt’s Shortform
After reading about Trump’s actions w.r.t. Greenland, I’m updating further away from
- Trump’s policies/actions are at least partially aimed at pursuing the national interests of the US (albeit possibly in a very misguided/incompetent way)
and further in favor of both
- Trump has gone insane,
- and/or Trump is intentionally acting on behalf of Putin.
I’d like to find more/better sources of evidence about “what is the US executive branch optimizing for?”; curious to hear suggestions.

(Also, to Americans: How high/low salience is the issue in the US? Also: curious to read your analysis of your chief executive’s behavior.)

rvnnt 28 Dec 2025 11:03 UTC
5 points
0
in reply to: Fabien Roger’s comment on: Fabien’s Shortform
Dominic Cummings (former Chief Adviser to the UK PM) has written some things about nuclear strategy and how it’s implemented in practice. IIUC, he’s critical of (i.a.) how Schelling et al.’s game-theoretic models are (often naively/blindly) applied to the real world.

rvnnt 23 Dec 2025 9:30 UTC
2 points
0
in reply to: brambleboy’s comment on: A friction in my dealings with friends who have not yet bought into the reality of AI risk
I updated a bit towards thinking that incompetence-at-reasoning is a more common/influential factor than I previously thought. Thanks.

However: Where do you think that moral realism comes from? Why is it a “thorny” issue?

rvnnt 20 Dec 2025 21:11 UTC
4 points
0
on: A Full Epistemic Stack: Knowledge Commons for the 21st Century
social-media-like interfaces for uncovering group wisdom and will at larger scales while eliciting more productive discourse

That seems like it might significantly help with raising the sanity waterline, and thus help with coordinating on AI x-risk, and thus be extremely high-EV (if it’s successfully implemented, widely adopted, and humanity survives for a decade or two beyond widespread adoption). ^[1]

Do you think it would be practically possible with current LLMs to implement a version of social media that promoted/suggested content based on criteria like
- validity of reasoning,
- concrete factual claims (whether correct or incorrect), vs vibing,
- thoughtful long-form writing, vs memeing,
- good-faith debate, vs e.g. adversarial tribalistic signaling?
1. ↩︎
  The “widely adopted” part seems difficult to achieve, though. The hypermajority of genpop humans would probably just keep scrolling TikTok and consuming outrage porn on X, even if Civilization 2.0 Wholesome Social Media were available.

rvnnt 20 Dec 2025 21:10 UTC
3 points
0
on: A Full Epistemic Stack: Knowledge Commons for the 21st Century

This discourse structure associates related claims and evidence, [...]

To make it practically possible for non-experts to efficiently make sense of large, spread-out collections of data (e.g. to answer some question about the discourse on some given topic), it’s probably necessary to not only rapidly summarize all that data, but also translate it into some easily-human-comprehensible form.

I wonder if it’s practically possible to have LMs read a bunch of data (from papers to Twitter “discourse”) on a given topic, and rapidly/on-demand produce various kinds of concise, visual, possibly interactive summaries of that topic? E.g. something like this, or a probabilistic graphical model, or some kind of data visualization (depending on what aspect of what kind of topic is in question)?

rvnnt 20 Dec 2025 21:10 UTC
3 points
0
on: A Full Epistemic Stack: Knowledge Commons for the 21st Century

Ideally perhaps, raw observations are reliably recorded, [...]

Do you have ideas for how to deal with counterfeit observations or (meta)data (e.g. deepfaked videos)?

rvnnt 16 Dec 2025 17:48 UTC
2 points
−3
on: A friction in my dealings with friends who have not yet bought into the reality of AI risk
Given that the basic case for x-risks is so simple/obvious ^[[1]] , I think most people arguing against any risk are probably doing so due to some kind of myopic/irrational subconscious motive. (It’s entirely reasonable to disagree on probabilities, or what policies would be best, etc.; but “there is practically zero risk” is just absurd.)

So I’m guessing that the deeper problem/bottleneck here is people’s (emotional) unwillingness to believe in x-risks. So long as they have some strong (often subconscious) motive to disbelieve x-risks, any conversation about x-risks is liable to keep getting derailed or be otherwise very unproductive. ^[[2]]

I think some common underlying reasons for such motivated disbelief include
1. Subconscious Map-Territory confusions:
  1. Believing X makes X feel more real. And so, System 1 “solves” x-risks by disbelieving them away.
  2. System 1 makes decisions—including decisions for what to believe—under the assumption that the current Map is the Territory. For example: If the current Map says that “x-risks are not real and that’s good because now I can keep making money developing AGI capabilities”, then according to that Map, it would be bad to update towards believing that x-risks are real (because then maybe you’d needlessly stop making money!)
2. Cognitive immune response. Lots of humans have a strong subconscious instinct along the lines of “If a clever human is trying, with abstract arguments, to convince me of X, and I anticipate that believing X would change my policy/behavior in ways that feel unpleasant, then I reject/disbelieve X”. ^[[3]]
I’m not sure what the best approaches are to addressing the above kinds of dynamics. Trying to directly point them out seems likely to end badly (at least with most neurotypical people). If you can somehow get people to (earnestly) do them, small mental exercises like Split and Commit or giving oneself a line of retreat might help for (1.)? For (2.), maybe
- avoid trying to persuade; instead ask questions that prompt the person to think concretely about the situation themself,
- show them something tangible, like some AI model doing something impressive, or that slowed-down video of a train station (along with “this is what humans would look like to an entity that thinks 100 times faster—statues”)?
If you try the above, I’d be curious to see a writeup of the results.
1. ↩︎
  Building a species of superhumanly smart & fast machine aliens without understanding how they work seems very dangerous. And yet, various companies and nations are currently pouring trillions of dollars into making that happen, and appear to be making rapid progress. (Experts disagree on whether there’s a 99% chance we all die, or if there’s only a 10% chance we all die and a 90% chance some corporate leaders become uncontested god-emperors, or if we end up as pets to incomprehensible machine gods, or if the world will be transformed beyond human comprehension and everyone will rely on personal AI assistants to survive. Sounds good, right?)
2. ↩︎
  A bit like trying to convince a deeply religious person via rational debate. It’s not really about the evidence/reasoning.
3. ↩︎
  I wouldn’t be too surprised if this kind of instinct were evolved, rather than just learned. Even neurotypical humans try to hack each other all the time, and clever psychopaths have probably been around for many, many generations.

rvnnt 15 Nov 2025 17:47 UTC
3 points
0
on: Human Values ≠ Goodness

Think of the stuff that, when you imagine it, feels really yummy.

Also worth taking into consideration: things that feel anti-yummy. Fear/disgust/hate/etc are also signals about your values.

rvnnt 15 Nov 2025 17:43 UTC
5 points
2
on: Human Values ≠ Goodness
I think the “your values” -framing itself already sneaks in assumptions which are false for a lot of minds/brains. Notably: most minds are not perfectly monolithic/unified things well-modeled as a coherent “you/I/me”. And other minds are quite unified/coherent, but are in the unfortunate situation of running on a brain that also contains other (more or less adversarial) mind-like programs/wetware.

Example:

It is entirely possible to have strongly-held values such as “I reject so-and-so arbitrary/disgusting parts of the reward circuitry Evolution designed into my brain; I will not become a slave to the Blind Idiot God’s whims and attempts to control me”. In that case, the “I” that holds those values clearly excludes at least some parts of its host brain’s yumminess-circuitry.^[1] (I.e., feelings of yumminess forced upon the mind are not signals about that mind’s values, but rather more like attempts by a semi-adversarial brain to hack that mind.)

Another example:

Alex has some shitty experiences in childhood, and strongly internalizes a schema S like “if I do X, I will be safe”, and thereafter has strong yumminess feelings about doing X. But later upon reflection, Alex realizes that yumminess feelings are coming from S, and that S’s implicit models of reality aren’t even remotely accurate now in adulthood. Alex would like to delete S from their brain, but can’t. So the strong yumminess-around-X persists. Is X one of Alex’s values?

So, I object to what I perceive to be an attempt to promote a narrative/frame about what constitutes “you/I/me” or “your values” for people in general. (Albeit that I’m guessing that there was no malice involved in that promoting.) Especially when it is a frame that seems to imply that many people (as they conceive of themselves) are not really/fully persons, and/or that they should let arbitrary brain-circuits corrupt their souls (if those brain-circuits happen to have the ability to produce feelings of yumminess).

Please be more careful about deploying/rolling your own metaethics.
1. ↩︎
  Maybe that “I” could be described as a learned mesaoptimizer, something that arose “unintentionally” from perspective of some imaginable/nonexistent Evolution-aligned mind-designer. But so what? Why privilege some imaginary Evolution fairy over an actually existing person/mind?